Building a Vertical Search Engine Over 2,196 University Publications

An Information Retrieval assignment asked for a working search engine over a real, live website — not a toy corpus. I picked Coventry University’s Centre for Healthcare and Community Transformation (CHCT) page on PurePortal, the university’s research database, and built a vertical search engine over its publications: crawler, TF-IDF index, cosine ranking, and a small web UI, no search library involved anywhere.

The finished crawl holds 2,196 publications from 117 department members. Source: github.com/rustamniraula90/ir-assignment (search-engine/).

Three things about this turned out to matter more than the TF-IDF math itself: PurePortal sits behind Cloudflare, so getting past that shaped the whole crawler design; a page cap that isn’t documented anywhere silently truncated an early version of the crawl; and building the ground truth for evaluation by hand made the limits of a bag-of-words ranker completely explicit rather than something I’d just have to take on faith.

---

Architecture: three modules, MongoDB in between

The whole system is a three-stage pipeline with MongoDB as the handoff point between stages, deliberately kept as separate processes rather than one monolith:

PurePortal (live site) → crawler.py → MongoDB → indexer.py → in-memory TF-IDF index → api.py → browser

• crawler.py — a long-running, resumable crawler. It never touches a request path; it just keeps MongoDB’s publications collection current.
• indexer.py — pure indexing and scoring logic. No web framework, no I/O beyond reading MongoDB. Given a query string, it returns ranked results.
• api.py — a FastAPI app that builds the index once at startup, serves a search page and a JSON endpoint, and rebuilds the index from MongoDB every hour in the background.

MongoDB holds two collections: urls, which doubles as the crawl frontier and queue, and publications, the extracted records. The TF-IDF index itself is never persisted — it’s rebuilt in memory from publications whenever the API starts or its hourly refresh fires. That split matters: the crawler can be killed, restarted, or left running for days without the search API caring, and the API can be redeployed without re-crawling anything.

Seeding from the wrong page silently caps a crawl at 75 results

The obvious way to crawl a department’s publications is to start from the organisation’s own aggregated publications listing and follow its pagination. That’s what the first version of the crawler did, and it always stopped at exactly 75 results — regardless of how many publications the department’s members actually had between them. That number turned out to be a hard cap on that listing page itself, not a bug in the pagination logic, and it’s not documented anywhere on the site.

A single prolific staff member can have several hundred publications, so 75 total was obviously wrong. The fix was to change what the crawl seeds from: the organisation’s persons listing instead, then each member’s own paginated /publications/ page individually. That gives three distinct URL types, which the crawl queue prioritizes in this order so the full member list is discovered early:

PRIORITY = {
    "profiles": 0,             # paginated list of department members
    "profile_publications": 1, # one member's paginated publication list
    "publication": 2,          # one publication's detail page
}

extract_content() dispatches on this type with a match statement, and each case both parses the current page and enqueues whatever it finds next — member profile pages enqueue that member’s publication listing, publication listings enqueue individual publication pages (plus their own next page), and publication pages just get scraped for title, abstract, keywords, authors and year.

The crawl frontier lives in MongoDB, not in memory

urls isn’t a log of what’s been crawled — it is the frontier. Every URL document carries a next_crawl_date, a priority, and a retries counter:

def get_next_job():
    return db_urls.find_one_and_update(
        {"next_crawl_date": {"$lt": datetime.now(timezone.utc)}},
        {"$set": {"next_crawl_date": datetime.now(timezone.utc) + timedelta(minutes=10)}},
        sort=[("priority", 1), ("next_crawl_date", 1)],
    )

find_one_and_update claims a job atomically and pushes its next_crawl_date forward by ten minutes as a claim window, so the crawler can be killed and restarted without two runs ever grabbing the same URL, and without losing the queue’s position. On success, mark_success() pushes next_crawl_date out by seven days rather than deleting the record — the same document gets recrawled on a weekly cadence for as long as the crawler keeps running, which reproduces a scheduled recrawl without cron or any external scheduler. On failure, schedule_retry() backs off by an hour, capped at five retries, before giving up until the normal weekly slot — a transient network blip costs an hour, not a week.

Getting past Cloudflare without driving a browser for every page

PurePortal sits behind Cloudflare’s Turnstile challenge, which a plain requests.Session cannot pass. Running every single page through a full browser would work but is slow — a Selenium round trip per publication, times 2,196 pages, adds up fast for a resource nobody but a crawl loop is going to see rendered.

The approach instead: open one persistent, undetected browser session (via seleniumbase’s uc_open_with_reconnect + uc_gui_click_captcha) at the start of a crawl run, solve the challenge once, then copy the resulting cf_clearance cookie into a plain requests.Session that handles every page after that:

def fetch_page(sb, url, max_challenge_retries=2):
    response = session.get(url, timeout=30)
    if response.status_code == 200 and not is_challenge_page(response.text):
        return response.text
    # ...fall back to the browser, re-solve, copy cookies back into `session`

is_challenge_page() checks the response for a <title> of “Just a moment…” or a #challenge-running element, since a 200 status alone doesn’t mean the response is real content — Cloudflare serves the interstitial with a 200 too. Most pages never touch the browser at all; it only comes back into play if the session cookie has expired or a fresh challenge appears.

Two failure modes get handled at different levels. An ordinary bad page — a timeout, a malformed response — is caught inside process_job() and goes through the normal schedule_retry() backoff. A dead browser session is a different, fatal failure (BrowserDiedError, raised on any WebDriverException or connection error while the browser is active), and it’s handled one level up, in crawl(): the whole browser session is torn down and relaunched from scratch after a ten-second pause, and the crawl loop resumes from whatever MongoDB’s queue currently holds — no in-memory state to lose.

robots.txt, and choosing availability over strict compliance

The crawler fetches and parses robots.txt once per crawl run and checks every candidate URL against it before fetching:

def can_fetch(rp, url):
    if rp is None:
        return True
    return rp.can_fetch("*", url)

If that initial fetch fails, rp is None, and can_fetch() degrades to allowing everything rather than halting the crawl. get_crawl_delay() degrades the same way, falling back to a fixed five-second delay if robots.txt didn’t specify one. That’s a deliberate trade: a single failed robots.txt fetch shouldn’t take down an otherwise-working crawl, and the crawl rate stays capped by the fallback delay either way, so the politeness contract with the site doesn’t actually go away — it just falls back to a sane default instead of blocking on a request that failed.

Per-field TF-IDF, not one vector per document

indexer.py builds three completely separate TF-IDF indexes — title, abstract, keywords — each with its own vocabulary and its own IDF table, rather than concatenating all the text into one vector per publication:

FIELD_WEIGHTS = {"title": 0.5, "keywords": 0.3, "abstract": 0.2}

The reason for keeping them separate: a single combined vector scores a query term appearing in the title exactly the same as the same term buried somewhere in a 300-word abstract, which throws away the strongest positional signal a publication record has. Across the full 2,196-publication corpus this produces three vocabularies of very different sizes — 3,284 terms for title, 3,065 for keywords, 10,350 for abstract — which on its own says something about how much more lexical variety a free-text abstract carries versus a controlled, short title.

Preprocessing is one shared pipeline, applied identically to indexed text and incoming queries — lowercase, split on non-letters, drop English stopwords, stem with Porter:

def clean_and_tokenize(text):
    text = text.lower()
    words = re.split(r"[^a-z]+", text)
    return [STEMMER.stem(w) for w in words if w and w not in STOP_WORDS]

That last point isn’t cosmetic: if query tokens and document tokens went through even slightly different preprocessing, cosine similarity between them wouldn’t be comparing the same vocabulary, and the whole ranking would be scoring noise.

Building one field’s index is standard vector-space TF-IDF — term frequency times inverse document frequency, an inverted index, and each document’s vector magnitude cached up front so scoring doesn’t recompute it per query:

def build_field_index(tokenized, n_docs):
    doc_freq = Counter()
    for tokens in tokenized.values():
        for term in set(tokens):
            doc_freq[term] += 1
    idf = {term: math.log(n_docs / df) for term, df in doc_freq.items()}

    inverted_index = defaultdict(dict)
    magnitude = {}
    for doc_id, tokens in tokenized.items():
        tf = term_frequency(tokens)
        vector = {term: weight * idf[term] for term, weight in tf.items()}
        magnitude[doc_id] = math.sqrt(sum(w * w for w in vector.values()))
        for term, weight in vector.items():
            inverted_index[term][doc_id] = weight
    return {"index": inverted_index, "magnitude": magnitude, "idf": idf}

The inverted index — term → {doc_id: weight} — is what keeps a query cheap: field_scores() only ever walks documents that share at least one term with the query, never the whole collection. Query terms with no entry in a field’s IDF table are dropped outright, rather than treated as zero-weight, since a term absent from the corpus vocabulary tells you nothing about relevance for that field.

Combining three cosine scores into one ranking

Scoring a query against one field is exactly a textbook vector-space search: turn the query into a TF-IDF vector using that field’s own IDF table, walk the inverted index to accumulate a dot product per candidate document, and divide by the two vector magnitudes to get cosine similarity — bounded to [0, 1] since none of the weights are negative.

def field_scores(field_index, tokens):
    qtf = term_frequency(tokens)
    query_vector = {term: weight * field_index["idf"][term]
                    for term, weight in qtf.items() if term in field_index["idf"]}
    magnitude_query = math.sqrt(sum(w * w for w in query_vector.values()))
    if magnitude_query == 0:
        return {}
    dot_products = defaultdict(float)
    for term, q_weight in query_vector.items():
        for doc_id, d_weight in field_index["index"].get(term, {}).items():
            dot_products[doc_id] += q_weight * d_weight
    return {doc_id: dot / (magnitude_query * field_index["magnitude"][doc_id])
            for doc_id, dot in dot_products.items() if field_index["magnitude"][doc_id] > 0}

The three per-field scores then combine with fixed weights that sum to 1, so the combined score stays on the same 0–1 cosine scale as any individual field:

combined = defaultdict(float)
for field, weight in FIELD_WEIGHTS.items():
    for doc_id, score in per_field[field].items():
        combined[doc_id] += weight * score

Title gets 0.5 because it’s short and written to state exactly what the publication is about — a query term landing there is the strongest single signal available. Keywords get 0.3: author-assigned and nearly as specific as title terms, but missing on roughly a third of the collection (63.4% coverage) and prone to generic labels. Abstract gets 0.2, the smallest share, precisely because it’s the longest field — a several-hundred-word abstract will contain almost any query term somewhere by chance, so a match there is the weakest per-term evidence of the three. I checked that ordering by hand against the test queries during development: swapping the title and abstract weights visibly let long abstracts outrank exact title matches.

The API keeps and returns all four numbers — combined score plus the three field scores — rather than collapsing to just the final rank, and the UI surfaces them per result so it’s visible why one publication outranked another instead of just trusting a single opaque number.

Evaluation: building ground truth that doesn’t cheat

There’s no official relevance judgment set for this collection, and judging by hand — reading the top 10 results for each query and marking them relevant — has two problems: it’s subjective, and it only ever judges the 10 documents the engine already chose to show, saying nothing about whether it missed something better sitting at position 11 or in the other 2,186.

Instead, evaluate.py applies one fixed rule to the entire collection, independent of the ranker being tested: stem the query’s terms, stem each publication’s title/abstract/keywords, and call a publication relevant once it shares at least 60% of the query’s distinct stemmed terms.

def relevant_urls(query):
    query_terms = set(clean_and_tokenize(query))
    threshold = max(1, math.ceil(len(query_terms) * QUERY_TERM_MATCH_FRACTION))
    relevant = set()
    for pub in db_publications.find({}, {"url": 1, "title": 1, "abstract": 1, "keywords": 1}):
        doc_terms = set(clean_and_tokenize(" ".join([pub.get("title", ""), pub.get("abstract", ""), " ".join(pub.get("keywords", []))])))
        if len(query_terms & doc_terms) >= threshold:
            relevant.add(pub["url"])
    return relevant

60% was chosen by testing both extremes first. Requiring every term (100%) sometimes found zero relevant documents for a genuine four-word query, since real publications rarely repeat every query word verbatim. Requiring just one shared term let something as generic as “health” alone mark a third of the collection relevant to every health-related query, which is clearly too loose to mean anything. 60% sits between those failure modes.

Five test queries were chosen to span the department’s actual subject areas — nursing, physical activity in older adults, health inequalities, mental health, and one narrow clinical topic (bariatric surgery) — rather than being picked to flatter the ranker.

Results

Query	Precision	Recall	F1	P@5	AP
community nursing intervention	0.50	0.037	0.069	0.80	0.925
physical activity older adults	1.00	0.110	0.198	1.00	1.000
health inequalities	0.70	0.304	0.424	0.80	0.826
mental health young people	0.70	0.115	0.197	1.00	0.962
bariatric surgery	1.00	0.400	0.571	1.00	1.000
Mean	0.78	0.193	0.292	0.92	MAP 0.943

Query response time, measured in-process across 25 calls: 1.51 ms mean, 1.54 ms median, 2.27 ms at the 95th percentile. That stays flat regardless of corpus size because a query only ever touches documents the inverted index lists against its own terms — never all 2,196 publications. The one cost that does scale with corpus size is rebuilding the whole index, about 2.8 seconds, which is exactly why that happens once at process start and then hourly in the background rather than inside a request.

Accuracy isn’t reported at all: with only a few dozen relevant documents out of 2,196 for any given query, true negatives dominate so completely that returning nothing at all would already score above 93%. It would be a meaningless number dressed up as a good one.

What the weak query and the perfect ones actually tell you

“Physical activity older adults” and “bariatric surgery” hit precision 1.00 and AP 1.00 because both use specific enough vocabulary that title, abstract and keyword scores all agree on the same handful of publications — there’s no ambiguity in that part of the corpus for those terms to exploit.

“Community nursing intervention” is the weak one, at precision 0.50. The five results the relevance rule rejects are genuinely about nursing — they just aren’t also about community care or a specific intervention. They share only one of the query’s three stemmed terms with a rejected document, but “nurs” alone carries enough TF-IDF weight on its own to still land in the top 10. A model that scored phrases rather than independent terms would very likely rank these lower, since it’s exactly the kind of failure a bag-of-words vector can’t distinguish: matching one strong term isn’t the same as matching the query’s actual meaning.

Two other misses expose something the ranker and the ground truth rule both share, since both are lexical, bag-of-words methods with no notion of meaning. A sports psychology paper about young footballers’ mental skills gets marked relevant to “mental health young people” purely because “mental,” “young” and “people” each appear somewhere in its text — not because it’s actually about the query’s subject. And for “health inequalities,” a paper on cross-country musculoskeletal disease trends that’s clearly on-topic gets marked irrelevant by the same rule, because it never happens to use the literal word “health.” That’s a vocabulary mismatch working against the evaluation itself, not just the ranker, and no amount of tuning the field weights fixes it — it needs something that reasons about meaning rather than counting shared strings.

Mean recall (0.193) looks low in isolation, but that’s a property of the evaluation setup, not the ranking: only the top 10 results per query are ever checked against every relevant document across the full 2,196-publication collection. A query with 134 relevant documents in the corpus (community nursing intervention) can retrieve 10 good results and still show single-digit recall by construction.

What I’d change

Field weights are hand-picked and fixed. They were sanity-checked against five queries during development, which is enough to catch an obviously wrong ordering but not enough to claim they’re optimal — a larger, less improvised relevance judgment set would let those weights actually be fit rather than eyeballed.

The ranker has no notion of phrases or term proximity, which is precisely what the “community nursing intervention” result exposes: matching “nurs” alone is treated identically to matching all three terms together. Bigram or phrase-aware scoring, even something simple like boosting documents where query terms appear adjacent, would likely fix that specific failure without touching the rest of the pipeline.

And the 60%-term-overlap relevance rule is itself lexical, so it inherits the same vocabulary-mismatch blind spot as the engine it’s grading — the musculoskeletal-disease paper marked irrelevant to “health inequalities” is a ground-truth error, not a ranking error, and no amount of retuning search_publications() will fix it. Any lexical evaluation has this ceiling built in; getting past it needs relevance judgments that don’t require the ground truth and the system under test to share the same weakness.