Rustam Niraula — Writing

Building a Vertical Search Engine Over 2,196 University Publications

Mon, 24 Aug 2026 00:00:00 GMT

An Information Retrieval assignment asked for a working search engine over a real, live website — not a toy corpus. I picked Coventry University’s Centre for Healthcare and Community Transformation (CHCT) page on PurePortal, the university’s research database, and built a vertical search engine over its publications: crawler, TF-IDF index, cosine ranking, and a small web UI, no search library involved anywhere.

The finished crawl holds 2,196 publications from 117 department members. Source: github.com/rustamniraula90/ir-assignment (search-engine/).

Three things about this turned out to matter more than the TF-IDF math itself: PurePortal sits behind Cloudflare, so getting past that shaped the whole crawler design; a page cap that isn’t documented anywhere silently truncated an early version of the crawl; and building the ground truth for evaluation by hand made the limits of a bag-of-words ranker completely explicit rather than something I’d just have to take on faith.

Architecture: three modules, MongoDB in between#

The whole system is a three-stage pipeline with MongoDB as the handoff point between stages, deliberately kept as separate processes rather than one monolith:

PurePortal (live site) → crawler.py → MongoDB → indexer.py → in-memory TF-IDF index → api.py → browser

crawler.py — a long-running, resumable crawler. It never touches a request path; it just keeps MongoDB’s publications collection current.
indexer.py — pure indexing and scoring logic. No web framework, no I/O beyond reading MongoDB. Given a query string, it returns ranked results.
api.py — a FastAPI app that builds the index once at startup, serves a search page and a JSON endpoint, and rebuilds the index from MongoDB every hour in the background.

MongoDB holds two collections: urls, which doubles as the crawl frontier and queue, and publications, the extracted records. The TF-IDF index itself is never persisted — it’s rebuilt in memory from publications whenever the API starts or its hourly refresh fires. That split matters: the crawler can be killed, restarted, or left running for days without the search API caring, and the API can be redeployed without re-crawling anything.

Seeding from the wrong page silently caps a crawl at 75 results#

The obvious way to crawl a department’s publications is to start from the organisation’s own aggregated publications listing and follow its pagination. That’s what the first version of the crawler did, and it always stopped at exactly 75 results — regardless of how many publications the department’s members actually had between them. That number turned out to be a hard cap on that listing page itself, not a bug in the pagination logic, and it’s not documented anywhere on the site.

A single prolific staff member can have several hundred publications, so 75 total was obviously wrong. The fix was to change what the crawl seeds from: the organisation’s persons listing instead, then each member’s own paginated /publications/ page individually. That gives three distinct URL types, which the crawl queue prioritizes in this order so the full member list is discovered early:

PRIORITY = {
    "profiles": 0,             # paginated list of department members
    "profile_publications": 1, # one member's paginated publication list
    "publication": 2,          # one publication's detail page
}

extract_content() dispatches on this type with a match statement, and each case both parses the current page and enqueues whatever it finds next — member profile pages enqueue that member’s publication listing, publication listings enqueue individual publication pages (plus their own next page), and publication pages just get scraped for title, abstract, keywords, authors and year.

The crawl frontier lives in MongoDB, not in memory#

urls isn’t a log of what’s been crawled — it is the frontier. Every URL document carries a next_crawl_date, a priority, and a retries counter:

def get_next_job():
    return db_urls.find_one_and_update(
        {"next_crawl_date": {"$lt": datetime.now(timezone.utc)}},
        {"$set": {"next_crawl_date": datetime.now(timezone.utc) + timedelta(minutes=10)}},
        sort=[("priority", 1), ("next_crawl_date", 1)],
    )

find_one_and_update claims a job atomically and pushes its next_crawl_date forward by ten minutes as a claim window, so the crawler can be killed and restarted without two runs ever grabbing the same URL, and without losing the queue’s position. On success, mark_success() pushes next_crawl_date out by seven days rather than deleting the record — the same document gets recrawled on a weekly cadence for as long as the crawler keeps running, which reproduces a scheduled recrawl without cron or any external scheduler. On failure, schedule_retry() backs off by an hour, capped at five retries, before giving up until the normal weekly slot — a transient network blip costs an hour, not a week.

Getting past Cloudflare without driving a browser for every page#

PurePortal sits behind Cloudflare’s Turnstile challenge, which a plain requests.Session cannot pass. Running every single page through a full browser would work but is slow — a Selenium round trip per publication, times 2,196 pages, adds up fast for a resource nobody but a crawl loop is going to see rendered.

The approach instead: open one persistent, undetected browser session (via seleniumbase’s uc_open_with_reconnect + uc_gui_click_captcha) at the start of a crawl run, solve the challenge once, then copy the resulting cf_clearance cookie into a plain requests.Session that handles every page after that:

def fetch_page(sb, url, max_challenge_retries=2):
    response = session.get(url, timeout=30)
    if response.status_code == 200 and not is_challenge_page(response.text):
        return response.text
    # ...fall back to the browser, re-solve, copy cookies back into `session`

is_challenge_page() checks the response for a <title> of “Just a moment…” or a #challenge-running element, since a 200 status alone doesn’t mean the response is real content — Cloudflare serves the interstitial with a 200 too. Most pages never touch the browser at all; it only comes back into play if the session cookie has expired or a fresh challenge appears.

Two failure modes get handled at different levels. An ordinary bad page — a timeout, a malformed response — is caught inside process_job() and goes through the normal schedule_retry() backoff. A dead browser session is a different, fatal failure (BrowserDiedError, raised on any WebDriverException or connection error while the browser is active), and it’s handled one level up, in crawl(): the whole browser session is torn down and relaunched from scratch after a ten-second pause, and the crawl loop resumes from whatever MongoDB’s queue currently holds — no in-memory state to lose.

robots.txt, and choosing availability over strict compliance#

The crawler fetches and parses robots.txt once per crawl run and checks every candidate URL against it before fetching:

def can_fetch(rp, url):
    if rp is None:
        return True
    return rp.can_fetch("*", url)

If that initial fetch fails, rp is None, and can_fetch() degrades to allowing everything rather than halting the crawl. get_crawl_delay() degrades the same way, falling back to a fixed five-second delay if robots.txt didn’t specify one. That’s a deliberate trade: a single failed robots.txt fetch shouldn’t take down an otherwise-working crawl, and the crawl rate stays capped by the fallback delay either way, so the politeness contract with the site doesn’t actually go away — it just falls back to a sane default instead of blocking on a request that failed.

Per-field TF-IDF, not one vector per document#

indexer.py builds three completely separate TF-IDF indexes — title, abstract, keywords — each with its own vocabulary and its own IDF table, rather than concatenating all the text into one vector per publication:

FIELD_WEIGHTS = {"title": 0.5, "keywords": 0.3, "abstract": 0.2}

The reason for keeping them separate: a single combined vector scores a query term appearing in the title exactly the same as the same term buried somewhere in a 300-word abstract, which throws away the strongest positional signal a publication record has. Across the full 2,196-publication corpus this produces three vocabularies of very different sizes — 3,284 terms for title, 3,065 for keywords, 10,350 for abstract — which on its own says something about how much more lexical variety a free-text abstract carries versus a controlled, short title.

Preprocessing is one shared pipeline, applied identically to indexed text and incoming queries — lowercase, split on non-letters, drop English stopwords, stem with Porter:

def clean_and_tokenize(text):
    text = text.lower()
    words = re.split(r"[^a-z]+", text)
    return [STEMMER.stem(w) for w in words if w and w not in STOP_WORDS]

That last point isn’t cosmetic: if query tokens and document tokens went through even slightly different preprocessing, cosine similarity between them wouldn’t be comparing the same vocabulary, and the whole ranking would be scoring noise.

Building one field’s index is standard vector-space TF-IDF — term frequency times inverse document frequency, an inverted index, and each document’s vector magnitude cached up front so scoring doesn’t recompute it per query:

def build_field_index(tokenized, n_docs):
    doc_freq = Counter()
    for tokens in tokenized.values():
        for term in set(tokens):
            doc_freq[term] += 1
    idf = {term: math.log(n_docs / df) for term, df in doc_freq.items()}

    inverted_index = defaultdict(dict)
    magnitude = {}
    for doc_id, tokens in tokenized.items():
        tf = term_frequency(tokens)
        vector = {term: weight * idf[term] for term, weight in tf.items()}
        magnitude[doc_id] = math.sqrt(sum(w * w for w in vector.values()))
        for term, weight in vector.items():
            inverted_index[term][doc_id] = weight
    return {"index": inverted_index, "magnitude": magnitude, "idf": idf}

The inverted index — term → {doc_id: weight} — is what keeps a query cheap: field_scores() only ever walks documents that share at least one term with the query, never the whole collection. Query terms with no entry in a field’s IDF table are dropped outright, rather than treated as zero-weight, since a term absent from the corpus vocabulary tells you nothing about relevance for that field.

Combining three cosine scores into one ranking#

Scoring a query against one field is exactly a textbook vector-space search: turn the query into a TF-IDF vector using that field’s own IDF table, walk the inverted index to accumulate a dot product per candidate document, and divide by the two vector magnitudes to get cosine similarity — bounded to [0, 1] since none of the weights are negative.

def field_scores(field_index, tokens):
    qtf = term_frequency(tokens)
    query_vector = {term: weight * field_index["idf"][term]
                    for term, weight in qtf.items() if term in field_index["idf"]}
    magnitude_query = math.sqrt(sum(w * w for w in query_vector.values()))
    if magnitude_query == 0:
        return {}
    dot_products = defaultdict(float)
    for term, q_weight in query_vector.items():
        for doc_id, d_weight in field_index["index"].get(term, {}).items():
            dot_products[doc_id] += q_weight * d_weight
    return {doc_id: dot / (magnitude_query * field_index["magnitude"][doc_id])
            for doc_id, dot in dot_products.items() if field_index["magnitude"][doc_id] > 0}

The three per-field scores then combine with fixed weights that sum to 1, so the combined score stays on the same 0–1 cosine scale as any individual field:

combined = defaultdict(float)
for field, weight in FIELD_WEIGHTS.items():
    for doc_id, score in per_field[field].items():
        combined[doc_id] += weight * score

Title gets 0.5 because it’s short and written to state exactly what the publication is about — a query term landing there is the strongest single signal available. Keywords get 0.3: author-assigned and nearly as specific as title terms, but missing on roughly a third of the collection (63.4% coverage) and prone to generic labels. Abstract gets 0.2, the smallest share, precisely because it’s the longest field — a several-hundred-word abstract will contain almost any query term somewhere by chance, so a match there is the weakest per-term evidence of the three. I checked that ordering by hand against the test queries during development: swapping the title and abstract weights visibly let long abstracts outrank exact title matches.

The API keeps and returns all four numbers — combined score plus the three field scores — rather than collapsing to just the final rank, and the UI surfaces them per result so it’s visible why one publication outranked another instead of just trusting a single opaque number.

Evaluation: building ground truth that doesn’t cheat#

There’s no official relevance judgment set for this collection, and judging by hand — reading the top 10 results for each query and marking them relevant — has two problems: it’s subjective, and it only ever judges the 10 documents the engine already chose to show, saying nothing about whether it missed something better sitting at position 11 or in the other 2,186.

Instead, evaluate.py applies one fixed rule to the entire collection, independent of the ranker being tested: stem the query’s terms, stem each publication’s title/abstract/keywords, and call a publication relevant once it shares at least 60% of the query’s distinct stemmed terms.

def relevant_urls(query):
    query_terms = set(clean_and_tokenize(query))
    threshold = max(1, math.ceil(len(query_terms) * QUERY_TERM_MATCH_FRACTION))
    relevant = set()
    for pub in db_publications.find({}, {"url": 1, "title": 1, "abstract": 1, "keywords": 1}):
        doc_terms = set(clean_and_tokenize(" ".join([pub.get("title", ""), pub.get("abstract", ""), " ".join(pub.get("keywords", []))])))
        if len(query_terms & doc_terms) >= threshold:
            relevant.add(pub["url"])
    return relevant

60% was chosen by testing both extremes first. Requiring every term (100%) sometimes found zero relevant documents for a genuine four-word query, since real publications rarely repeat every query word verbatim. Requiring just one shared term let something as generic as “health” alone mark a third of the collection relevant to every health-related query, which is clearly too loose to mean anything. 60% sits between those failure modes.

Five test queries were chosen to span the department’s actual subject areas — nursing, physical activity in older adults, health inequalities, mental health, and one narrow clinical topic (bariatric surgery) — rather than being picked to flatter the ranker.

Results#

Query	Precision	Recall	F1	P@5	AP
community nursing intervention	0.50	0.037	0.069	0.80	0.925
physical activity older adults	1.00	0.110	0.198	1.00	1.000
health inequalities	0.70	0.304	0.424	0.80	0.826
mental health young people	0.70	0.115	0.197	1.00	0.962
bariatric surgery	1.00	0.400	0.571	1.00	1.000
Mean	0.78	0.193	0.292	0.92	MAP 0.943

Query response time, measured in-process across 25 calls: 1.51 ms mean, 1.54 ms median, 2.27 ms at the 95th percentile. That stays flat regardless of corpus size because a query only ever touches documents the inverted index lists against its own terms — never all 2,196 publications. The one cost that does scale with corpus size is rebuilding the whole index, about 2.8 seconds, which is exactly why that happens once at process start and then hourly in the background rather than inside a request.

Accuracy isn’t reported at all: with only a few dozen relevant documents out of 2,196 for any given query, true negatives dominate so completely that returning nothing at all would already score above 93%. It would be a meaningless number dressed up as a good one.

What the weak query and the perfect ones actually tell you#

“Physical activity older adults” and “bariatric surgery” hit precision 1.00 and AP 1.00 because both use specific enough vocabulary that title, abstract and keyword scores all agree on the same handful of publications — there’s no ambiguity in that part of the corpus for those terms to exploit.

“Community nursing intervention” is the weak one, at precision 0.50. The five results the relevance rule rejects are genuinely about nursing — they just aren’t also about community care or a specific intervention. They share only one of the query’s three stemmed terms with a rejected document, but “nurs” alone carries enough TF-IDF weight on its own to still land in the top 10. A model that scored phrases rather than independent terms would very likely rank these lower, since it’s exactly the kind of failure a bag-of-words vector can’t distinguish: matching one strong term isn’t the same as matching the query’s actual meaning.

Two other misses expose something the ranker and the ground truth rule both share, since both are lexical, bag-of-words methods with no notion of meaning. A sports psychology paper about young footballers’ mental skills gets marked relevant to “mental health young people” purely because “mental,” “young” and “people” each appear somewhere in its text — not because it’s actually about the query’s subject. And for “health inequalities,” a paper on cross-country musculoskeletal disease trends that’s clearly on-topic gets marked irrelevant by the same rule, because it never happens to use the literal word “health.” That’s a vocabulary mismatch working against the evaluation itself, not just the ranker, and no amount of tuning the field weights fixes it — it needs something that reasons about meaning rather than counting shared strings.

Mean recall (0.193) looks low in isolation, but that’s a property of the evaluation setup, not the ranking: only the top 10 results per query are ever checked against every relevant document across the full 2,196-publication collection. A query with 134 relevant documents in the corpus (community nursing intervention) can retrieve 10 good results and still show single-digit recall by construction.

What I’d change#

Field weights are hand-picked and fixed. They were sanity-checked against five queries during development, which is enough to catch an obviously wrong ordering but not enough to claim they’re optimal — a larger, less improvised relevance judgment set would let those weights actually be fit rather than eyeballed.

The ranker has no notion of phrases or term proximity, which is precisely what the “community nursing intervention” result exposes: matching “nurs” alone is treated identically to matching all three terms together. Bigram or phrase-aware scoring, even something simple like boosting documents where query terms appear adjacent, would likely fix that specific failure without touching the rest of the pipeline.

And the 60%-term-overlap relevance rule is itself lexical, so it inherits the same vocabulary-mismatch blind spot as the engine it’s grading — the musculoskeletal-disease paper marked irrelevant to “health inequalities” is a ground-truth error, not a ranking error, and no amount of retuning search_publications() will fix it. Any lexical evaluation has this ceiling built in; getting past it needs relevance judgments that don’t require the ground truth and the system under test to share the same weakness.

Rebuilding My Portfolio: From a Static Single Page to Astro, Pagefind, and Cloudflare

Tue, 11 Aug 2026 00:00:00 GMT

For two years the portfolio was about as simple as a site gets: one index.html, one 611-line style.css, one src/main.js, and Vite to bundle it. That was the entire repository. It was fast, cost nothing to run, and never broke.

It also couldn’t hold an article. Every position on that page was hand-placed markup, so “add a blog” meant hand-writing another page, and another, forever. Three things I wanted instead: write in Markdown and git push, get real search without paying for a search API, and deploy without a step in between.

This is the move, and what got built on top of it afterwards.

The rules first#

I wrote the constraints down before picking anything, because the tools are the easy half.

Static output. Pages ship as HTML. JavaScript where something genuinely needs to be interactive.
Markdown in, site out. A post is a file. No CMS, no database, no admin panel.
Search with no backend. No Algolia, no Elasticsearch, no per-query cost. $0/month has to stay $0/month.
Don’t lose what worked. The hero canvas, the Open Graph tags, the Person JSON-LD, the analytics snippet. Those were the parts of the old file with accumulated value in them.

Concern	Choice
Build	Astro 7, static output
Markdown + components	`@astrojs/mdx`
Search	Pagefind, post-build
Social cards	Satori → resvg, at build time
Hosting	Cloudflare Workers static assets, built on push

Astro renders components to HTML and only hydrates what you explicitly ask it to, which is constraint 1 for free. Pagefind is the interesting one: it doesn’t index source files or ship a client-side index like Fuse.js. It runs after the build, reads the HTML Astro just emitted, and writes a chunked index. The browser downloads only the fragments a query touches.

Before and after#

.
├── index.html
├── package.json
├── src
│   └── main.js
├── style.css
└── wrangler.jsonc

.
├── astro.config.mjs
├── worker/
│   └── index.ts                  # /api/contact, and the asset server
└── src/
    ├── content.config.ts          # the schema every post is checked against
    ├── content/blog/              # posts, as Markdown
    ├── data/                      # site, categories, contact rules
    ├── lib/                       # post helpers, OG card renderer
    ├── components/
    ├── layouts/BaseLayout.astro   # <head>, SEO, schema
    ├── pages/
    │   ├── index.astro            # home
    │   ├── 404.astro
    │   ├── rss.xml.ts
    │   ├── sitemap.xml.ts
    │   ├── og/[slug].png.ts       # one social card per post
    │   └── blog/
    │       ├── index.astro
    │       ├── [slug].astro
    │       └── category/[category]/index.astro
    └── styles/global.css          # tokens in, components below

No top-level style.css and no main.js. The stylesheet moved in, and main.js came apart into whatever component owned each behaviour.

Typed frontmatter#

Content collections are what make a folder of Markdown behave like a database.

const blog = defineCollection({
  loader: glob({ pattern: "**/*.{md,mdx}", base: "./src/content/blog" }),
  schema: z.object({
    title: z.string(),
    description: z.string(),
    pubDate: z.date(),
    category: z.enum(categorySlugs),
    tags: z.array(z.string()).optional(),
    featured: z.boolean().optional(),
    summary: z.string().optional(),
  }),
});

A typo in frontmatter is a build failure, not a blank spot on a deployed page. category is an enum over a hand-maintained list, so catagory: systems stops the deploy rather than producing a post filed under nothing.

One layout owning the shell#

BaseLayout.astro holds what has to be true on every page: <head>, header, footer, and the metadata the old index.html had collected. That last part is the one worth being careful about — Open Graph tags, the Person JSON-LD, the analytics snippet, all sitting in a file I was about to delete, which is exactly when that sort of thing disappears without anyone noticing.

const canonical = new URL(Astro.url.pathname, Astro.site ?? Astro.url).href;

Canonical and social URLs are derived rather than pasted. isHome scopes the Person schema to one page instead of repeating it everywhere. Posts add BlogPosting and a BreadcrumbList; the breadcrumb is passed in as props, so the template doesn’t have to know where in the hierarchy it sits.

Fonts without a network#

Three families, all self-hosted, resolved at build time from the @fontsource packages:

fonts: [
  {
    name: "Fraunces",
    cssVariable: "--font-fraunces",
    provider: fontProviders.local(),
    fallbacks: ["Georgia", "ui-serif", "serif"],
  },
],

fontProviders.local() means npm run build never reaches out to Google, and Astro emits a metric-matched fallback face so the first paint uses something the right shape rather than Times. Fraunces carries the optical-size axis, so headings pick up the high-contrast display cut as they grow without a second file.

The hero canvas#

Worth a section because the first version was quietly broken.

The original drew a random walk mirrored across four quadrants, accumulating into a symmetric lattice. It ran on setInterval(fn, 10) — roughly 100fps, forever, redrawing the whole canvas every 10ms whether or not anyone was looking. A mousemove listener called the resize handler, which reset the walk position, so moving the mouse snapped the pattern. No prefers-reduced-motion, no devicePixelRatio handling.

The drawing is now a flow field instead. Around seventy particles drift through a field built from three offset sine waves, and the trail is decayed with a 1.7% white wash per frame rather than clearRect. That wash is the whole trick — it fades old segments toward the paper while new ones draw on top, which is what makes it read as long exposure instead of scribble.

const angleAt = (x, y) =>
  Math.sin(x * 0.0021 + t * 0.21) +
  Math.cos(y * 0.0027 - t * 0.16) +
  Math.sin((x + y) * 0.0013 + t * 0.11);

Seventy particles and k=3 nearest neighbours is a few thousand float operations a frame. A library would have cost more in bytes than the graph does.

The lifecycle is where the previous version was failing: one requestAnimationFrame loop, paused when the tab is hidden or the hero scrolls out of view, DPR capped at 2, and a single settled frame under prefers-reduced-motion.

Categories instead of tag soup#

The first blog index derived its sidebar from whatever tags happened to be in use, with counts. Nine posts produced twenty-two categories, most with one item, which is a filter nobody can use.

It’s a hand-maintained list now, with a rule that a category waits for roughly three posts before it earns a page:

export const categories = [
  { slug: "systems", name: "Systems & Infrastructure", icon: "hub", description: "..." },
  { slug: "algorithms", name: "Algorithms", icon: "account_tree", description: "..." },
  { slug: "machine-learning", name: "Machine Learning & Statistics", icon: "psychology", description: "..." },
] as const;

as const plus z.enum means adding a category is one edit in one file and a typo is a build error. The filter is plain links to /blog/category/<slug>/ — no client-side filtering, so it works with JavaScript off and every filtered view is shareable. I had a “Load More” button too, which went: with nine posts it hides content from crawlers and from Ctrl-F, and client-side pagination is the wrong tool at this size.

Code blocks that know what they are#

Nine posts, 102 fenced blocks. Two things I wanted that weren’t there.

Filenames. A fence can carry a title, and a Shiki transformer lifts it onto the <pre>, so the header bar above a block can name the file:

```go title="crawler.py"

One trap: the fence has to declare a language. Shiki reads a bare ``` title="x" as the language name, swallows the title, and data-language degrades to plaintext. Cost me one block and about ten minutes.

Copy buttons and line highlighting are a small script over the rendered <pre> — posts are Markdown, so this enhances the output rather than wrapping it in a component. The overflow fade uses a token rather than a literal, because its position has to track the header height above it and a magic number there silently desyncs the moment the padding changes.

Search#

Two markers do the work:

<article data-pagefind-body>
  <h1 data-pagefind-meta="title">{post.data.title}</h1>

Found a data-pagefind-body element on the site.
↳ Ignoring pages without this tag.

Once any page declares it, Pagefind indexes only pages that have it, and only that element’s subtree. The home page and the feed drop out automatically; searching returns articles rather than fragments of my own navigation. The table of contents carries data-pagefind-ignore because the heading text is already inside the body and indexing it twice just skews relevance.

The UI is Pagefind’s Component UI — custom elements, so mounting is markup and a module script:

<pagefind-input instance="blog" autofocus="false"></pagefind-input>
<pagefind-summary instance="blog"></pagefind-summary>
<pagefind-results instance="blog"></pagefind-results>

Elements sharing an instance attribute find each other, which is what lets the input, the count and the results sit independently in the layout. Two details that cost time: the script needs type="module", and in Astro it also needs is:inline, because /pagefind/ only exists in build output and the bundler has nothing to resolve at compile time. Custom elements don’t upgrade until their definition loads, so astro dev renders an inert element rather than an error — a :not(:defined) rule reserves the input’s height so nothing below it jumps when it paints.

/ and ⌘K focus the field. The deep link is ?q=, which is also the SearchAction target in the WebSite schema.

Social cards, generated#

Every post gets a 1200×630 card rendered at build time by Satori and rasterised with resvg, from a handful of frontmatter fields.

export const GET: APIRoute<{ post?: Post }> = async ({ props }) => {
  const { post } = props;
  return pngResponse(
    await renderCard({
      kicker: `Writing / ${getCategory(post.data.category).name}`,
      title: post.data.title,
      footerLeft: site.name,
      footerRight: isoDate(post.data.pubDate),
    })
  );
};

Satori reads WOFF and TTF but not WOFF2, so the cards load the static @fontsource faces while the site itself serves the variable WOFF2 files. Same fonts, different packaging.

Previously og:image pointed at the app icon, so every link I’d ever shared rendered as a generic badge. That was the single worst bug on the site and it took two years to notice.

The contact form and its Worker#

The static site is served by a Worker, and Cloudflare only invokes the script when a request doesn’t match a static asset — so almost nothing reaches it. One route is handled: POST /api/contact.

if (typeof name !== "string" ||
    typeof email !== "string" ||
    typeof message !== "string") {
  return { status: "invalid", code: 400 };
}

The rules that validate the request live in src/data/contact.ts, which the Worker imports and the form imports. One module, so the length limits and the email pattern can’t drift apart between the two — which is the failure mode of every other form I’ve written, where the browser check and the server check disagree and one of them is wrong.

It accepts JSON and form-encoded bodies. With JavaScript off the form posts normally and the Worker answers with a 303 to a static page under /contact/<status>/, so a failed submission still shows a real result. With JavaScript on, it’s a fetch and a status toast.

Deploy#

The repository is connected to the Worker. git push to the default branch is the deploy: Cloudflare clones, runs npm run build, publishes. Nothing on my machine, no CI provider in front of it.

{
  "assets": { "directory": "./dist" },
  "compatibility_flags": ["nodejs_compat"],
  "observability": { "enabled": true }
}

assets.directory is the contract between the two halves — Astro’s output is what gets uploaded and served.

Publishing a post is now: add a file with valid frontmatter, push. Astro validates it against the schema and renders every route, Pagefind indexes the output, Cloudflare serves it. Because validation happens in the build, a malformed post fails before it reaches the edge and the previous deployment stays up. That’s what makes push-to-publish feel ordinary rather than nerve-wracking.

One thing I removed on purpose#

The redesign had a dark mode. It looked fine in isolation — a full second set of tokens under prefers-color-scheme: dark, background at #14161a, and the hero canvas re-reading its stroke colour when the scheme flipped. I cut the whole branch.

The reason is that white here isn’t the absence of dark mode, it’s the material. Sections separate by a change of tone rather than by a card edge, and the canvas fades toward #ffffff with a rgba(255, 255, 255, 0.006) wash per frame to get the long-exposure trail — both of those decisions assume a light ground. A dark variant isn’t a reskin of that, it’s a second site. And the only people who saw mine were the ones whose OS asked for it, which meant I was reviewing my work in one material and shipping another.

So: one theme, white, regardless of preference. The exception is the contact band, which is dark on every device:

/* Single theme. There is no `prefers-color-scheme` branch: the site is
   white on a dark-preference OS, the same as it is on a light one. The
   hero canvas no longer re-reads its stroke on a theme flip either. */

.band-ink isn’t a theme either. It re-declares the token block inside its own scope, so bg-surface, text-on-surface and friends keep working inside the band and no component has to know it landed there. Cheaper to reason about than a data-theme attribute that half the components would have had to check, and it gets color-scheme: dark for free — which is what stops the browser rendering a light scrollbar and light form controls in the middle of the page.

No database, no server, no search service. Twenty HTML pages, an index directory, and a Worker that only wakes up when someone fills in the contact form.

The structural migration was the whole job and it was a straightforward one. Astro gave me typed content and static output, Pagefind gave me search without a backend, Cloudflare gave me deploys without a pipeline. Nothing about it needed a rewrite of how the site worked — and the parts that did get rewritten later taught me more than the parts that didn’t.

Model Selection on EEG Signals: Least Squares, AIC/BIC and Rejection ABC by Hand

Thu, 16 Jul 2026 00:00:00 GMT

An MSc statistics assignment handed me 201 samples of simulated EEG data: four input channels x1–x4, one output channel y, sampled every 2 ms over 0.4 seconds. Five candidate polynomial regression architectures were given. Pick the right one, justify the choice, validate it, then estimate the two most influential parameters with Approximate Bayesian Computation.

The interesting constraint was that everything had to be implemented from the mathematics — no lm(), no AIC(), no abc package. Normal equations, log-likelihood, information criteria and rejection sampling all written out in base R matrix algebra. Source and rendered notebook: github.com/rustamniraula90/eeg-signal-analysis.

That constraint turned out to matter. Twice, writing a formula out by hand made visible something a library call would have handled silently — and in both cases the pipeline produced numbers that looked fine but weren’t. Those are the parts of this worth reading.

The data, before any model#

Before fitting anything, four questions about the data are worth answering, because each one rules something in or out later.

Are the inputs independent? They are emphatically not:

      x1    x2    x3    x4     y
x1 1.000 0.259 0.584 0.805 0.857
x2 0.259 1.000 0.523 0.364 0.194
x3 0.584 0.523 1.000 0.708 0.526
x4 0.805 0.364 0.708 1.000 0.775
y  0.857 0.194 0.526 0.775 1.000

x1 and x4 correlate at 0.805 with each other, and 0.857 / 0.775 with the output. x2 is nearly useless linearly (0.194 with y). This predicts something concrete: any model containing both x1 and x4 terms will have a nearly singular XᵀX, and their individual coefficients will be unstable even if the model’s predictions are fine. Collinearity attacks interpretation before it attacks fit.

Is anything non-Gaussian? The per-channel skewness says yes, and only in one place. The inputs are all near-symmetric (|skew| ≤ 0.32), but the output has skewness −1.79 — a long left tail, driven by a minimum of −9.60 against a median of 0.41. The inputs are drawn symmetric; the output isn’t. Something in the generating process is asymmetric, and a symmetric polynomial in the inputs can only reproduce that through odd powers.

What shape are the relationships? Loess curves through y against each input come back visibly bent, not straight. That’s the whole justification for a polynomial basis rather than plain multiple linear regression.

How much data is there? n = 201, and the candidate models want 3–6 parameters. That ratio is comfortable, but it’s also why BIC (whose penalty is k·log n = 5.30k here) will bite noticeably harder than AIC (2k).

Least squares written out is a conditioning problem#

The estimator is one line of linear algebra:

θ̂ = (Xᵀ X)⁻¹ Xᵀ y

least_squares <- function(Xmat, yvec) {
  XtX <- t(Xmat) %*% Xmat
  theta_hat <- solve(XtX) %*% t(Xmat) %*% yvec
  return(theta_hat)
}

That’s the textbook form, and calling it is where the polynomial basis starts to hurt. The design matrix for Model 1 holds a column of x4 (range ±8.7) next to a column of x1⁴ (range up to 5,800). Squaring the matrix to form XᵀX squares that disparity too. The condition number of XᵀX:

Model	κ(XᵀX)
Model 1 (with `x1²`, `x1³`, `x1⁴`, `x2⁴`)	2.8 × 10⁶
Model 2 (with `x4`, `x1³`, `x3⁴`)	7.4 × 10⁴

Double precision carries about 16 significant digits, so at κ ≈ 10⁶ you’re spending six of them on the inversion. Model 1 survives it. A model with x1⁵ or x1⁶ in the same basis would not, and solve() would return numbers rather than an error.

What I’d change. solve(t(X) %*% X) is the formula, not the algorithm. qr.solve(X, y) reaches the same estimate through a QR decomposition without ever forming XᵀX, so it operates at κ(X) rather than κ(X)² — roughly half the digits lost. Writing the normal equations out by hand is the right call for an assignment about understanding least squares; it’s the wrong call for one about computing it. That the two differ, and why, was the first thing this exercise taught me.

RSS ranks fit; it does not rank models#

With five architectures fitted on the full dataset:

Model	Terms	k	RSS
Model 1	`x4, x1², x1³, x2⁴, x1⁴`	6	35.397
Model 2	`x4, x1³, x3⁴`	4	2.140
Model 3	`x3³, x3⁴`	3	463.312
Model 4	`x2, x1³, x3⁴`	4	20.259
Model 5	`x4, x1², x1³, x3⁴`	5	2.136

The first thing to notice looks like a bug: Model 1 has the most parameters and nearly the worst fit. Six parameters, RSS 35.4 — sixteen times worse than Model 2’s four parameters.

It isn’t a bug, and understanding why sharpened my intuition more than the rest of the fitting did. “More parameters never increase RSS” is true within a nested family — where the bigger model can always reproduce the smaller one by zeroing a coefficient. These five models are not nested. Model 1 contains x2⁴ and x1⁴ but has no x3⁴ term, and Model 2 has one. There is no setting of Model 1’s six coefficients that reproduces Model 2. The count of parameters is irrelevant; what matters is whether the span of the columns contains the truth.

And x3⁴ is clearly part of the truth. Every model containing it lands at RSS ≈ 2; every model without it lands between 20 and 463. Model 3, which has x3³ and x3⁴ but nothing else, is the worst of the five — so x3⁴ is necessary but nowhere near sufficient. The signal needs a term from x1 as well.

The comparison that actually decides the assignment is Model 2 against Model 5, because those two are nested: Model 5 is Model 2 plus an x1² term. It fits better, as it must — but by 0.004 in RSS, a 0.2% improvement. And its estimated coefficient on x1² is 0.00033, which is indistinguishable from zero.

Log-likelihood, and the σ² that got assumed#

Turning RSS into a likelihood requires an assumption that RSS itself doesn’t make: that the errors are i.i.d. Gaussian. Under that assumption,

ln L = −(n/2)·ln(2π) − (n/2)·ln σ̂²  −  RSS / (2σ̂²)        where  σ̂² = RSS / (n − 1)

compute_loglik <- function(rss, n) {
  sigma2_hat <- rss / (n - 1)
  loglik <- -(n/2)*log(2*pi) - (n/2)*log(sigma2_hat) - (1/(2*sigma2_hat))*rss
  return(list(loglik = loglik, sigma2 = sigma2_hat))
}

Two things about that n - 1 are worth being explicit about, since it’s the specification I was given rather than the one I’d choose.

The unbiased residual variance for a regression divides by n - k, not n - 1, because fitting k coefficients already consumed k degrees of freedom. With n = 201 and k ≤ 6 the difference is under 3%, and — critically — it’s a monotone function of RSS applied identically to every model, so the AIC/BIC ranking is untouched. It’s the right shortcut for a comparison and the wrong one for reporting σ² as a quantity. Later in the pipeline, that distinction stops being cosmetic.

The Gaussian assumption is testable, and this is where the Q-Q plots earn their place in the pipeline rather than being a box to tick. For Model 2 the residuals sit on the reference line across the full range: skewness −0.04, excess kurtosis 0.15, Shapiro-Wilk W = 0.9975, p = 0.99. The likelihood is licensed. For Models 1, 3 and 4 the same plots bend away at both tails — their residuals still contain structure the model failed to absorb, which means their reported log-likelihoods are computed under an assumption their own residuals contradict.

That reframed the diagnostic for me. A Q-Q plot isn’t a check performed after selection. It’s a check on whether the quantity you’re selecting with is meaningful, and it should be read before the AIC table, not after.

AIC and BIC: the same answer, and a useful disagreement in margin#

AIC = 2k − 2·ln L          BIC = k·ln(n) − 2·ln L

Model	k	RSS	ln L	AIC	BIC
Model 2	4	2.140	171.32	−334.65	−321.44
Model 5	5	2.136	171.52	−333.05	−316.53
Model 4	4	20.259	−54.59	117.18	130.39
Model 1	6	35.397	−110.67	233.34	253.16
Model 3	3	463.312	−369.14	744.27	754.18

Both criteria pick Model 2, which is the answer I want — agreement between two criteria with different penalty philosophies is much stronger evidence than either alone.

The Model 2 / Model 5 pair is where the arithmetic becomes legible. Model 5’s extra x1² term buys +0.20 of log-likelihood. AIC charges 1 unit of log-likelihood per parameter, so the trade is 0.20 earned against 1.00 charged, and Model 5 loses by 1.60. BIC charges log(201)/2 = 2.65 per parameter, so it loses by 4.11 — the same verdict, 2.6× more emphatic.

This is the parsimony penalty doing exactly the job it exists for. On RSS alone Model 5 wins every time, and would keep winning as I bolted on more powers of x1, each one soaking up a little noise. The criteria convert “does this term buy more fit than it costs in flexibility?” into a number, and the answer for x1² is no. Its coefficient of 0.00033 was already whispering that; AIC and BIC just make it a decision rather than a judgement call.

So Model 2 is:

y = 0.1436·x4  +  0.01004·x1³  −  0.001913·x3⁴  +  0.4831

Reading the coefficients requires care. x3⁴’s coefficient is 75× smaller than x4’s, which says nothing about importance — the regressors live on wildly different scales. Multiply each coefficient by the standard deviation of its own column to get comparable contributions:

Term	\|θ\|	sd(column)	sd(θ · column)
`x4`	0.1436	3.15	0.452
`x1³`	0.01004	144.4	1.450
`x3⁴`	0.001913	232.9	0.446

x1³ is the dominant driver by a factor of three, despite having the second-smallest coefficient. It also explains the output’s −1.79 skewness: a cubic term is the odd-powered, asymmetry-producing part of the model, and −0.0019·x3⁴ pulls hard in one direction only. The structure the exploratory analysis flagged is visible in the fitted parameters. Hold on to this table — it comes back to bite in the ABC section.

Validation: 95% intervals that covered 80%#

Splitting 70/30 (140 train, 61 test), re-estimating on the training slice, and predicting the held-out points with 95% intervals:

pred_var <- sapply(seq_len(nrow(X_test)), function(i) {
  xi <- X_test[i, , drop = FALSE]
  sigma2_test * (1 + xi %*% XtX_inv_train %*% t(xi))
})

The (1 + xᵢ(XᵀX)⁻¹xᵢᵀ) factor is right, and worth pausing on: the 1 is the irreducible noise on a new observation and the quadratic form is the uncertainty in the fitted surface itself, which grows as you move away from the centre of the training data. Including both makes this a prediction interval rather than a confidence interval on the mean — the correct choice when the question is “where will the next observation fall.”

The training coefficients came back at (0.1404, 0.01017, −0.00188, 0.48595) against full-data (0.1436, 0.01004, −0.00191, 0.48307) — stable to three significant figures on 70% of the data, which is the real evidence that Model 2 isn’t overfitting. Validation RSS was 0.650 on 61 points, versus 2.140 on 201; per-point, 0.0107 against 0.0107. Out-of-sample error is identical to in-sample error.

And then the coverage came back at 80.3%.

That number is wrong in an interesting way. Under-coverage normally means overfitting — but the per-point RSS just ruled overfitting out. Something else was shrinking the intervals, and the culprit was the variance estimate:

sigma2_test <- rss_test / (nrow(X_train) - 1)   # 0.650 / 139

That divides the test RSS by the training sample count. It’s a units mismatch: 61 points’ worth of squared error spread across 140 degrees of freedom, giving σ² = 0.00468 where the model’s actual noise level is 0.0107 — a factor of 2.4 too small, so every interval is 1.56× too narrow. Rerunning with the denominators aligned:

σ̂² definition	σ̂²	Mean half-width	Coverage
`rss_test / (n_train − 1)`	0.00468	0.136	80.3%
`rss_test / (n_test − k)`	0.01141	0.213	93.4%
`rss_train / (n_train − k)`	0.01133	0.212	93.4%

93.4% against a nominal 95%, on 61 points — that’s four misses where you’d expect three, comfortably within binomial noise. The model was calibrated the whole time; the reported interval wasn’t.

This is the failure mode I’d never have found with predict(fit, interval = "prediction"), because that function’s whole job is knowing which n and which k belong in the denominator. Two symbols that both spell “sample size” got crossed, no error was raised, and the pipeline produced a plausible-looking plot of intervals that were half again too tight. The check that caught it was empirical coverage — counting how many held-out points actually landed inside — and it’s now the first thing I compute after any interval, precisely because it tests the arithmetic rather than trusting it.

Rejection ABC, and a tolerance that could never accept#

The last phase estimates a posterior over two parameters without writing down a likelihood. Rejection ABC is almost embarrassingly simple:

Draw candidate parameters from a prior.
Simulate a dataset from them.
Keep the draw if the simulated data is within ε of the observed data.
The kept draws approximate the posterior.

Step 3 is the entire method, and step 3 is where I got it wrong.

The obvious tolerance is a fraction of the reference fit — 5% of Model 2’s RSS, ε = 0.107. Run 20,000 simulations against that and zero are accepted. Not “few.” Zero.

Writing the simulator by hand is what showed me why, because the noise term is right there in the line:

y_sim <- X_full %*% theta_candidate + rnorm(n, mean = 0, sd = sqrt(sigma2_ref))

Every simulated dataset gets a fresh draw of noise at all 201 points. So even with candidate parameters exactly equal to the truth, the distance Σ(y_obs − y_sim)² is a sum of squared differences between two independent noise realisations. Its expectation is roughly n·σ² = 201 × 0.0107 = 2.15:

Naive tolerance (5% of RSS):                  0.107
Expected distance from noise injection alone: 2.150   (20.1× larger)
Smallest distance seen across 20,000 draws:   3.231

The tolerance was set 20× below the floor of the distance distribution. No parameter values could have cleared it, so the acceptance rate was zero for reasons that had nothing to do with parameters at all.

The fix is to stop guessing an absolute distance and let the simulations calibrate it — run all 20,000, then set ε to the 1st percentile of the observed distances:

acceptance_quantile <- 0.01
tolerance_epsilon <- as.numeric(quantile(distances, probs = acceptance_quantile))

ε lands at 4.239 and 200 draws are accepted, by construction. This is standard practice for rejection ABC and it’s better than a guess for a structural reason: the scale of the distance metric depends on n, on σ², and on how the simulator injects noise, none of which you know before running it. A quantile is scale-free. It also makes the honest trade explicit — ε and the acceptance rate are the same knob, and you’re choosing between posterior sharpness and effective sample size, not discovering a tolerance that’s objectively correct.

The generalisable lesson: when a rejection sampler accepts nothing, check the floor of your distance metric before touching the priors. I lost time assuming my prior windows were badly placed. They were fine. The threshold was below the minimum achievable value, and one line of arithmetic — n × σ² — would have said so immediately.

What the posterior says, and what selecting by magnitude cost me#

The assignment asks for the two “most influential” parameters, operationalised as the two largest by absolute value. For Model 2 that’s bias (0.4831) and x4 (0.1436), with x1³ and x3⁴ held fixed. Priors were uniform, centred on the least-squares estimates, ±50%.

The 200 accepted draws:

Parameter	LS estimate	Posterior mean	Prior sd	Posterior sd	Shrinkage
`bias`	0.48307	0.48291	0.1401	0.0238	5.9×
`x4`	0.14358	0.14328	0.0417	0.0081	5.1×

Posterior means recover the least-squares estimates to four significant figures. That agreement is the real result: a likelihood-free method that only ever simulates forward, never inverting anything, lands on the same answer as the closed-form estimator. When an analytic solution exists, ABC should reproduce it — and here it does, which is what makes it trustworthy on problems where no closed form is available.

Shrinkage of ~5-6× from prior to posterior is the data doing work; the joint posterior is a compact blob rather than the ridge that collinearity would produce, and the posterior correlation between the two parameters is −0.08, effectively independent. That last part makes sense: x4 is mean-centred (mean 0.015), so shifting the intercept barely trades off against its slope.

But the parameter selection is where I’d push back on the specification. Ranking by |θ| is ranking by coefficient magnitude, and coefficient magnitude is an artifact of the regressor’s units. From the contribution table earlier:

Term	\|θ\| rank	Contribution rank
`bias`	1	— (constant, no variance)
`x4`	2	2 (sd 0.452)
`x1³`	3	1 (sd 1.450)
`x3⁴`	4	3 (sd 0.446)

The |θ| criterion selected the intercept — which contributes no variance at all, being constant — and skipped x1³, the term that drives three times more variation in y than anything else in the model. The ABC posterior is over the two parameters that happen to be attached to the smallest-scale columns.

The estimates are still correct; the question they answer is just less interesting than the one I’d have chosen. Ranking by |θⱼ| · sd(xⱼ) would have put x1³ and x4 under the microscope — the two terms actually shaping the signal — and the resulting joint posterior would likely have shown the collinearity structure the exploratory correlation matrix predicted, since x1 and x4 correlate at 0.805. That’s the follow-up run worth doing, and it costs nothing but changing one order() call.

What’s worth keeping from this#

RSS ranks models only within a nested family. Across non-nested architectures, six parameters can fit sixteen times worse than four; what matters is whether the column span contains the truth, and here that meant containing x3⁴.
AIC and BIC agreeing is worth more than either alone, and the margin between them is informative — BIC’s log(n)/2 = 2.65 per parameter versus AIC’s 1.0 turned the same verdict on x1² into a 2.6× stronger one.
Q-Q plots belong before the AIC table, not after. They test whether the Gaussian likelihood you’re ranking with is licensed at all; three of five models here failed that test while still reporting a log-likelihood.
Coefficient magnitude is not importance. Multiply by the regressor’s standard deviation before ranking anything, or the intercept wins a contest it isn’t in.
Empirical coverage is the check that catches interval bugs. Nominal 95% covering 80.3% traced back to one crossed denominator — test RSS over training count — that no error message would ever have flagged.
When a rejection sampler accepts nothing, check the floor of your distance metric first. Re-simulating noise puts an n·σ² lower bound on the distance, and no tolerance below it can accept regardless of the parameters.
Writing least squares as solve(t(X) %*% X) teaches you the estimator and hides the conditioning; QR gets the same answer at half the digit loss. The formula and the algorithm are different things, and for polynomial bases that difference is measured in lost significant figures.

Drawing a Family Tree With Thousands of People: Server-Side Layout and Level-of-Detail Rendering

Wed, 08 Apr 2026 00:00:00 GMT

A family tree has an awkward property: it has no natural size limit. One family adds a few dozen people and stops. Another keeps going for eight generations and passes a thousand. The screen showing it, meanwhile, is always about 1400 by 900 pixels.

That mismatch is the whole engineering problem, and it splits into two questions that turn out to be independent:

Who decides where each person’s box goes — and how often does that get decided?
What does the client actually draw, given that almost none of the tree is on screen at any moment?

This is a journal of working both out for srishtiuniverse.com, across a Spring Boot backend, a React web client and a Flutter mobile client. I’m only writing about tree building and rendering here; everything else about the system is out of scope.

Where the first version spent its time#

The original arrangement was the obvious one. The backend assembled the entire tree — every person, every relationship, every node’s details — and handed it over. Each client then took that graph and worked out the geometry itself: the web app computed a layout and rendered it through ReactFlow, and there was a separate layout implementation on the server side for its own rendering path.

Two things about that shape bothered me once trees got large.

The layout was recomputed on every single view. Opening the tree meant solving the layout again, from scratch, in the client. But the arrangement of a tree only depends on the tree’s shape — and the shape doesn’t change when you look at it. I was paying a per-view cost for an answer that was the same every time.

Geometry had two independent implementations. Two layout engines, two sets of spacing rules, two chances to disagree. Any picture of the tree — the interactive view, an export — had to be produced by whichever engine that path happened to use.

And underneath both: rendering cost scaled with the whole tree, not the visible part. ReactFlow gives every node a real component in the DOM. That’s excellent when nodes are interactive things you want to style and attach handlers to, and a few hundred of them is comfortable. A thousand people means a thousand mounted components, each with layout, style recalculation and paint work, all of it happening whether or not you’re looking at that branch.

Concept: layout is a pure function of the tree, so compute it once#

The reframe that made everything else fall out: a tree and a picture of a tree are two different artifacts with two different lifetimes.

The tree changes when someone edits it — a person is added, a person is removed, and so on. Between those events, every coordinate in the picture is constant. Anything constant between rare events doesn’t belong on the read path at all; it belongs computed once and stored.

So layout moved to the backend and became stored geometry. When the shape of the tree changes, the layout is recomputed and the results are saved: an (x, y) per person, and a list of points per edge. Nothing recalculates on a view.

That changes what the API returns. Instead of a graph for the client to solve, it returns the solution:

public class GraphResponse {
    private List<Node> nodes;      // id, data, children, position
    private List<Edge> edges;      // sourceId, targetId, points
    private Metadata metadata;     // nodeWidth, nodeHeight, actionWidth
    private RootNode root;         // where to point the camera initially
}

The metadata block is small but load-bearing. Box dimensions are an input to the layout, so the clients have to agree with the server about them exactly — if the server spaces nodes for a 250-wide box and a client draws 260, every gap is wrong. Sending the dimensions that produced the layout means the clients can’t drift.

Concept: linear-time tidy trees#

With layout server-side and computed only on change, it can be a proper algorithm rather than something that has to be cheap enough to run in a browser on every page view. The one that fits a tree is Buchheim et al.’s improvement on Walker’s algorithm — the linear-time refinement of Reingold–Tilford “tidy” layout.

It gives you the properties you actually want from a genealogy view: parents centered over their children, siblings evenly spaced, no subtree ever overlapping its neighbour, and the whole thing as narrow as those constraints allow. It runs in two walks over the tree.

The first walk assigns a preliminary x to every node, bottom-up:

private void firstWalk(BuchheimNode v) {
    if (v.isLeaf()) {
        BuchheimNode ls = v.leftSibling();
        v.prelim = ls != null ? ls.prelim + NODE_WIDTH + X_GAP : 0;
    } else {
        BuchheimNode defaultAncestor = v.leftmost();
        for (BuchheimNode w : v.children) {
            firstWalk(w);
            defaultAncestor = apportion(w, defaultAncestor);
        }
        executeShifts(v);

        double mid = (v.leftmost().prelim + v.rightmost().prelim) / 2.0;
        BuchheimNode ls = v.leftSibling();
        if (ls != null) {
            v.prelim = ls.prelim + NODE_WIDTH + X_GAP;
            v.mod = v.prelim - mid;       // remember the offset, don't push children yet
        } else {
            v.prelim = mid;
        }
    }
}

A leaf goes just right of its left sibling. A parent wants to sit centered over its children — but it also has to clear its own left sibling’s subtree, so it may need to move right. The naive fix is to shift the entire subtree beneath it, which is what makes Walker’s original algorithm quadratic: deep trees shift the same nodes over and over.

Buchheim’s contribution is to defer those shifts. A node records a pending offset in mod rather than applying it, apportion resolves conflicts between adjacent subtrees using threaded contour pointers (nextLeft/nextRight walk the outline of a subtree without traversing its interior), and executeShifts distributes accumulated corrections across siblings in one pass. Then a single top-down walk turns deferred offsets into absolute coordinates:

private void secondWalk(BuchheimNode v, double m, int depth, Map<String, double[]> out) {
    out.put(v.id, new double[]{ v.prelim + m, MARGIN + depth * (NODE_HEIGHT + Y_GAP) });
    for (BuchheimNode child : v.children) {
        secondWalk(child, m + v.mod, depth + 1, out);
    }
}

Accumulating m + v.mod down the recursion is what makes each node’s final x the sum of every deferred shift above it — collected in one traversal instead of applied repeatedly. Every node is touched a constant number of times, so the whole layout is O(n).

Two properties matter more than the speed, given that this output is stored and shared:

It’s deterministic. The same tree produces byte-identical geometry every time. No random seeds, no iteration counts, no convergence thresholds — which is what makes the result safe to persist and to hand to different clients.
Depth maps directly to y. depth * (NODE_HEIGHT + Y_GAP) means generations are rows. That’s not a coincidence to be grateful for; it’s the reason a hierarchical layout is the right family here rather than a force-directed one.

Edges are precomputed too, as orthogonal elbows — four points from the bottom-center of the parent, out to a midpoint row, across, and into the top-center of the child:

double px = pp[0] + NODE_WIDTH / 2.0, py = pp[1] + NODE_HEIGHT;   // parent bottom-center
double cx = cp[0] + NODE_WIDTH / 2.0, cy = cp[1];                 // child top-center
double my = (py + cy) / 2.0;                                      // shared midpoint row

updateEdge(parentId, childId, List.of(
        new GraphResponse.Position(px, py),
        new GraphResponse.Position(px, my),
        new GraphResponse.Position(cx, my),
        new GraphResponse.Position(cx, cy)
));

So the clients never route a line. An edge arrives as a polyline; drawing it is moveTo then lineTo per point. Every geometric decision in the picture was made once, upstream.

Concept: one canvas, and a camera#

With geometry arriving precomputed, the client’s job collapses to painting. That makes the per-node-component model the wrong trade: you’re paying for a thousand mounted, styled, individually-tracked elements to display what is really a single picture.

Both clients now draw the whole tree into one surface — Canvas 2D on the web, a CustomPainter in Flutter — under a camera transform:

ctx.translate(vp.x, vp.y);
ctx.scale(vp.scale, vp.scale);

That’s the entire panning and zooming implementation. The viewport (vp) holds a translation and a zoom, everything is drawn in world coordinates, and the transform maps world to screen. Panning changes two numbers.

The payoff is that you can now ask a question you couldn’t ask before: what part of the world is on screen? Invert the transform and you have it as a rectangle:

const viewL = -vp.x / vp.scale;
const viewT = -vp.y / vp.scale;
const viewR = viewL + cw / vp.scale;
const viewB = viewT + ch / vp.scale;

Culling: draw cost tracks the viewport, not the tree#

Given that rectangle, every node gets one cheap rejection test before any drawing happens:

if (nx + metadata.nodeWidth < viewL || nx > viewR ||
    ny + metadata.nodeHeight < viewT || ny > viewB) return;

Four comparisons to skip a node entirely. Zoomed in on one family, the loop still visits every node but draws maybe twenty — so the expensive part of a frame is proportional to what’s visible, and a tree with 200 people costs the same to render as one with 2000 at the same zoom.

Edges need one extra consideration, and it’s the kind of thing you only find by looking at the result: an edge can be visible while neither of its endpoints is. A line from a parent above the screen to a child below it crosses the viewport, and testing endpoints alone would drop it. So edges are culled on their own bounding box, with padding:

const padding = 500;
if (maxX < viewL - padding || minX > viewR + padding ||
    maxY < viewT - padding || minY > viewB + padding) return;

The Flutter side precomputes each edge’s bounds once instead of recomputing per frame — the same idea, cached: if (!edge.bounds.overlaps(visibleRect)) continue;.

And since a static tree shouldn’t burn frames, painting is behind a dirty flag: an animation-frame loop runs continuously but only repaints when something actually changed — a pan, a zoom, a newly loaded image.

Concept: three levels of detail#

Culling handles “zoomed in.” Zoomed out is the opposite problem: now everything is on screen, and the naive approach draws a thousand rounded cards, a thousand avatar images and a thousand measured, truncated text strings — to produce a picture in which each node is eight pixels wide and none of the text is legible.

The observation is that at low zoom, the most expensive things to draw are the ones carrying the least information. Nobody reads names at 6% zoom; they’re reading the shape of the tree — where the branches are, how deep it goes, which subtree is dense. So detail is tied to zoom, in three tiers:

const lod = isExporting
  ? "full"
  : vp.scale > 0.3
    ? "full"
    : vp.scale > 0.12
      ? "simple"
      : "minimal";

full (zoom above 30%) — rounded card, avatar clipped into a circle, name with ellipsis truncation measured against the box width, action affordance.
simple (above 12%) — card and border, one centered name string. No images, no per-node measurement.
minimal — a filled rectangle, and thinner edge strokes. Nothing else.

if (lod === "minimal") {
  ctx.fillRect(nx, ny, metadata.nodeWidth, metadata.nodeHeight);
  return;
}

That early return is doing a lot of work. It skips a rounded-rect path, a stroke, an image draw, a font set, a text measurement and a fill — per node, for every node on screen, in exactly the regime where the most nodes are on screen. The whole-tree view is the cheapest view to draw rather than the most expensive one.

Text measurement is the sneaky cost, incidentally. measureText in the truncation loop shortens a string one character at a time until it fits — fine for twenty visible cards, not fine for a thousand. Tiering it away at simple matters more than the images do.

Two supporting details in the same spirit: images are cached by URL with a load callback that flips the dirty flag, so a scroll past 50 faces doesn’t re-request anything and a late-arriving image triggers exactly one repaint. And past 20 nodes, a minimap draws the whole tree as dots with a rectangle marking the current viewport — cheap to draw from positions you already have, and it’s what makes navigating a tree far larger than the screen feel bounded.

Both clients use the same thresholds and the same culling maths. Web and mobile are different rendering APIs, but the geometry they consume and the decisions they make about it are identical, which means the tree looks the same on a phone and a laptop.

What you give up, and getting it back#

Drawing into a canvas means there are no per-node objects. Nothing to attach a click handler to, nothing to query, nothing the accessibility tree or the browser’s own hit-testing knows about. That’s the real cost of this approach, and it has to be paid back deliberately.

Since positions are stored server-side, both interactions resolve through coordinates. A click is converted from screen space back into world space with the same transform, inverted:

const worldX = (mouseX - vpx) / scale;
const worldY = (mouseY - vpy) / scale;

and the person occupying that point is looked up by coordinate. The x-offset within the box even distinguishes a tap on the card from a tap on the action strip beside it:

GetPersonCoordinateResponse.CoordinateType type = GetPersonCoordinateResponse.CoordinateType.NODE;
if (x > (position.getX() + NODE_WIDTH)) type = GetPersonCoordinateResponse.CoordinateType.ACTION;

Search works the same way in reverse: a name resolves to a position, and the camera animates there. The client never scans the tree for a match — it asks where to point, then points.

Design note. This trades a round trip per interaction for not maintaining a spatial index in each client. Given that both clients need identical behaviour and the positions are already stored, resolving centrally keeps one implementation instead of three. A client-side quadtree or uniform grid is the alternative if local, zero-latency hit-testing becomes the priority — the positions are all there in the payload to build one from.

Design notes for the next pass#

Culling is a linear scan. Every frame iterates all nodes to test four inequalities each. That’s cheap next to drawing, and it kept the code simple, but it’s still O(n) per frame — the part that would matter first on a tree an order of magnitude larger. Bucketing nodes into a coarse grid by position, then visiting only the buckets the viewport touches, turns the scan into a lookup. Worth doing when profiles say the scan, rather than the drawing, is the frame budget.

The LOD thresholds are tuned to the current box size. 0.3 and 0.12 were chosen by zooming until text stopped being readable, against a 250×80 box. A more principled version derives them from the rendered size — switch away from text when cap height falls below ~10 device pixels — which would then hold automatically if node dimensions ever change.

The edge padding is a heuristic. 500 world units comfortably covers the longest elbow at the spacing in use. Deriving it from the maximum edge extent would make it self-adjusting.

Export takes the other path deliberately. Saving a PNG or PDF renders off-screen at full detail regardless of zoom, then scales down to stay inside browser canvas size limits — the one case where you want every name drawn, because the output is meant to be zoomed into later.

What I took away#

The idea I keep reusing from this is the split I started with: the tree and the picture of the tree are different artifacts with different lifetimes. Once that was explicit, most of the design followed. Geometry that only changes on edit gets computed once and stored. Geometry the client already has doesn’t need recomputing. Detail nobody can see doesn’t need drawing.

The satisfying part is how the scaling behaviour inverted. Before, every additional person made every view of the tree more expensive for everyone. Now the layout cost is paid once per edit, and the drawing cost is bounded by the size of the screen — which is the one quantity in this whole problem that never grows.

UDP Hole Punching: How Two Machines Behind NAT Talk Directly

Mon, 22 Dec 2025 00:00:00 GMT

Two laptops on opposite sides of the world, each behind a home router. Neither has a public IP. Neither owner will configure port forwarding. They want to send each other a file directly, without a server in the middle paying for the bandwidth.

My instinct said that was impossible. Every packet aimed at either machine arrives at a router with no idea which internal device it belongs to, and gets dropped. Yet this is exactly how Zoom calls, BitTorrent swarms and multiplayer game sessions work. The technique is UDP hole punching, and it exploits a specific property of how NAT devices decide what to let in.

I wanted to understand two things properly: how a direct path gets opened between two unaddressable hosts, and what you have to rebuild yourself once you’re moving bulk data over a protocol that guarantees nothing. So I built a peer-to-peer chat and file sharing system in Go and worked through both: github.com/rustamniraula90/gop2p.

These are my notes from that build.

Concept 1: NAT is a phone system with no direct extensions#

A router doing Network Address Translation has one public address and many internal devices. When an internal device sends a packet out, the router picks an unused public port, rewrites the source address, and records the pairing:

internal 192.168.1.14:3001  ⇄  public 203.0.113.7:51820

Return traffic hitting 203.0.113.7:51820 gets rewritten back and delivered. Traffic arriving at any other port has no entry to match and is discarded.

Three properties follow, and they turn out to be the whole basis of hole punching:

Mappings are created by outbound packets. Nothing else creates them. You cannot ask for one.
Mappings are usually destination-filtered. Most consumer routers only accept return packets from the address the outbound packet went to. (This is “port-restricted cone” NAT — it matters later.)
Mappings expire. UDP has no connection state, so the router guesses. Idle entries can be evicted in as little as 30 seconds.

So neither peer can be called. But both can dial, and a mapping created by dialing will accept the reply. If both dial each other at the same moment, each one’s outbound packet opens the hole the other’s packet needs. That simultaneity is the trick, and arranging it is the entire problem.

Concept 2: A rendezvous server that learns your address for you#

Peer A cannot know its own public address — the NAT that assigned it sits outside A’s view. But any server A talks to can see it, because translation already happened by the time the packet arrives. So the first job of a coordination server is to be a mirror.

This is why I have the server read the address off the socket rather than trust anything the client claims:

n, remote, err := conn.ReadFromUDP(buf)
// ...
go handler.HandlePacket(remote, buf[:n])

func (r Registry) Register(id string, name string, addr *net.UDPAddr) {
    r.clients[id] = &ClientInfo{
        ID: id, Name: name,
        IP:   addr.IP,     // the post-NAT address, observed
        Port: addr.Port,
        LastSeen: time.Now(),
    }
}

Once I’d written that, STUN made sense to me for the first time: it’s built on the same principle — the observer tells you who you appear to be. The server here isn’t a relay for data, just a directory of “where each peer appears to live,” assembled from packets it received.

Keepalives exist to fight timeouts, not to detect death#

Because mappings expire on idle, a peer that goes quiet becomes unreachable even though it’s still running. So the client re-announces itself on a timer:

ticker := time.NewTicker(10 * time.Second)
for range ticker.C {
    pm.sendHeartbeat()
}

Ten seconds against a 30-second eviction window. What I found interesting is the inversion from what “heartbeat” normally means: this isn’t the client proving liveness to the server, it’s the client keeping its own NAT hole propped open. That’s why the server handles register and heartbeat with the same function — a heartbeat is a re-register that skips the acknowledgement.

Concept 3: Address exchange as a consent step#

Building the directory made a privacy consequence obvious that I hadn’t considered up front: listing peers means publishing where people live. So I split discovery from address exchange. Browsing the directory returns identities with addresses deliberately withheld:

peers = append(peers, protocol.PeerInfoPayload{
    ID:   c.ID,
    Name: c.Name,
    IP:   nil,   // withheld until both sides consent
    Port: 0,
})

Addresses only go out after both parties agree to connect, which makes the handshake a four-message consent flow through the server:

A                          Server                          B
|-- CONNECT_REQUEST(B) ------->|                             |
|                             |-- CONNECT_FORWARD(A) ------->|
|                             |<------- CONNECT_ACCEPT(A) ---|
|<-- PEER_INFO(B: ip:port) ----|--- PEER_INFO(A: ip:port) -->|
|                                                            |
|============= PUNCH / PUNCH_ACK (direct) ==================>|
|<========= the server is now out of the picture ============|

The last server step is the load-bearing one: both PEER_INFO messages go out from the same handler, microseconds apart. That’s not tidiness, it’s the mechanism. Hole punching needs both sides transmitting at nearly the same instant, and the only shared clock they have is “when the server told me.”

Concept 4: Simultaneous open, and why the first packets are supposed to fail#

Once each side knows where to aim, both fire. The first packets die, and that’s the design working as intended:

A’s punch reaches B’s router. B hasn’t sent anything to A yet, so there’s no mapping for A. Dropped.
But A’s punch did create a mapping on A’s side, valid for B’s address.
B’s punch, moments later, reaches A’s router and finds that hole waiting. It gets through.
A replies. B’s router now has its own mapping from B’s punch, so the reply lands too.

A dropped packet isn’t a failure — it’s the cost of opening your own side. Which means the strategy is to retry on an interval until confirmation arrives:

func (pm *P2PManager) startPunching(peer *Peer) {
    addr := &net.UDPAddr{IP: peer.IP, Port: peer.Port}
    // send packet and pray
    for i := 0; i < 10; i++ {
        if pm.Peers[peer.ID].State == StateConnected {
            return
        }
        pm.udp.Send(addr, protocol.UDPMessage{
            Type:    protocol.TypePunch,
            Payload: map[string]string{"from": pm.identity.ID},
        })
        time.Sleep(500 * time.Millisecond)
    }
}

Ten attempts over five seconds converts a race into a near-certainty. The // send packet and pray comment is the one I wrote while figuring this out, and I kept it because it describes the mechanism accurately: you can’t verify the hole is open except by something coming back through it.

And when something does come back, I take the address from the packet rather than the directory:

peer.State = StateConnected
peer.IP = remote.IP     // trust the socket, not the directory
peer.Port = remote.Port

If the NAT reassigned a port between registration and the punch, the server’s record is stale and the source address on the wire is ground truth. Same principle as the server observing remote in the first place, one layer down.

Where hole punching stops working#

Property 2 said most routers filter by destination address. Symmetric NAT goes further: it allocates a different public port for every distinct destination. The port the server observed is therefore not the port your peer’s traffic will use, so your peer aims at a hole that will never exist. Retrying can’t fix that.

This is where production systems pair STUN with TURN — a relay that carries traffic when punching fails. Falling back costs you the bandwidth savings that are the whole point of P2P, so it’s a last resort rather than a default. My implementation punches, and if punching fails the peer stays disconnected; understanding why it fails was the goal, and a relay would have hidden that.

Concept 5: UDP gives you a datagram, and nothing else#

Suppose the hole is open. I now had a channel that can lose packets, reorder them, duplicate them and corrupt them, with no notification of any of it. Moving a file across that meant rebuilding, by hand, the guarantees TCP would have provided:

Guarantee	Who provides it	Mechanism
Message boundaries	you	framing, or one message per datagram
Integrity	you	checksum/hash per unit
Ordering	you	explicit sequence numbers
Completeness	you	track what’s missing, re-request
Flow control	you	limit outstanding requests

Writing that table out is what turned “UDP is unreliable” from a slogan into a checklist. The transfer became a request/response protocol over indexed pieces:

Downloader                                   Uploader
    |--- DOWNLOAD_REQUEST(file, reqID) --------->|   compress + hash
    |<-- DOWNLOAD_INFO(chunkCount, sizes) -------|
    |--- CHUNK_REQUEST(reqID, 0) --------------->|
    |<-- CHUNK_RESPONSE(0, data, sha256) --------|
    |            ... repeat ...                  |

Chunking: sequence numbers are what make loss recoverable#

Splitting the payload into fixed-size indexed chunks buys you everything else. An index means you can detect a gap, re-request one piece without restarting, and reassemble out-of-order arrivals. Negotiating up front tells the receiver how many pieces exist, so it can allocate its whole table before the first byte arrives:

chunkCount := int(upload.CompressedSize) / upload.ChunkSize
if int(upload.CompressedSize)%upload.ChunkSize != 0 {
    chunkCount++
}

I compress before chunking rather than per chunk, for two reasons: the compressor sees the whole file’s redundancy instead of a 16 KB window, and the piece count is known in advance.

Integrity: verify each piece before trusting it#

Corruption is silent, so each chunk carries a hash of its own contents, computed by the sender during preparation, and the receiver refuses to store anything that doesn’t match:

hash := sha256.Sum256(data)
computedSHA := hex.EncodeToString(hash[:])
if computedSHA != sha {
    download.Chunks[index].Status = ChunkError
    download.Chunks[index].RetryCount++
    return fmt.Errorf("chunk SHA mismatch")
}

Verified chunks land as individual files named by index, which is what makes ordering a non-problem: chunk_37 is correct whenever it arrives, and order only matters at assembly, where pieces are concatenated by index and decompressed.

There’s a second integrity check hiding in that design, and I only noticed it afterwards. Gzip has its own internal structure, so if the pieces were assembled in the wrong order, decompression fails rather than silently producing a corrupt file. Layering a format that detects its own corruption on top of per-chunk hashes gave me end-to-end validation I hadn’t written.

Flow control: a window, because both extremes are wrong#

One request at a time wastes a full round-trip per chunk — on a 100 ms link that’s 160 KB/s regardless of how fat the pipe is. All requests at once overruns buffers and drops most of them. So I keep a bounded number in flight:

windowSize := 8
for i := 0; i < windowSize; i++ {
    pm.requestNextChunk(requestID)
    time.Sleep(time.Millisecond * 10)
}

and refill on each arrival, which holds the window full for the rest of the transfer:

if err := pm.Downloader.SaveChunk(resp.RequestID, resp.ChunkIndex,
    resp.ChunkData, resp.ChunkSHA); err != nil {
    return
}
pm.requestNextChunk(resp.RequestID)

This is a receiver-driven window, the inverse of TCP’s sender-driven one, and I like it for peer-to-peer: the side that knows its own downlink capacity sets the rate, and a peer serving five downloads doesn’t have to reason about any of them.

Design notes for the next iteration#

The table listed five guarantees. This version implements framing, integrity, ordering and flow control; the notes below are where I stopped, and why each one clarified something.

Retransmission is out of scope in this pass. A chunk marked ChunkRequesting stays there — nothing re-arms it, so a lost datagram narrows the window by one. I built the state machine a timer would drive (RetryCount is in the struct, ready for it) before building the timer, and doing it in that order is what made the role of timers concrete: reliability over UDP is fundamentally about deciding how long you’re willing to wait, because you can’t otherwise distinguish “lost” from “slow.” Adding a per-chunk deadline that returns ChunkRequesting to ChunkPending is the natural next step.

Frame size is a fragmentation trade-off. ChunkData []byte inside a JSON message is base64-encoded, so a 16 KB chunk is ~22 KB on the wire — past a 1500-byte MTU, so each chunk is IP-fragmented into ~15 packets, and losing any one fragment discards the chunk. I chose 16 KB for fewer round-trips; measured against fragmentation, a binary frame under ~1400 bytes trades round-trips for a smaller blast radius per loss, and pairs better with retransmission. Worth measuring both.

The relay’s registry is single-process simple. Each packet is handled in its own goroutine and they share one map[string]*ClientInfo. That’s fine for the two-client scenario I was testing; a sync.RWMutex (or moving registration onto a single owning goroutine) is what makes it safe under real concurrency. LastSeen is recorded and not yet consumed — it’s the hook for evicting stale peers.

Request and response share a type tag. TypeListPeersRequest and TypeListPeersResponse are both 0x04, disambiguated by direction: the server never receives a response and the client never receives a request. It’s compact and works for a two-role topology. Distinct tags become worthwhile as soon as a node can play both roles.

What I took away#

The thing I actually wanted — two clients behind separate NATs, no port forwarding, no relay carrying their data, finding each other through a coordinator that only ever tells them where to aim — works, and files arrive byte-for-byte intact.

What I didn’t expect was how much I’d learn about TCP by not having it. Sequence numbers, acknowledgements, retransmission timers, MSS negotiation: I’d always filed those as protocol trivia. Building a transfer that needs each one, and feeling exactly which problem each solves, made TCP look less like a spec to memorise and more like a list of hard-won answers.

Backpropagation From First Principles: What a Neural Network Actually Does

Thu, 25 Jul 2024 00:00:00 GMT

model.fit(x, y) is where my understanding of neural networks used to stop, because that one call hides everything I wanted to know. What is being adjusted? By how much? How does a number computed at the output end up changing a weight five layers back?

It turns out the concepts are simpler than the framework APIs suggest: a network is a pile of multiply-add, its error is a single number, and training is repeatedly asking “which direction should each weight move to make that number smaller?” All of it fits in your head at once.

To force myself to actually hold it there, I built a multi-layer perceptron in plain Java that learns to recognise handwritten digits — no TensorFlow, no DL4J, not even a matrix library. The only dependency is a logging framework: github.com/rustamniraula90/hand-written-number-recognition.

These are my notes from working through it.

Concept 0: A network is a function you can shape#

Before any learning happens, a network is just a parameterised function. Mine turns 784 numbers (a 28×28 image, flattened) into 10 numbers (confidence per digit) through two intermediate layers:

784  ──[W₁ 784×128]──▶  128  ──[W₂ 128×64]──▶  64  ──[W₃ 64×10]──▶  10
      +bias, ReLU              +bias, ReLU           +bias, softmax

Those weight matrices are the model — 784×128 + 128×64 + 64×10 ≈ 109,000 numbers. Training means finding values for them. Everything else is bookkeeping.

I declared the shape rather than hardcoding it, mostly to keep that idea visible while I worked:

MLPNetworkOptimized network = MLPNetworkOptimized.builder(2)
        .addInputLayer(28 * 28)
        .addOutputLayer(10)
        .addHiddenLayer(128)
        .addHiddenLayer(64)
        .setLearningRate(0.001)
        .setBatchSize(10)
        .setEpoch(20)
        .build();

Why initialization isn’t arbitrary#

Start every weight at zero and every neuron in a layer computes the same thing forever — identical gradients, identical updates, no differentiation. Start them too large and activations compound layer over layer until they saturate. So weights start random, but scaled to layer width:

public static double xavier(int in, int out) {
    return random.nextGaussian() * Math.sqrt(2.0 / (in + out));
}

That’s Xavier initialization: variance inversely proportional to fan-in plus fan-out, so signal magnitude stays roughly constant as it propagates.

Design note. I pass the network’s overall input and output sizes for every layer rather than each layer’s own fan-in and fan-out, so all layers share one variance. It trains fine, which taught me something useful about the shape of this parameter: initialization is a conditioning concern and fairly forgiving of imprecision, unlike a gradient sign error, which is fatal immediately. Per-layer fan-in/fan-out is the textbook form and the easy refinement.

Concept 1: The forward pass is multiply, add, squash#

Each layer computes output = activation(W·input + b). Written out per neuron, the whole idea is eight lines:

double z = 0.0;
for (int j = 0; j < neuron.weights.length; j++) {
    z += neuron.weights[j] * activations[j];
}
z += neuron.bias;
neuron.output = relu(z);

The activation function is the only nonlinear step, and without it the network collapses. Stack two linear layers and you get another linear layer — depth would buy nothing. ReLU (max(0, x)) is the usual choice: trivially differentiable, and it doesn’t saturate for positive inputs.

I used the bounded variant, ReLU6:

public static double relu(double x) {
    return Math.clamp(x, 0, 6);
}

public static double reluDerivative(double x) {
    return x > 0 ? 1 : 0;
}

Design note. The ceiling of 6 is the trick MobileNet uses to keep activations bounded, and I liked it for stability. The paired derivative here is plain ReLU’s, which returns 1 above the clamp where ReLU6’s true derivative is 0. In this network it makes no observable difference — the small learning rate and scaled init keep pre-activations well below 6 — but writing the two functions next to each other is what made the general rule stick for me: an activation and its derivative are a matched pair, and changing one means revisiting the other. Clamping the derivative at 6 as well is a one-line change and the right call if the ceiling ever binds.

The output layer answers a different question#

Hidden layers produce features. The output layer has to produce a probability distribution over 10 classes — ten numbers, non-negative, summing to 1. That’s softmax: exponentiate, then normalise.

public static double[] softmax(double[] x) {
    double sum = 0;
    double[] result = new double[x.length];
    for (int i = 0; i < x.length; i++) {
        result[i] = Math.exp(x[i]);
        sum += result[i];
    }
    for (int i = 0; i < x.length; i++) result[i] /= sum;
    return result;
}

Exponentiating first is what makes it a soft max: it amplifies differences, so a logit slightly ahead becomes a lot more probable, without the hard winner-take-all of picking the maximum.

Design note. Library implementations subtract the maximum logit before exponentiating, because Math.exp overflows around 710. Mine skips that, which is fine for MNIST-scale logits and is the first thing I’d add if inputs got wilder. Finding out why that subtraction exists — by not having it — was more instructive than reading about it.

Concept 2: Error has to be one number#

You can’t optimise ten outputs at once. Gradient descent needs a single scalar to reduce, so the ten probabilities and the correct answer collapse into one loss. For classification that’s cross-entropy:

public static double crossEntropy(double[][] actual, double[][] expected) {
    double loss = 0;
    for (int i = 0; i < actual.length; i++) {
        for (int j = 0; j < actual[i].length; j++) {
            loss += expected[i][j] * Math.log(actual[i][j]);
        }
    }
    return -loss;
}

With one-hot targets every term but the correct class multiplies by zero, so the loss is just -log(probability assigned to the right answer). Assign the truth 0.9 and you’re barely penalised; assign it 0.01 and the log punishes you enormously. That asymmetry is the point: confident wrong answers should hurt far more than uncertain ones.

Concept 3: Backpropagation is the chain rule, applied backwards#

This is the part frameworks hide, and the part I built this for. We have one number (the loss) and 109,000 weights, and we need ∂loss/∂w for each. Computing them independently would mean 109,000 passes through the network. Backpropagation gets all of them in one backwards pass by reusing work.

The insight that unlocked it for me: for any weight, ∂loss/∂weight = (how much the loss changes with that neuron’s output) × (what that neuron’s input was). The first factor — call it delta — is the only hard part, and each layer’s delta can be computed from the layer after it. So you compute deltas once, right to left, and every weight update reads off a delta you already have.

Step one: the delta at the output layer. This is where the earlier choices pay off. Differentiate cross-entropy composed with softmax and all the exponentials and logarithms cancel:

delta[i][j] = outputs[i][outputs[i].length - 1][j] - expectedOutputs[i][j];

prediction − target. That’s the entire output-layer gradient. Softmax and cross-entropy are paired because their derivatives collapse like this, and seeing it fall out of the algebra was the most satisfying moment of the project.

Step two: update this layer, then push delta backwards. For each weight, gradient = delta × the input that flowed through it, averaged over the batch:

for (int i = weight.length - 1; i >= 0; i--) {          // layers, back to front
    for (int j = 0; j < weight[i].length; j++) {
        for (int k = 0; k < weight[i][j].length; k++) {
            double gradientSum = 0.0;
            for (int l = 0; l < batchSize; l++) {
                gradientSum += delta[l][k] * inputs[l][i][j];
            }
            weight[i][j][k] -= learningRate * (gradientSum / batchSize);
        }
    }
    // ... bias update, same shape ...
    if (i > 0) {
        // delta for the previous layer: weighted sum of this layer's deltas,
        // scaled by the derivative of that layer's activation
        newDelta[j][k] = sum * MathUtil.reluDerivative(outputs[j][i-1][k]);
        delta = newDelta;
    }
}

Three ideas live in that loop:

weight -= learningRate * gradient is gradient descent. The gradient says which direction increases loss; step the other way, scaled small so you don’t overshoot.
The previous layer’s delta is this layer’s deltas weighted by the connections between them — error attributed backwards in proportion to contribution.
Multiplying by the activation derivative closes the chain rule. A neuron whose ReLU output was zero contributed nothing, so it receives no blame.

Concept 4: Mini-batches, and why batch size is a real choice#

You could update after every sample (noisy, slow) or after all 10,000 (smooth, expensive, glacial progress). Mini-batching splits the difference: accumulate gradients over N samples, then step once.

I implemented it as buffer-and-fire rather than a nested loop — the backward pass is called after every sample but only acts when the buffer fills:

private void backwardPass(int epoch) {
    if (index == batchSize) {
        index = 0;
        double loss = MathUtil.crossEntropy(finalOutput, this.expectedOutputs);
        tuneParameters();
        this.inputs = new double[batchSize][weight.length][];
        this.outputs = new double[batchSize][weight.length][];
        this.expectedOutputs = new double[batchSize][];
    }
}

It keeps the training loop flat, at the cost of coupling one index to three buffers. Averaging over the batch (gradientSum / batchSize) is what makes the learning rate independent of batch size — otherwise doubling the batch would double every step.

Design note. predict() calls the same forwardPass(), which writes into those training buffers at the current index. Train-then-predict is exactly how the program runs, so this is fine as written; separating inference state from training state is the change I’d make before ever interleaving the two.

The lesson I didn’t expect: data layout decides what you can express#

I wrote this network twice, and the comparison taught me more than either version alone.

The first, MLPNetwork, is the design an OO instinct reaches for: a Layer of Neuron objects, each owning its weights, bias, input and output. It reads beautifully. Watch where the weight update sources its input activation:

double input = layerIndex == 0 ? inputs[k][j] : prevLayer.neurons[j].output;
gradientSum += delta[k][i] * input;

For the first layer, inputs[k][j] indexes sample k of the batch. Everywhere else, prevLayer.neurons[j].output is one field on a shared object — it holds whatever the most recent forward pass left there. A neuron object can hold one activation; mini-batch backprop needs one per sample in flight. The structure quietly asserts something the algorithm doesn’t agree with.

The second version, MLPNetworkOptimized, replaced objects with flat arrays and kept per-sample buffers:

private double[][][] inputs;   // [batchIndex][layer][neuron]
private double[][][] outputs;

which is why its update can say inputs[l][i][j] — sample l, layer i, neuron j.

I named it “Optimized” because I was chasing speed: flat double[][][] beats chasing pointers through a hundred thousand objects, and the JIT keeps the inner loop in registers. What I got alongside the speed was a structure that could hold the state the maths actually needs. That reframed the whole exercise for me, and it’s the part that generalises furthest beyond neural networks: choosing the data layout is choosing which behaviours are expressible. Neuron.output looked like good encapsulation and was really a claim about dimensionality.

It also explained something I’d taken for granted. Every framework represents a batch as a tensor with a leading batch dimension, and I’d always read that as a vectorisation detail. It isn’t only that — the batch axis has to exist somewhere, and putting it in the data structure is what makes the gradient definable.

Before any of it: the data is a binary blob#

One more concept, because it’s the first wall I hit. MNIST doesn’t ship as CSV — it’s IDX, a big-endian header followed by raw bytes. There’s no library call; you assemble integers from bytes yourself:

private static int readInt(FileInputStream fis) throws IOException {
    return (fis.read() << 24) | (fis.read() << 16) | (fis.read() << 8) | fis.read();
}

Then you assert the magic number (2051 for images, 2049 for labels) so a wrong file fails loudly instead of training on garbage, and you normalise pixels into a small range so the first layer’s weights don’t have to absorb a factor of 255. Mine divides by 254.0, which puts a saturated pixel a hair above 1.0 — immaterial at this scale, and a good reminder that scaling constants deserve the same attention as the algorithm. The most effective verification I found was writing a few parsed samples back out as PNGs and looking at them; a transposed dataset is instantly obvious to an eye and invisible to an assertion.

Design note on the run configuration. main trains on the t10k files (10,000 images) and evaluates on the train files (60,000). Conventionally that’s the other way round, and I kept it because the property that matters held — the evaluation set is strictly disjoint from what it learned on — and it makes for a harsher test: less data to learn from, six times as much to be judged on. Shuffling between epochs is the other knob I left alone; with fixed batch composition repeating 20 times, adding a shuffle is the cheapest available improvement.

What’s worth keeping from this#

A network is multiply-add plus one nonlinearity; depth without the nonlinearity is worthless.
Softmax and cross-entropy are paired because their gradients collapse to prediction − target.
Backprop is one reverse pass that reuses each layer’s delta, and that reuse is the entire reason training is tractable.
An activation and its derivative are one decision, not two.
Frameworks aren’t hiding the calculus. They’re hiding numerical stability guards, initialization scaling, shuffling and the batch axis — and the clearest way to see the shape of that work is to leave it out and notice what starts mattering.

Observability: Why Your Logs Need Structure, Correlation and a Pipeline

Tue, 01 Dec 2020 00:00:00 GMT

A request comes in and takes 4 seconds. Nobody knows why. You SSH into a box, grep a log file, find the line, and it says:

2020-12-01 14:23:07 INFO  Getting test with id: 4471

Which tells you almost nothing. Was the database slow? Which of the 40 requests in flight was this one? Did it call another service, and was that slow? You have a fact with no context, on one machine, in a format only a human can read.

I’d hit that wall enough times to want to understand the alternative properly rather than copy a config from a blog post. Three concepts do almost all the work: making log events machine-readable, giving every event a correlation identifier tying it to the request it belongs to, and running a pipeline that collects events off ephemeral machines into one queryable place.

So I built a working stack end to end — a Spring Boot service instrumented and shipped into Elasticsearch, Logstash, Kibana and Elastic APM, all in Docker Compose: gitlab.com/rustam.niraula90/apm.

These are my notes from wiring it up.

Concept 1: Logs are data, so stop formatting them as prose#

The line above is a string. To answer “all requests slower than 2s for user 4471 on the checkout path,” something has to re-derive fields from that string with regular expressions — and those regexes break every time someone edits a log message.

Structured logging inverts the relationship: emit the fields, let the tooling render the sentence. Same event, as JSON:

{
  "@timestamp": "2020-12-01T14:23:07.412Z",
  "level": "INFO",
  "logger_name": "com.apm.apm.test.TestController",
  "thread_name": "http-nio-8080-exec-3",
  "message": "Getting test with id: 4471",
  "trace": { "trace_id": "af3d1c...", "span_id": "b1e4..." }
}

Now level:ERROR AND trace.trace_id:af3d1c* is a query, not a parsing project.

The thing that sold me is that in Spring Boot this is a logging configuration change, not a code change. Application code keeps making ordinary SLF4J calls, and an encoder turns each event into JSON on the way out:

<appender name="jsonConsoleAppender" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
        <providers>
            <timestamp><timeZone>UTC</timeZone></timestamp>
            <logLevel/>
            <message/>
            <loggerName/>
            <threadName/>
            <mdc/>
            <stackTrace/>
        </providers>
    </encoder>
</appender>

That separation is the part worth internalising: structured logging is a transport decision, not an API your business logic should know about. The controller still just writes log.info("Getting test with id: {}", id).

Two choices in that config I made deliberately. Timestamps in UTC, because correlating events across machines in different zones is otherwise guesswork. And <stackTrace/> as its own field, so a Java exception stays one event instead of becoming 30 unparseable lines — the classic failure mode of line-oriented log shipping.

Concept 2: Correlation IDs — the field that makes logs useful#

Structure alone isn’t enough. With 40 concurrent requests you have 40 interleaved conversations and no way to follow one. You need a shared key on every event belonging to the same request.

That’s a trace ID: generated when a request enters the system, attached to every log line it produces, and propagated to every downstream service it calls. Filter by it and you get one request’s complete story, in order, across every process it touched.

The plumbing that makes this ergonomic is MDC — Mapped Diagnostic Context — a thread-local map the logging framework merges into every event. Instrumentation puts the IDs in; the encoder pulls them out:

<pattern>
    {
    "trace": {
    "trace_id": "%mdc{X-B3-TraceId}",
    "span_id": "%mdc{X-B3-SpanId}",
    "parent_span_id": "%mdc{X-B3-ParentSpanId}"
    }
    }
</pattern>

The three fields encode a hierarchy, and the distinction is what took me longest to internalise:

trace_id — one per end-to-end request, identical across every service.
span_id — one per unit of work inside it (an HTTP handler, a DB query).
parent_span_id — which unit spawned this one.

With parent links, a flat event stream reconstructs into a tree, which is what lets a UI show a waterfall — and a waterfall is what answers “where did the 4 seconds go?”

X-B3-* is the B3 propagation format from Zipkin: the IDs travel as HTTP headers, so a downstream service adopts the caller’s trace instead of starting its own. Realising that this header contract is the entire mechanism of distributed tracing — everything else is presentation — was the moment the topic stopped feeling like magic.

One detail in that config: the same keys are excluded from the generic <mdc/> provider.

<mdc>
    <excludeMdcKeyName>traceId</excludeMdcKeyName>
    <excludeMdcKeyName>X-B3-TraceId</excludeMdcKeyName>
    ...
</mdc>

Otherwise every trace ID appears twice — once flat, once nested — and Elasticsearch indexes both. Structure your fields once, in the shape you’ll query.

Concept 3: Instrumentation without touching your code#

Logs tell you what your code chose to say. Traces need timing around everything — every HTTP handler, every JDBC call, every outbound request — and hand-wrapping all of that isn’t realistic.

The answer is a runtime agent. On the JVM, -javaagent loads before main and rewrites bytecode as classes load, wrapping known framework entry points with timing and ID propagation:

entrypoint: ["java",
  "-javaagent:/app/libs/elastic-apm-agent-1.19.0.jar",
  "-Delastic.apm.application_packages=com.apm",
  "-Delastic.apm.service_name=apm-service",
  "-Delastic.apm.server_urls=http://apm-server:8200",
  "-cp", "/app/resources:/app/classes:/app/libs/*",
  "com.apm.apm.ApmApplication"]

Four flags is the whole integration. There’s no tracing code in the application, and yet every controller method, every Hibernate query and every outgoing call gets a span with a parent link. The agent also seeds the MDC keys the logging config above reads — which is exactly how logs and traces end up sharing an ID without either subsystem knowing about the other. Watching that happen for the first time is what made the design click.

application_packages is the flag I’d have skipped if I hadn’t read what it does: it tells the agent which frames are yours, so stack traces highlight your code rather than 60 lines of framework internals.

The trade-off is worth stating plainly: agents give you breadth for free, but they only know the frameworks they were taught, they cost startup time and some overhead, and they’re opaque when they misbehave. Modern practice has largely converged on OpenTelemetry for this role — same agent concept, vendor-neutral wire format — and that’s what I’d reach for now.

Concept 4: The pipeline — collect, parse, store, visualise#

Application containers are ephemeral. Logs written inside them die with them, so events have to be moved somewhere durable. Every observability stack, whatever the brand names, is the same four stages:

  app (stdout, JSON)
        │
    ┌───▼────┐   collect       ┌──────────┐   parse/enrich
    │Filebeat│ ──────────────▶ │ Logstash │ ─────────────┐
    └────────┘                 └──────────┘              │
                                                    ┌────▼─────────┐
  app (APM agent) ──── traces ───▶ APM Server ────▶ │Elasticsearch │
                                                    └────┬─────────┘
                                                         │ query
                                                    ┌────▼───┐
                                                    │ Kibana │
                                                    └────────┘

Collect (Filebeat) — a lightweight agent that tails logs and forwards them. Deliberately dumb, because it runs everywhere.
Parse and enrich (Logstash) — heavier processing: reshape fields, add metadata, route.
Store and index (Elasticsearch) — the inverted index that makes ad-hoc queries over billions of events fast.
Visualise (Kibana) — search, dashboards, trace waterfalls.

Splitting collect from parse is the architectural decision that matters. Collection has to be cheap and ubiquitous; parsing is CPU-hungry and centralised. Keeping the sidecar dumb is what makes running one per host affordable.

Discovery by label, not by path#

The obvious way to collect container logs is to mount known paths. That breaks the moment a container is rescheduled or a new service appears — someone has to go update config.

The better concept is autodiscovery: the collector watches the Docker socket and configures itself from container metadata, so a service opts itself in.

filebeat.autodiscover:
  providers:
    - type: docker
      labels.dedot: true
      templates:
        - condition:
            contains:
              container.labels.collect_logs_with_filebeat: "true"
          config:
            - type: container
              paths:
                - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
              processors:
                - decode_json_fields:
                    when.equals:
                      docker.container.labels.decode_log_event_to_json_object: "true"
                    fields: ["message"]
                    target: ""
                    overwrite_keys: true

The service declares its own participation with two labels:

labels:
  collect_logs_with_filebeat: "true"
  decode_log_event_to_json_object: "true"

Add a service, give it the labels, and it appears in Kibana. Nothing about the logging infrastructure changes. Configuration flows from the workload to the platform, rather than the platform maintaining a registry of workloads — the same inversion Kubernetes label selectors are built on, which is where I recognised the pattern from.

That second label solves a specific annoyance I ran straight into. Docker wraps container stdout in its own JSON envelope, so a carefully structured event arrives as a string stuffed inside a message field — JSON inside JSON, and none of the inner fields are queryable. decode_json_fields with target: "" re-parses that inner string and hoists the fields to the top level. Anyone shipping JSON logs through Docker meets this; knowing the name of the fix saves an afternoon.

Concept 5: Reproducible topology#

Six services with a specific wiring order isn’t something to set up by hand twice. Declaring the topology — dependency order, private network, environment — is what makes the whole thing one command:

services:
  apm:
    depends_on:
      - apm_db
      - filebeat
      - apm-server
    environment:
      SPRING.DATASOURCE.URL: jdbc:mysql://apm_db:3306/APM?createDatabaseIfNotExist=true
    networks:
      - apm-network

Two things this taught me beyond “use Compose.” Service names are DNS inside the network, so jdbc:mysql://apm_db:3306 resolves by name and no IPs appear in config anywhere. And environment-variable overrides mean the same build artifact runs locally and in the stack with only its wiring changed — the twelve-factor rule, and Spring’s relaxed binding (SPRING.DATASOURCE.URL → spring.datasource.url) exists precisely to make it painless.

I built the image with Jib rather than a Dockerfile:

<plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <configuration>
        <from><image>openjdk:8-jdk-alpine</image></from>
        <to><image>${docker.registry.host}/${docker.registry.repository}:${docker.registry.version}</image></to>
    </configuration>
</plugin>

Jib layers dependencies separately from application classes, so a code change rebuilds only the thin top layer while dependencies stay cached. That layering insight generalises to any Dockerfile: order build steps from least-frequently-changed to most.

Design notes: what this stack is scoped for#

I built this to understand the concepts on a laptop, and the configuration reflects that. Worth being explicit about the gap between this and a production deployment, since the delta is itself a useful list:

Elasticsearch runs discovery.type=single-node as root with no authentication. One node means no replicas, and open unauthenticated search is fine on a private Docker network and nowhere else. Production wants a multi-node cluster with TLS and role-based access.
Credentials are inline (MYSQL_ROOT_PASSWORD: root). Fine for a disposable local stack; secrets management is the real answer.
No retention policy. Log volume grows without bound, so index lifecycle management — rollover and delete — is the next thing to add.
No sampling. Tracing every request is exactly right at demo traffic and untenable at scale, which is what head-based and tail-based sampling exist for.
The Logstash output index is logstash-s%{+YYYY.MM.dd}, one character off the conventional logstash- prefix. Harmless in isolation, and a neat demonstration that index naming is a contract: dashboards and lifecycle policies match on those patterns, so the convention is worth pinning down before you have dashboards.

What I took away#

The concepts turned out to be small and to compose cleanly. Structure your events so machines can read them. Stamp every one with the ID of the request that caused it. Let an agent instrument what you’d never wrap by hand. Ship it all somewhere durable and queryable. Get those four right and the 4-second request stops being a mystery — you filter by its trace ID and read the waterfall.

What I hadn’t expected was how much of the work is plumbing decisions rather than tooling choices: UTC or local, nested or flat fields, labels or paths, collect-then-parse or parse-at-source. The brand names change every few years; those decisions don’t.

Backtracking: Solving Problems by Guessing Systematically

Tue, 17 Mar 2020 00:00:00 GMT

An empty Sudoku grid has 81 cells and 9 possible values each. That’s 9⁸¹ — around 10⁷⁷ — grids, comparable to the number of atoms in the observable universe. Yet a laptop solves Sudoku in milliseconds.

The gap between those two facts is backtracking, and once I understood it I started seeing it everywhere. It’s how you search a space too large to enumerate: not by being clever about which answer is right, but by being disciplined about abandoning wrong ones early.

I built a Sudoku solver in the browser mainly so I could watch it happen — the grid updates cell by cell as the algorithm guesses, gets stuck and reverses out. Code: gitlab.com/rustam.niraula90/sudoku-solver.

These are my notes from building it.

Concept 1: A solution space is a tree, and most of it is dead#

The reframing that made this click: stop thinking about 10⁷⁷ complete grids and think about partial ones. Fill the first cell — 9 branches. Fill the second — 9 more each. That’s a tree, and a path from the root to depth 81 is a candidate solution.

The crucial part is that the tree is mostly dead on arrival. If the first two cells both hold a 5 in the same row, every one of the 9⁷⁹ grids below that node is invalid. You never visit them; you don’t even need to know how many there are. Cutting a branch at depth 2 discards an astronomical subtree for the cost of one comparison.

That’s the whole trick. Backtracking is depth-first search plus the discipline of testing validity at every step rather than at the end. The exponential space never materialises because you’re never holding more than one partial candidate at a time.

Concept 2: The constraint check is the real engine#

Everything depends on cheaply answering one question: can this value go in this cell? For Sudoku the rules are row, column and 3×3 box uniqueness:

function possible(x, y, n) {
    for (let i = 0; i < 9; i++) {
        if (puzzle[x][i] == n || puzzle[i][y] == n) {
            return false;
        }
    }
    let x0 = Math.floor(x / 3) * 3;
    let y0 = Math.floor(y / 3) * 3;
    for (let i = 0; i < 3; i++) {
        for (let j = 0; j < 3; j++) {
            if (puzzle[x0 + i][y0 + j] == n) {
                return false;
            }
        }
    }
    return true;
}

Two details I’ve reused many times since:

Row and column share one loop. puzzle[x][i] walks the row while puzzle[i][y] walks the column — same index, two constraints, one pass. Constraint checks run millions of times, so this is the one place in a backtracking solver where micro-optimisation genuinely pays for itself.

The box origin is integer division. Math.floor(x / 3) * 3 snaps any coordinate to the top-left corner of its 3×3 block. Mapping a point to the region containing it by rounding down to a multiple of the region size turns out to be a pattern that shows up constantly — tiling, spatial hashing, pagination.

Concept 3: Guess, recurse, undo#

With a validity test in hand, the solver is remarkably short. Find an empty cell, try each candidate, recurse, and undo the guess if the recursion doesn’t pan out:

async function solve() {
    for (let x = 0; x < 9; x++) {
        for (let y = 0; y < 9; y++) {
            if (puzzle[x][y] == 0) {
                for (let n = 1; n < 10; n++) {
                    if (sudokuSolver.possible(x, y, n)) {
                        puzzle[x][y] = n;                        // guess
                        sudokuSolver.updateViewValue();
                        await new Promise(r => setTimeout(r, sudokuSolver.bufferTime));
                        await sudokuSolver.solve();              // recurse
                        if (!sudokuSolver.isSolved()) {
                            puzzle[x][y] = 0;                    // undo
                        } else {
                            return;                              // done
                        }
                    }
                }
                return;   // no candidate worked — dead end, unwind
            }
        }
    }
}

Three lines carry the idea:

puzzle[x][y] = n — commit to a guess.
await solve() — explore everything that follows from it.
puzzle[x][y] = 0 — restore the state exactly as it was before trying the next candidate.

That third line is the one I’d underestimated. Your state has to be restorable. Here it’s trivial, because a guess writes one cell and undoing is one assignment — and that’s not luck, it’s a consequence of representing the puzzle as a plain mutable grid. In harder problems (graph colouring, N-queens with incremental constraint tables, dependency resolution) you either design the mutation to be cheaply reversible or you pay to copy state at every branch. Picking a representation you can undo is the actual design work.

The bare return at the end of the cell loop is subtle and load-bearing: if no value fits an empty cell, this branch is unsolvable, so stop immediately. That early return is the pruning.

Concept 4: How a recursive search reports success#

My version detects completion by scanning the grid for empty cells:

function isSolved() {
    for (let x = 0; x < 9; x++) {
        for (let y = 0; y < 9; y++) {
            if (puzzle[x][y] == 0) return false;
        }
    }
    return true;
}

Design note. This checks a global condition — “is the whole grid full?” — after each recursive call, rather than having solve() return a boolean that propagates up the stack. Both terminate correctly, and pruning dominates the runtime either way, so the visible behaviour is the same. The difference is in how the stack unwinds: a boolean return lets each of the ~81 frames exit immediately, while a global check re-derives the answer at every frame and restarts its cell scan from (0,0) on each descent.

Writing it the second way and then reasoning about the first is what gave me the principle I now apply by reflex: a recursive function should report its own outcome rather than have its caller re-derive it from shared state. Threading the answer through the return value is what keeps unwinding O(depth) instead of O(depth × grid). It’s a one-word change here, and a habit worth having before the problem is bigger.

Concept 5: Animating a recursive algorithm in a single-threaded language#

This is the part I actually built the project to explore. JavaScript has one thread running both my code and the DOM. A synchronous recursive solve finishes in a few milliseconds and repaints exactly once — you’d see an empty grid, then a finished one, and learn nothing about the search.

To watch it, the algorithm has to hand control back to the browser between steps:

sudokuSolver.updateViewValue();
await new Promise(r => setTimeout(r, sudokuSolver.bufferTime));

new Promise(r => setTimeout(r, ms)) is the idiomatic sleep — a promise that resolves on a timer. Awaiting it suspends the function, returns control to the event loop so the browser can lay out and paint, and resumes where it left off. Cooperative yielding: the algorithm volunteers the thread rather than being preempted.

What makes this practical is that async/await composes through recursion. await solve() inside solve() means a pause at depth 40 correctly suspends all 40 frames above it, and they resume in order. Doing the same with callbacks would mean turning the recursion inside out into an explicit state machine with a manual stack. Coroutines let me keep the natural recursive shape and get the pause, and that realisation has outlived the project — it’s the same reason generators and async iterators feel so useful for streaming work.

The delay is a slider bound to bufferTime, 0 to 500 ms. At 0 the search rips through the grid; at 250 ms you can follow individual guesses and watch it reverse out of a dead end. Making speed a control instead of a constant is what turned this from a visualisation into something I learned from — seeing it stall and unwind is what made the concept concrete in a way pseudocode never did.

Where this idea showed up next#

Sudoku was the friendly introduction. The same four-part structure — candidate generation, constraint check, recursive descent, undo — turned out to be the backbone of:

N-Queens and other placement puzzles
Graph colouring and scheduling under conflict constraints
Regular expression backtracking (also where catastrophic blowup comes from, when the pruning isn’t effective)
SAT solvers, where DPLL is backtracking plus much smarter propagation
Dependency resolution in package managers, picking versions under compatibility constraints

And the natural next step is the same in all of them: better ordering. My solver takes cells in reading order and values in ascending order. Choose the most constrained cell first — fewest legal candidates remaining — and you fail faster, which means you prune higher in the tree. That single heuristic (MRV, minimum remaining values) typically cuts the search by orders of magnitude without changing the structure of the algorithm at all. Adding it is the obvious next iteration here, and it’s cheap: sort the empty cells by candidate count instead of scanning them in order.

Backtracking isn’t a trick for one puzzle. It’s a general answer to “the space is too big to enumerate,” and the answer is: don’t enumerate it — walk it, and leave early.

Genetic Algorithms: Evolving a Solution You Don't Know How to Compute

Wed, 16 Oct 2019 00:00:00 GMT

Some problems you can solve directly. Others you can only recognise a good answer to. Given a route through 50 cities you can measure its length instantly, but computing the shortest one means confronting 50! possibilities. Given a walking gait you can measure how far the robot got, but deriving the optimal joint sequence from first principles is hopeless.

For that second class — evaluating is cheap, deriving is impossible — you can borrow the only algorithm known to have designed a working eye: evolution. Generate random candidates, keep the ones that score better, copy them with small errors, repeat. No gradient, no model of the problem, no idea in advance what a good solution looks like.

I built a browser simulation to see whether that really works: a population of dots, each executing a fixed sequence of random moves, evolving over generations until they find a goal. Code: gitlab.com/rustam.niraula90/genetic-algorithm.

These are my notes from building it and from what surprised me afterwards.

Concept 1: The genome — encode the solution, not the solver#

The first decision in any genetic algorithm is what a candidate solution is as data. I didn’t appreciate how much rides on this until later: get it wrong and nothing else can work.

Here a candidate is an entire pre-committed movement plan — 500 direction codes, generated at random:

export default class Brain {
  constructor() {
    this.step = 0;
    this.size = window.$BRAIN_SIZE;      // 500
    this.directions = [];
    this.mutationRate = (window.$MUTATION_PERCENTAGE / 100);
    this.randomize();
  }

  randomize() {
    for (var i = 0; i < this.size; i++) {
      this.directions[i] = getRndInteger(0, 5);   // 0=stay, 1=left, 2=up, 3=right, 4=down
    }
  }
}

Three properties make this workable, and I’ve since treated them as the general criteria:

Fixed length. Every genome is 500 integers, so any two are structurally interchangeable and mutation can’t produce something malformed.

Every value is legal. A direction code is 0–4 and all five mean something (0 is “stand still”). There’s no such thing as an invalid genome, so mutation never needs repair logic — a large simplification worth designing for deliberately.

No intelligence inside. The dot has no goal-seeking behaviour at all. It cannot see the target. It just replays its list:

move() {
    if (this.brain.directions.length > this.brain.step) {
      this.currentMove = this.brain.directions[this.brain.step];
      this.brain.step++;
    } else {
      this.dead = true;      // out of moves
    }
    switch (this.currentMove) {
      case 1: this.positionX -= window.$ONE_STEP; break;   // left
      case 2: this.positionY -= window.$ONE_STEP; break;   // up
      case 3: this.positionX += window.$ONE_STEP; break;   // right
      case 4: this.positionY += window.$ONE_STEP; break;   // down
      default: break;                                       // stay
    }
}

That last property is what made the result convincing to me. Any competence that emerges came from selection, not from logic I wrote.

Concept 2: Fitness — the only thing you’re allowed to tell it#

The fitness function is your entire communication channel with the algorithm. It isn’t a hint or a heuristic; it’s the definition of “better,” and evolution optimises exactly what you write, including the parts you didn’t mean.

calculateFitness() {
    if (this.reachedGoal) {
      this.fitness = 1 + 1.0 / (this.brain.step * this.brain.step);
    } else {
      var distanceToGoal = this.calculateDistance(this.positionX, this.positionY, 100, 30);
      this.fitness = 1.0 / distanceToGoal;
    }
}

Two regimes, and the structure matters more than the formulas:

Not there yet: 1/distance. Closer is better, and inverting distance turns “minimise” into “maximise” while compressing the scale — halving your distance doubles your score, so improvements near the goal count for far more than the same absolute gain far away.

Arrived: 1 + 1/steps². The 1 + guarantees any dot that reaches the goal outscores every dot that didn’t, no matter how close the near-miss got. That hard tier boundary encodes a priority: first solve it, then optimise it. Inside the winners’ tier, 1/steps² rewards doing it faster, and squaring makes that pressure sharp.

This shape — a discontinuous jump between “task achieved” and “not achieved,” with a smooth gradient inside each tier — is the most reusable thing I took from the project. The smooth part gives the population a slope to climb when nobody has succeeded yet; the jump makes sure partial credit never outranks real success.

The gradient part is not optional. If fitness were simply “1 if you reached the goal, else 0,” the first generation would be all zeroes, selection would have nothing to prefer, and the search would never start. Distance gives evolution something to work with before anyone has won. Designing that intermediate signal is usually the hard part of applying a GA to a real problem, and it’s where I’d expect to spend my time on the next one.

Concept 3: The generational loop#

Evolution is four phases repeated. Run every candidate until it can’t continue; score them; build a new population from the survivors; perturb it.

if (!population.allDotsDead() && !population.complete) {
    population.update();               // 1. simulate this generation
} else {
    population.calculateFitness();     // 2. score everyone
    population.naturalSelection();     // 3. breed the next generation
    population.mutateBabies();         // 4. introduce variation
    population.show();
    population.complete = false;
}

A generation ends when every dot is either dead (out of bounds or out of moves) or has arrived:

update() {
    if (!this.dead && !this.reachedGoal) {
      this.move();
      if (this.positionX <= 1 || this.positionY <= 1 ||
          this.positionX >= 198 || this.positionY >= 398) {
        this.dead = true;
      } else if ((this.positionX > 90 && this.positionX < 105) &&
                 (this.positionY > 20 && this.positionY < 35)) {
        this.reachedGoal = true;
      }
    }
}

Hitting a wall is fatal, which means the constraint is expressed through mortality rather than through the fitness function — an obstacle doesn’t reduce your score, it truncates your run, freezing your position at the wall and therefore your distance. I like this more than I expected to: constraints implemented as early termination avoid the penalty-weight tuning that constraint terms in a fitness function usually drag in.

Concept 4: Mutation is the only source of novelty#

Copying survivors preserves what works. Mutation invents anything new:

mutate() {
    for (var i = 0; i < this.directions.length; i++) {
      var rand = Math.random();
      if (rand < this.mutationRate) {
        this.directions[i] = getRndInteger(0, 5);
      }
    }
}

Each gene independently has a mutationRate chance of being redrawn. The rate is the central tuning knob, and both extremes fail in opposite ways:

Too low → offspring are near-copies, the population converges on a local optimum and stops improving.
Too high → good genomes get scrambled faster than selection can consolidate them, and the search degenerates into random guessing.

I exposed it as a live input in the UI, along with population size, genome length and step length, which turned out to be the best decision in the project — you can watch convergence stall at 2% and watch it thrash at 80%. My default is 30%, which is high next to the conventional 1–5%, and the reason is directly tied to the selection design in the next section.

Concept 5: Elitism — don’t lose the best thing you found#

Because reproduction is random, the best solution in a generation can fail to be reproduced accurately and the population gets worse. Elitism fixes that: copy the champion into the next generation unmutated.

newDots[0] = this.dotList[this.bestDotIndex].getBaby();
newDots[0].isBest = true;
this.bestDot = this.dotList[this.bestDotIndex];

mutateBabies() {
    this.dotList.forEach((dot) => {
      if (!dot.isBest) {
        dot.brain.mutate();     // the elite is deliberately skipped
      }
    });
}

This guarantees monotonic improvement: best-ever fitness can never decrease. Cheap insurance, and nearly every practical GA includes some form of it. The elite is also rendered in a distinct colour, so you can watch the current champion’s path against the cloud of its mutated descendants — the clearest visual in the whole simulation, and the thing that made the mechanism obvious to me.

Design note: what my selection strategy is, and what it costs#

Here’s the part I find most interesting in hindsight. Parents are chosen like this:

selectParent() {
    return this.dotList[this.bestDotIndex];
}

Every child descends from the single best dot. Meanwhile the code computes a total fitness across the population:

calculateFitnessSum() {
    this.fitnessSum = 0;
    for (var i = 0; i < this.dotList.length; i++) {
      this.fitnessSum += this.dotList[i].fitness;
    }
}

A fitness sum has one classical purpose: fitness-proportionate selection, better known as roulette wheel. Pick a random number in [0, fitnessSum), walk the population accumulating fitness, take whoever you land on. A dot with twice the fitness is twice as likely to be a parent, but a mediocre dot still gets a chance. I built the odometer while working out how selection should behave, and shipped the simpler strategy — pure elitist selection — because it was enough to solve the maze.

What I have, then, is closer to parallel hill climbing: one parent, cloned a thousand times, each copy randomly perturbed, keep the best, repeat. It works, and having both designs side by side in my head is what finally explained what proportionate selection contributes:

Diversity as insurance against local optima. With one parent, the whole population sits in one neighbourhood of the search space. If that neighbourhood is a dead end — a path hugging a wall that can’t be extended without dying — the only escape is a lucky mutation from that single lineage. With proportionate selection the second- and fifth- and fiftieth-best dots also reproduce, so the population explores several regions at once, and one of them may route around the obstacle that trapped the leader.

Ancestral variety, which is what crossover needs. Full GAs also combine parents: first half of one genome, second half of another. That only helps if the parents are meaningfully different. Descend everyone from one ancestor and crossover has nothing to mix — which is why there’s no crossover in this version. The two choices are linked, and I only saw that after the fact.

It also explains my 30% mutation rate. With a population collapsed to one lineage, mutation is the only source of diversity, so it has to carry a load that selection would normally share. That gave me the sharpest lesson of the project: a genetic algorithm’s power comes from maintaining a diverse population, not from mutation volume. Adding roulette-wheel selection and single-point crossover — both small changes, and the sum is already computed — should let the mutation rate drop by an order of magnitude while searching better, because variation would come from having many good ideas instead of from repeatedly perturbing one. That’s the next iteration.

When to reach for this#

Evolutionary search fits when:

You can evaluate a candidate but not derive the best one.
There’s no usable gradient — the space is discrete, or the objective isn’t differentiable.
Very good is acceptable; provably optimal isn’t required.

And it’s the wrong tool when a gradient exists (use it — it’s vastly faster), when evaluation is expensive (thousands of candidates × many generations gets brutal), or when the problem has structure a purpose-built algorithm exploits.

What made this worth building wasn’t the maze getting solved. It was watching a thousand dots spray outward in generation one and, by generation forty, trace a coherent path to a goal none of them could see — carrying nothing but a list of 500 random numbers, filtered by the only feedback they ever got.