Recall Regression Testing in CI

Approximate nearest-neighbor search trades exactness for speed, which means recall — the fraction of the true nearest neighbors an index actually returns — is a property you can only measure, never assume. A green build that compiled the schema and inserted fixtures tells you nothing about whether hnsw.ef_search is still high enough after the corpus tripled, whether a bumped ivfflat.probes quietly halved accuracy, or whether swapping the embedding model shifted the vector space out from under a tuned index. Recall regressions are silent: latency looks fine, queries return results, and users simply get subtly worse answers until someone notices weeks later. The fix is to treat recall as a testable contract — compute exact brute-force ground truth, compare the approximate index against it over a held-out query sample, assert a threshold, and fail the build on a regression. This page builds that gate end to end: the two ground-truth strategies and when each applies, the recall_at_k implementation, the SQL that forces an exact scan and proves it did so, and the GitHub Actions wiring that runs it against an ephemeral pgvector instance on every pull request.

Up: Monitoring, Observability & Operations

Architectural Divergence & Trade-offs #

Every recall test needs a reference answer — the exact nearest neighbors — to compare the approximate index against. There are two ways to obtain it, and the choice shapes how fast the test runs and how often it breaks for the wrong reason.

Compute ground truth on the fly with a forced exact scan. For each probe query, run the same ORDER BY embedding <=> $1 search but force PostgreSQL to ignore the ANN index, so it computes the true distance to every row and returns the mathematically exact top-k. You force this by disabling index and bitmap scans for the session (SET LOCAL enable_indexscan = off; SET LOCAL enable_bitmapscan = off;) or, more bluntly, by dropping the index in the test database. The result is always correct by construction and always current with the fixture data, because it is recomputed from the same rows the index was built on. The cost is time: a brute-force scan is O(N) per query, so ten thousand probes over a million-row corpus is a serious CI expense.

Cache a golden set of ground-truth neighbor IDs. Precompute the exact top-k once, store the neighbor IDs alongside each probe query, and check them into the repository or an artifact store. The test then only runs the approximate query and compares against the cached IDs — no brute-force scan in the hot path, so it is dramatically faster and CI cost is flat regardless of corpus size. The catch is invalidation: the golden set is only valid for the exact (corpus, embedding model) pair it was generated from. Add, remove, or re-embed rows, or switch the model that produces the vectors, and the cached neighbors are stale — comparing a new index against them measures nothing. A golden set therefore needs a fingerprint (a hash of the corpus contents and the model version) and a regeneration step that fires whenever either changes.

Most teams run both at different cadences: the fast golden-set gate on every pull request for a quick signal, and the slower on-the-fly computation nightly or on the branch that touches ingestion or the embedding model, where regenerating the golden set is part of the job. Whichever you use, the approximate query in the test must run with the same hnsw.ef_search (or ivfflat.probes) you run in production — otherwise the number you gate on describes a search configuration no user will ever experience.

Parameter Space & Diagnostic Workflow #

A recall test has its own set of knobs, and getting them wrong produces a number that is either meaningless or non-reproducible. The sample must be large enough that recall is statistically stable, the search parameters must mirror production, and every source of randomness must be seeded.

The single most important diagnostic is proving the ground-truth query is actually exact — that PostgreSQL did a sequential scan and computed every distance, rather than quietly using the ANN index and handing you approximate results as if they were the truth. Wrap the ground-truth query and inspect its plan:

BEGIN;
SET LOCAL enable_indexscan   = off;
SET LOCAL enable_bitmapscan  = off;
SET LOCAL enable_indexonlyscan = off;

EXPLAIN (ANALYZE, BUFFERS)
SELECT id
FROM items
ORDER BY embedding <=> '[0.12, -0.03, ...]'::vector
LIMIT 10;
COMMIT;

The plan must contain a Seq Scan on items feeding a Sort (or top-N heapsort) — never Index Scan using items_embedding_hnsw. If the HNSW index appears despite the enable_* settings (it can, if only enable_indexscan was toggled and the ANN path is reached another way), drop the index in the test database instead. Then fetch the exact and approximate result sets side by side to see the difference concretely:

-- Exact ground truth (index disabled for this transaction)
SET LOCAL enable_indexscan = off;
SET LOCAL enable_bitmapscan = off;
SELECT array_agg(id ORDER BY dist) AS truth_ids
FROM (
  SELECT id, embedding <=> $1 AS dist
  FROM items ORDER BY embedding <=> $1 LIMIT 10
) t;

-- Approximate result at the production search width
SET LOCAL hnsw.ef_search = 40;
SELECT array_agg(id ORDER BY dist) AS approx_ids
FROM (
  SELECT id, embedding <=> $1 AS dist
  FROM items ORDER BY embedding <=> $1 LIMIT 10
) t;

The overlap between truth_ids and approx_ids, averaged over the whole probe sample, is recall@k. A per-query overlap that is occasionally very low while the mean looks acceptable points at a few pathological queries sitting near dense partition boundaries in the vector space — worth inspecting before you accept the aggregate.

Step-by-Step Implementation #

The test has five stages: fix a reproducible probe sample, capture exact ground truth, run the approximate search at production parameters, compute recall in Python, and gate the merge in CI.

Step 1 — Build a seeded, held-out probe sample. Reproducibility starts here: the same queries must be chosen on every run, so seed the sampler and store the chosen IDs rather than re-sampling.

import numpy as np

def sample_probe_ids(all_ids: np.ndarray, n: int = 10_000, seed: int = 42):
    rng = np.random.default_rng(seed)               # deterministic across runs
    return rng.choice(all_ids, size=n, replace=False)

Step 2 — Capture exact ground truth with the index disabled. For the on-the-fly strategy, run the forced-exact query per probe; for the golden-set strategy, run this once and persist the result. Either way, verify the plan is a sequential scan before trusting the output.

async def ground_truth(conn, qvec: str, k: int = 10) -> list[int]:
    async with conn.transaction():
        await conn.execute("SET LOCAL enable_indexscan = off")
        await conn.execute("SET LOCAL enable_bitmapscan = off")
        rows = await conn.fetch(
            "SELECT id FROM items ORDER BY embedding <=> $1 LIMIT $2", qvec, k
        )
    return [r["id"] for r in rows]

Step 3 — Run the approximate search at the production search width. Set hnsw.ef_search (or ivfflat.probes) to the exact value production uses — anything else measures a configuration you do not ship.

async def approx_neighbors(conn, qvec: str, k: int = 10, ef_search: int = 40):
    async with conn.transaction():
        await conn.execute(f"SET LOCAL hnsw.ef_search = {ef_search}")
        rows = await conn.fetch(
            "SELECT id FROM items ORDER BY embedding <=> $1 LIMIT $2", qvec, k
        )
    return [r["id"] for r in rows]

Step 4 — Compute recall@k and assert the threshold. Compare ID sets, never row order — two results with the same members in a different order have perfect recall. Enforce both an absolute floor and a regression tolerance against a stored baseline.

import numpy as np

def recall_at_k(approx: list[list[int]], truth: list[list[int]], k: int) -> float:
    hits = total = 0
    for a, t in zip(approx, truth):
        truth_set = set(t[:k])
        hits  += len(set(a[:k]) & truth_set)
        total += len(truth_set)
    return hits / total

BASELINE  = 0.972     # last accepted recall@10, checked into the repo
FLOOR     = 0.95      # absolute minimum
TOLERANCE = 0.02      # max allowed drop vs baseline

recall = recall_at_k(approx_results, truth_results, k=10)
assert recall >= FLOOR, f"recall@10 {recall:.4f} below floor {FLOOR}"
assert recall >= BASELINE - TOLERANCE, (
    f"recall@10 regressed {BASELINE - recall:.4f} (> {TOLERANCE}) vs baseline {BASELINE}"
)
print(f"recall@10 = {recall:.4f}  (floor {FLOOR}, baseline {BASELINE})")

Step 5 — Wire it into GitHub Actions with an ephemeral pgvector service. Use the official pgvector/pgvector image as a service container, health-gate it, load the fixture, build the index, and run the test as a required check so a failure blocks the merge.

name: recall-regression
on: pull_request
jobs:
  recall:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: pgvector/pgvector:pg16
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: vectors
        ports: ["5432:5432"]
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s --health-timeout 5s --health-retries 12
    env:
      PGHOST: localhost
      PGUSER: postgres
      PGPASSWORD: postgres
      PGDATABASE: vectors
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install "psycopg[binary]" asyncpg numpy
      - run: python scripts/load_fixture.py        # corpus + seeded probe set
      - run: python scripts/build_index.py          # CREATE INDEX ... USING hnsw
      - run: python scripts/recall_test.py          # exits non-zero on regression

Marking the recall job a required status check in branch protection is what turns the assertion into a gate. For the index-build step itself and the error classes it can hit, see index validation and error categorization; the recall you measure here is only trustworthy if that build completed cleanly.

Validation & Recall Testing #

A recall gate is only as good as its own correctness — a test that silently measures the wrong thing is worse than none, because it manufactures confidence. Validate the harness before you trust its verdict.

Prove the ground truth is exact. Run the ground-truth query under EXPLAIN (ANALYZE) inside the same disabled-scan transaction the test uses and assert the plan text contains Seq Scan and not the HNSW index name. Automate this as the first assertion in the test so a configuration slip fails loudly rather than corrupting the recall number:

EXPLAIN (FORMAT JSON)
SELECT id FROM items ORDER BY embedding <=> '[...]'::vector LIMIT 10;
-- Assert the plan's "Node Type" is "Seq Scan", not "Index Scan".

Prove determinism. Run the whole test twice in the same CI job; the two recall values must be identical to several decimal places. Any variation means an unseeded random source has leaked in — a re-sampled probe set, a non-deterministic tie-break, or parallel workers reordering results. Seed every generator and sort deterministically.

Prove the gate actually catches regressions. Add a negative test that lowers hnsw.ef_search to a value known to break recall and assert the harness fails. A gate that has never been observed to fail on a real regression is untested. This mirrors the production side, where the recall Gauge exported to Prometheus catches drift that only shows up under real data growth — the CI gate blocks bad code before merge, the runtime metric catches erosion after deploy.

Sanity-check the sample size. If recall wobbles more than the tolerance across two seeds, the probe sample is too small to gate on; increase it until the run-to-run variance is well below your 2% regression threshold. Ten thousand probes is a reasonable floor; boundary-heavy corpora may need more.

Failure Modes & Gotchas #

• Ground truth accidentally uses the index. Setting only enable_indexscan = off can still leave an ANN path reachable, or a later statement in the same session resets it. The result: your “exact” reference is itself approximate, and recall reads as a suspiciously perfect 1.0. Always verify the plan is a Seq Scan, and prefer dropping the index in the test database when in doubt.
• Non-deterministic query sampling. Re-sampling the probe set each run with an unseeded RNG makes recall jitter between builds, producing flaky failures that erode trust in the gate. Seed the sampler and persist the chosen IDs so every run measures the identical query set.
• Model-version drift invalidates the golden set. A cached ground-truth set is bound to the embedding model that produced the vectors. Upgrade the model — even a minor version — and the true nearest neighbors change, so comparing a new index against old golden IDs measures noise. Fingerprint the golden set with the model version and corpus hash and regenerate on any change.
• Test and production search parameters disagree. Measuring recall at hnsw.ef_search = 200 while production runs at 40 reports a number no user experiences — usually far too optimistic. Pin the test to the exact production ef_search / ivfflat.probes values, ideally read from the same config source.
• Comparing ordered lists instead of ID sets. Recall is about set membership among the top-k, not ordering. Comparing lists element-wise understates recall whenever the approximate and exact results contain the same neighbors in a different order. Always intersect sets.
• Too-small a probe sample. A few hundred queries gives a recall estimate with enough variance to swing across your regression tolerance by chance, causing random red builds. Use at least ten thousand probes so the mean is stable.
• Baseline never updated. If an intentional, accepted change lowers recall slightly, the baseline must be bumped in the same PR or every subsequent build fails against a number the team already agreed to leave behind.

Monitoring & Alerting Hooks #

The CI gate and production monitoring are two ends of the same measurement, and they share the same ground-truth machinery. In CI, the signal is a pass/fail plus the recorded recall number; export that number rather than discarding it so recall trends are visible over time. A minimal approach pushes the CI recall to a Prometheus Pushgateway or writes it to a metrics table keyed by commit:

CREATE TABLE IF NOT EXISTS recall_history (
    commit_sha  text        NOT NULL,
    index_name  text        NOT NULL,
    k           int         NOT NULL,
    recall      double precision NOT NULL,
    ef_search   int,
    measured_at timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (commit_sha, index_name, k)
);

Plotting recall from this table against measured_at reveals slow erosion that no single build would flag — the case where each PR drops recall by 0.3%, individually under the 2% tolerance, but cumulatively catastrophic over a quarter. That is the same failure the runtime recall Gauge in the Prometheus metrics setup is designed to catch on the production side, and the two should agree: if CI reports recall@10 of 0.97 but production reports 0.90, the fixture corpus no longer resembles production data and the gate is protecting the wrong distribution.

In CI itself, keep the EXPLAIN-based ground-truth verification as a hard assertion and surface the per-query recall distribution, not just the mean, so a build can flag when a small set of probes collapses even while the average holds. When a regression does fire, the first triage question is whether the index build or the search width changed — the algorithm-level trade-offs behind that decision are laid out in HNSW vs IVFFlat algorithm selection, and the parameter interactions that most often move recall are in optimizing m and ef_construction parameters.

FAQ #

Related #

• Prometheus metrics for pgvector search — the runtime recall Gauge that catches drift the CI gate cannot see
• HNSW vs IVFFlat algorithm selection — the index choice a recall regression most often traces back to
• Index validation and error categorization — verifying the build the recall test depends on
• Optimizing m and ef_construction parameters — the parameters that move measured recall
• Up: Monitoring, Observability & Operations