Building a Ground-Truth Recall Test Set

This page shows how to construct an exact nearest-neighbour ground-truth set for pgvector recall testing: a seeded, stratified sample of query vectors paired with their true top-k neighbour IDs, computed by brute force and versioned by embedding model and data snapshot. It scopes the problem narrowly: how to build the golden neighbours correctly, where to store them, and when to regenerate them so a stale set never lies to your CI gate.

Up: Recall Regression Testing in CI

A recall benchmark is only as trustworthy as the ground truth it compares against. If the golden neighbours were computed with the index enabled, or against a different embedding model than production runs, the recall number is fiction — it can read 0.99 while real search quality has collapsed. The ground-truth set is therefore a deliberately versioned artifact: exact neighbours, computed with the index disabled, keyed to the exact model version and data snapshot they describe, with an explicit refresh policy. The benchmark that consumes it is described in automating recall@k benchmarks in GitHub Actions; this page is about producing the golden set it reads.

Prerequisites #

• A pgvector corpus representative of production, or a sampled snapshot of it, loaded into a table you can query.
• The embedding model identifier and version that produced the vectors — you will key the golden set on it. A model swap invalidates the entire set.
• Python 3.11+ with psycopg and numpy.
• Enough time or compute for a full brute-force pass. Exact ground truth is O(n·q·d); budget it deliberately rather than discovering the cost in CI.
• A metric decided in advance so the ground-truth distance operator matches the one production queries use.

Step-by-step #

1. Draw a seeded, stratified query sample #

Random uniform sampling over-represents dense regions of the embedding space and misses the tail queries where recall actually breaks. Stratify by a meaningful attribute — document cluster, source, or language — and draw a fixed count per stratum with a seeded RNG so the sample is reproducible.

import numpy as np

def stratified_query_sample(ids, strata, per_stratum=40, seed=7):
    rng = np.random.default_rng(seed)
    ids, strata = np.asarray(ids), np.asarray(strata)
    picked = []
    for s in np.unique(strata):
        pool = ids[strata == s]
        take = min(per_stratum, len(pool))
        picked.extend(rng.choice(pool, size=take, replace=False))
    return sorted(picked)

2. Compute exact neighbours with the index disabled #

For each sampled query, get the true top-k by forcing a brute-force scan. Disabling enable_indexscan and enable_bitmapscan for the session guarantees the plan is a full Seq Scan, so the result is exact rather than the index’s own approximation. If the corpus is small you can instead compute this entirely in NumPy; both must agree.

import psycopg

def exact_neighbours(conn, query_ids, k=10):
    golden = {}
    with conn.cursor() as cur:
        cur.execute("SET enable_indexscan = off")
        cur.execute("SET enable_bitmapscan = off")
        for qid in query_ids:
            cur.execute(
                "SELECT id FROM items "
                "WHERE id <> %s "
                "ORDER BY embedding <#> (SELECT embedding FROM items WHERE id = %s) "
                "LIMIT %s",
                (qid, qid, k),
            )
            golden[qid] = [r[0] for r in cur.fetchall()]
    return golden

3. Key and store the golden set #

Persist the neighbours alongside the identity of what they describe: the embedding model version and a data snapshot identifier. Without that key, a later model swap silently reuses neighbours computed for the old vectors. Store it as a table for in-database joins, or as a JSON/.npy file checked into the test fixtures.

CREATE TABLE recall_ground_truth (
    model_version text  NOT NULL,
    snapshot_id   text  NOT NULL,
    query_id      int   NOT NULL,
    k             int   NOT NULL,
    neighbor_ids  int[] NOT NULL,
    generated_at  timestamptz DEFAULT now(),
    PRIMARY KEY (model_version, snapshot_id, query_id, k)
);

def store_golden(conn, model_version, snapshot_id, golden, k):
    with conn.cursor() as cur:
        for qid, nbrs in golden.items():
            cur.execute(
                "INSERT INTO recall_ground_truth "
                "(model_version, snapshot_id, query_id, k, neighbor_ids) "
                "VALUES (%s, %s, %s, %s, %s) "
                "ON CONFLICT (model_version, snapshot_id, query_id, k) "
                "DO UPDATE SET neighbor_ids = EXCLUDED.neighbor_ids, "
                "generated_at = now()",
                (model_version, snapshot_id, qid, k, nbrs),
            )
    conn.commit()

4. Define and enforce a refresh policy #

Regenerate the golden set whenever the thing it describes changes: a new embedding model, a re-chunking, or a corpus mutation past a drift threshold. Encode the trigger as a comparison between the current model/snapshot identity and the stored key, and fail loudly when they diverge.

def needs_refresh(conn, current_model, current_snapshot):
    with conn.cursor() as cur:
        cur.execute(
            "SELECT DISTINCT model_version, snapshot_id FROM recall_ground_truth")
        stored = cur.fetchall()
    return (current_model, current_snapshot) not in stored

Categorizing which changes should invalidate the set — a benign metadata edit versus a genuine embedding change — follows the same taxonomy used for index validation and error categorization.

Parameter reference #

Verification #

Confirm the golden set is exact and complete before any benchmark reads it. Re-run one query through the brute-force path and check the stored neighbours match, and verify every sampled query has a row for the required k.

def verify_golden(conn, model_version, snapshot_id, query_ids, k):
    with conn.cursor() as cur:
        cur.execute(
            "SELECT count(DISTINCT query_id) FROM recall_ground_truth "
            "WHERE model_version=%s AND snapshot_id=%s AND k=%s",
            (model_version, snapshot_id, k))
        stored = cur.fetchone()[0]
    assert stored == len(query_ids), f"golden set incomplete: {stored}/{len(query_ids)}"

-- sanity: no neighbour list is shorter than k (a truncated brute-force pass)
SELECT query_id, cardinality(neighbor_ids) AS n
FROM recall_ground_truth
WHERE k = 10 AND cardinality(neighbor_ids) < 10;

An empty SQL result and a passing Python assertion mean the set is complete and every query has its full exact neighbour list.

Troubleshooting #

• Recall silently jumps after a model deploy. The golden set still carries the old model_version, so it grades new vectors against stale neighbours. Enforce needs_refresh in CI and regenerate on any model change before comparing recall.
• Ground truth is not reproducible between regenerations. The query sample was unseeded or an unstable ORDER BY broke ties arbitrarily. Fix the RNG seed and add a deterministic tiebreak (ORDER BY embedding <#> probe, id) so equal-distance neighbours sort consistently.
• A full brute-force pass is too expensive to run. Reduce scope, not correctness: shrink the corpus to a representative stratified snapshot and cap per_stratum, but never substitute the index’s own results for ground truth. Cache the golden set as an artifact so the cost is paid once per snapshot, not per CI run.
• Some queries return fewer than k neighbours. The corpus is smaller than k after the id <> qid self-exclusion, or duplicate vectors collapsed ranks. Confirm the corpus size exceeds k and dedupe before sampling.
• Golden neighbours disagree between the SQL and NumPy paths. A metric or normalization mismatch — the SQL used <#> on unnormalized vectors while NumPy assumed cosine. Normalize identically in both and use the operator that matches the production query, then regenerate.

Related #

• Automating recall@k benchmarks in GitHub Actions — the CI job that compares index results to this golden set
• Index validation and error categorization — deciding which changes should invalidate the set
• HNSW vs IVFFlat algorithm selection — the index type whose recall this set measures
• A Grafana dashboard for vector search latency — where the resulting recall value is surfaced over time
• Up: Recall Regression Testing in CI