Binary Quantization for pgvector with bit Columns

This page shows how to binary-quantize embeddings into pgvector bit columns and use them for a fast, memory-cheap first pass in a two-stage retrieval design. It scopes the problem narrowly: how to build an HNSW index over binary_quantize(embedding) with bit_hamming_ops, retrieve a cheap Hamming shortlist, and then rerank that shortlist by exact fp32 distance so recall survives the ~32x storage reduction.

Up: Quantization & halfvec Storage

Binary quantization replaces each fp32 coordinate with a single sign bit: positive becomes 1, non-positive becomes 0. A 1536-dimensional vector that occupies 4*1536 + 8 = 6152 bytes collapses to a bit(1536) of 192 bytes plus overhead — a roughly 32x reduction in the indexed payload, and Hamming distance over bit strings is a popcount on XOR that is dramatically cheaper than fp32 arithmetic. The catch is a recall cliff: a single-bit-per-dimension representation is too coarse to rank final results directly. The fix is two-stage retrieval — use the bit index to pull a generous shortlist by Hamming distance, then rerank that shortlist against the exact fp32 vectors. This page sits under quantization and halfvec storage; if you are still choosing between HNSW and IVFFlat for the bit index, see HNSW vs IVFFlat algorithm selection.

Prerequisites #

• PostgreSQL 15+ with pgvector 0.7.0 or newer — binary_quantize(), the bit operator classes, and bit_hamming_ops all require 0.7+.
• An existing fp32 embedding vector(1536) column retained as the source of truth; binary quantization never replaces it, because the rerank stage needs exact distances.
• Embeddings normalized in fp32 (the sign pattern is stable, but keeping the fp32 column exact matters for the rerank), per normalizing embeddings before pgvector insertion.
• A recall@k harness and ground-truth query set, so you can measure recall with and without reranking.
• A storage baseline for the fp32 table and index, established with pgvector storage overhead analysis, to quantify what binary quantization actually saves.

Step-by-step procedure #

1. Build a bit index without a new stored column #

Use an expression index over binary_quantize(embedding)::bit(1536) so you keep the fp32 column intact and get the bit index for free. binary_quantize returns a bit string; the explicit ::bit(1536) cast fixes the length so the operator class can index it.

-- fp32 column stays the source of truth; the index is a bit expression
CREATE INDEX doc_chunks_bq_idx ON doc_chunks
    USING hnsw ((binary_quantize(embedding)::bit(1536)) bit_hamming_ops);

2. Retrieve a cheap Hamming shortlist #

Query the bit index with the Hamming distance operator <~>, quantizing the probe vector the same way. Pull a shortlist far larger than your final k — a few hundred candidates — because the coarse bit ranking will not place the true nearest neighbors first.

-- stage 1: coarse shortlist by Hamming distance over the bit index
SELECT doc_id, chunk_index
FROM doc_chunks
ORDER BY binary_quantize(embedding)::bit(1536)
         <~> binary_quantize($1::vector(1536))::bit(1536)
LIMIT 200;

3. Rerank the shortlist by exact fp32 distance #

Wrap stage 1 in a CTE, then reorder only those candidates by the exact fp32 cosine distance <=> against the retained vector column. The expensive fp32 math runs on a few hundred rows, not the whole table, so it is cheap while restoring the ranking the bit stage lost.

-- two-stage retrieval: coarse Hamming shortlist, then exact fp32 rerank
WITH shortlist AS (
    SELECT doc_id, chunk_index, embedding
    FROM doc_chunks
    ORDER BY binary_quantize(embedding)::bit(1536)
             <~> binary_quantize($1::vector(1536))::bit(1536)
    LIMIT 200                              -- rerank_shortlist
)
SELECT doc_id, chunk_index,
       embedding <=> $1::vector(1536) AS distance
FROM shortlist
ORDER BY embedding <=> $1::vector(1536)   -- exact fp32 rerank
LIMIT 10;                                  -- final k

4. Tune the shortlist size against recall #

The shortlist multiplier — rerank_shortlist / k — is the recall lever. Too small and the true neighbors never enter the rerank; too large and you erode the speed advantage. Sweep it with a recall@k measurement against fp32 ground truth and pick the smallest multiplier that plateaus.

def recall_at_k(approx_ids, truth_ids, k: int) -> float:
    hits = sum(len(set(a[:k]) & set(t[:k])) for a, t in zip(approx_ids, truth_ids))
    return hits / (k * len(truth_ids))
# sweep shortlist in {50, 100, 200, 400}; keep the smallest that plateaus recall

Parameter reference #

Verification #

Prove the storage saving landed, and measure recall with and without the rerank so the cliff is visible.

-- bit index size vs the fp32 index, to confirm the ~32x reduction
SELECT indexrelname, pg_size_pretty(pg_relation_size(indexrelid)) AS idx_size
FROM pg_stat_user_indexes
WHERE relname = 'doc_chunks'
ORDER BY pg_relation_size(indexrelid) DESC;

-- stage-1-only ranking (NO rerank) — expect a visible recall drop vs fp32
SELECT doc_id, chunk_index
FROM doc_chunks
ORDER BY binary_quantize(embedding)::bit(1536)
         <~> binary_quantize($1::vector(1536))::bit(1536)
LIMIT 10;

Run recall_at_k on the stage-1-only result and on the two-stage result against fp32 ground truth. The gap between them is the recall the rerank recovers; a healthy setup shows the two-stage recall landing close to the fp32 baseline while the stage-1-only recall sits well below it.

Troubleshooting #

• Recall falls off a cliff. You are ranking on bit/<~> alone. Single-bit quantization cannot order final results; wrap stage 1 in the CTE and rerank by exact <=> on the fp32 column (step 3). Reranking is not optional for binary quantization.
• Recall is still low even with reranking. The shortlist is too small, so the true neighbors never reach the rerank. Increase rerank_shortlist (and, if needed, hnsw.ef_search on the bit index) and re-sweep recall@k until it plateaus, per step 4.
• operator <~> is not supported or opclass errors at index creation. A Hamming/cosine mismatch: bit_hamming_ops must pair with <~>, and the indexed expression must be binary_quantize(embedding)::bit(d). Using <=> against the bit index, or vector_cosine_ops on a bit expression, is rejected.
• function binary_quantize(vector) does not exist. The pgvector build predates 0.7. Install 0.7+ and ALTER EXTENSION vector UPDATE; — binary quantization, the bit opclasses, and <~> all arrived in 0.7.
• The index barely saved space. You added a second stored bit column and kept a redundant fp32 index, or the heap dominates the footprint. Use the expression index (step 1) rather than a materialized column, and baseline the real breakdown with pgvector storage overhead analysis; the ~32x figure is the indexed vector payload, not the whole table.

Related #

• Quantization & halfvec Storage — the storage-reduction stage this technique belongs to
• HNSW vs IVFFlat algorithm selection — choosing the index type for the bit shortlist stage
• pgvector storage overhead analysis — baselining what binary quantization actually saves
• Normalizing embeddings before pgvector insertion — keeping the fp32 column exact for the rerank stage
• Up: Quantization & halfvec Storage