Quantization and halfvec Storage for pgvector

A 1536-dimension embedding stored as pgvector’s default vector type costs 4 × 1536 + 8 = 6152 bytes per row, and at ten million rows that is roughly 60 GB of vector payload before you add a single index — an HNSW graph on top can double it again. Most of those bytes are precision you do not need: modern embedding models produce values whose semantic content survives being stored at half precision, and much of the retrieval signal survives being stored as a single bit per dimension. pgvector 0.7 exposes this directly through halfvec (fp16, half the bytes, negligible recall loss) and bit columns with binary_quantize() (32× smaller, but a recall cliff unless you rerank). Choosing the right storage type — and, critically, ordering normalization before the downcast — is the highest-leverage decision for controlling memory and cost in a large pgvector deployment. This page compares the storage types, shows how to index halfvec through an expression index without giving up your fp32 source, builds the two-stage binary-quantize-then-rerank pattern, and validates that the recall you traded away is the recall you meant to trade. It sits within pgvector Architecture & Vector Fundamentals.

Up: pgvector Architecture & Vector Fundamentals

Architectural Divergence & Trade-offs #

pgvector offers four storage representations for a dense embedding, and they span two orders of magnitude in size at very different recall costs. The decision is not “smallest wins” — it is how much recall you can trade, and whether you are willing to add a rerank stage to buy that recall back.

vector (fp32) is the default and the reference point: 4 × d + 8 bytes per row, full precision, works with vector_cosine_ops, vector_l2_ops, and vector_ip_ops. It is the correct source of truth and the type you rerank against.

halfvec (fp16) stores each component in two bytes — 2 × d + 8 per row, almost exactly half the storage and, crucially, half the index working set. For embeddings from current models the recall loss versus fp32 is negligible because fp16 still carries about three decimal digits of precision, far more than the semantic signal requires. It needs pgvector 0.7+ and the halfvec_cosine_ops / halfvec_l2_ops operator classes. This is the type that should be your default at scale.

bit (binary quantization) stores one bit per dimension via binary_quantize(), which maps each component to 1 if positive and 0 otherwise — d / 8 bytes plus overhead, roughly 32× smaller than fp32. Distance is Hamming (via bit_hamming_ops), which is extremely fast, but a single bit per dimension discards magnitude entirely, so recall collapses if you use it alone. It is only viable as the first stage of a two-stage retrieval that reranks the shortlist against full-precision vectors.

sparsevec is a separate axis: it stores only non-zero components as index/value pairs, which is the right choice for genuinely sparse representations (SPLADE, bag-of-words style vectors) but wasteful for the dense embeddings this page is about. Note it here only so it is not confused with the dense-quantization options.

For most deployments the progression is: start on vector, move the index (and often the column) to halfvec once storage or build memory becomes the constraint, and adopt binary quantization plus rerank only when you need to serve a very large corpus from a memory budget that fp16 cannot meet. The type-selection reasoning that precedes this — including when a different base type is warranted — is developed in vector data type selection, and the raw byte accounting that these formulas come from is worked through in pgvector storage overhead analysis.

Parameter Space & Diagnostic Workflow #

The levers here are the storage type, the operator class it forces, and — for the two-stage pattern — the shortlist size that trades rerank cost against recall. The one non-negotiable ordering rule is that any L2 normalization must happen at full precision before the cast to a narrower type, because normalizing after the downcast bakes the quantization error into the unit vector and cannot be undone.

Before choosing a type, measure what the current representation actually costs on disk. pg_relation_size on the table and its index tells you where the bytes are and what a downcast would save:

SELECT pg_size_pretty(pg_relation_size('doc_embeddings'))               AS heap,
       pg_size_pretty(pg_relation_size('doc_embeddings_embedding_idx')) AS index,
       pg_size_pretty(pg_total_relation_size('doc_embeddings'))         AS total,
       count(*)                                                         AS rows
FROM doc_embeddings;

To confirm a halfvec expression index is the one actually serving queries — rather than a leftover fp32 index — inspect the index definitions and their sizes side by side. If both exist, the planner will pick whichever the query’s cast matches, so an unexpectedly large plan usually means the query is not casting to the half-precision type the index was built on:

SELECT indexrelid::regclass                        AS index_name,
       pg_size_pretty(pg_relation_size(indexrelid)) AS size,
       indexdef
FROM pg_stat_user_indexes
JOIN pg_indexes ON indexname = indexrelid::regclass::text
WHERE relname = 'doc_embeddings';

Step-by-Step Implementation #

The following adopts halfvec for the index while keeping an fp32 source column, then layers the binary-quantize-and-rerank pattern on top. Each step is runnable on pgvector 0.7+.

Step 1 — Normalize at full precision, then store. Cosine similarity is only equivalent to inner product on unit vectors, and quantization amplifies any normalization error, so normalize the fp32 vector before it ever touches a narrower type. The full derivation of why order matters lives in normalizing embeddings before pgvector insertion.

import numpy as np

def to_unit_fp32(vec) -> list[float]:
    a = np.asarray(vec, dtype=np.float32)     # normalize at fp32...
    n = np.linalg.norm(a)
    if n == 0.0:
        raise ValueError("zero vector cannot be normalized")
    return (a / n).tolist()                   # ...cast to narrower type happens in SQL

Step 2 — Keep the fp32 column, index it as halfvec via an expression. You do not need to convert the stored column to gain the fp16 index savings. An expression index on embedding::halfvec(1536) builds the HNSW graph in half-precision — halving the index size and the build’s memory working set — while the heap keeps the fp32 source for exact rerank.

-- source column stays fp32
CREATE TABLE doc_embeddings (
    doc_id      text NOT NULL,
    chunk_index int  NOT NULL,
    embedding   vector(1536) NOT NULL,
    PRIMARY KEY (doc_id, chunk_index)
);

-- HNSW graph built in fp16 through an expression index
CREATE INDEX doc_embeddings_hv_idx
    ON doc_embeddings
    USING hnsw ((embedding::halfvec(1536)) halfvec_cosine_ops);

Step 3 — Query with the matching cast. The planner only uses the expression index when the query casts the same way. Order by the fp16 distance to hit the index; the result ordering is effectively identical to fp32 for typical embeddings.

SELECT doc_id, chunk_index
FROM doc_embeddings
ORDER BY embedding::halfvec(1536) <=> $1::halfvec(1536)
LIMIT 10;

Because the index working set is now half the size, more of it fits in shared buffers, which also relaxes the memory pressure that dominates large HNSW builds — the same pressure tuned through optimizing m and ef_construction parameters.

Step 4 — Add a binary-quantized column for the cheap first stage. For a corpus too large to serve from fp16 memory, precompute a bit code and index it with Hamming distance. binary_quantize maps each positive component to 1 and everything else to 0.

ALTER TABLE doc_embeddings
    ADD COLUMN embedding_bit bit(1536)
    GENERATED ALWAYS AS (binary_quantize(embedding)::bit(1536)) STORED;

CREATE INDEX doc_embeddings_bit_idx
    ON doc_embeddings
    USING hnsw (embedding_bit bit_hamming_ops);

Step 5 — Two-stage retrieval: Hamming shortlist, then fp32 rerank. The inner query pulls a wide shortlist ranked by cheap Hamming distance on the bit index; the outer query reranks only that shortlist by exact fp32 distance. This recovers nearly full-precision recall while the expensive exact comparison touches a couple hundred rows instead of the whole table.

WITH shortlist AS (
    SELECT doc_id, chunk_index, embedding
    FROM doc_embeddings
    ORDER BY embedding_bit <~> binary_quantize($1::vector(1536))::bit(1536)
    LIMIT 200                                    -- ~20x the final k
)
SELECT doc_id, chunk_index
FROM shortlist
ORDER BY embedding <=> $1::vector(1536)          -- exact fp32 rerank
LIMIT 10;

Validation & Recall Testing #

Quantization is a deliberate recall trade, so validation means proving the storage you saved cost you the recall you expected — no more. Measure both the bytes reclaimed and recall@k against an fp32 ground truth.

Storage saved. Confirm the fp16 index is materially smaller than its fp32 equivalent. Build both temporarily and compare:

SELECT 'fp32' AS variant, pg_size_pretty(pg_relation_size('doc_embeddings_fp32_idx'))
UNION ALL
SELECT 'fp16', pg_size_pretty(pg_relation_size('doc_embeddings_hv_idx'));

The halfvec index should land near half the size of the fp32 one; the bit index a small fraction of either.

Recall@k against fp32 ground truth. Compute exact nearest neighbors with an unindexed fp32 scan as truth, then measure how many the quantized path recovers. Run this for halfvec (expect ~0.99) and for binary-without-rerank (expect a visible drop) versus binary-with-rerank (expect it to climb back):

import asyncpg, asyncio

async def recall_at_k(pool, probes, k=10):
    async with pool.acquire() as conn:
        hits = 0
        for q in probes:                         # q is a normalized fp32 literal
            truth = await conn.fetch(             # exact fp32 nearest neighbors
                "SELECT doc_id FROM doc_embeddings "
                "ORDER BY embedding <=> $1::vector LIMIT $2", q, k)
            approx = await conn.fetch(
                "SELECT doc_id FROM doc_embeddings "
                "ORDER BY embedding::halfvec(1536) <=> $1::halfvec(1536) LIMIT $2",
                q, k)
            hits += len(set(r["doc_id"] for r in truth)
                        & set(r["doc_id"] for r in approx))
        return hits / (len(probes) * k)

Confirm the plan uses the expected index. A quantized query that silently falls back to a scan is not testing what you think. EXPLAIN must show the halfvec or bit index, not a Seq Scan:

EXPLAIN (ANALYZE, BUFFERS)
SELECT doc_id FROM doc_embeddings
ORDER BY embedding::halfvec(1536) <=> $1::halfvec(1536)
LIMIT 10;

If the plan shows a sequential scan, the query’s cast does not match the operator class the index was built with — the most common cause of a quantization change that “did nothing.” The storage math behind sizing all of this for a large corpus is worked end to end in calculating pgvector storage requirements for 10M embeddings.

Failure Modes & Gotchas #

• Binary recall cliff without rerank. Ranking final results directly by Hamming distance discards all magnitude information and drops recall dramatically. Binary quantization is only ever a first-stage filter; the fp32 rerank over its shortlist is mandatory, not optional.
• Normalizing after the downcast. Casting to halfvec or bit first and normalizing afterward bakes the quantization error into the unit vector, which cannot be recovered. Always normalize at fp32, then cast — the order is load-bearing.
• Operator-class mismatch. CREATE INDEX ... USING hnsw (col halfvec_cosine_ops) on a vector column, or querying an fp16 index with an fp32 <=> cast, either fails to build or silently forces a sequential scan. The column type, the index operator class, and the query’s cast must all agree.
• halfvec on pgvector below 0.7. halfvec, binary_quantize, and sparsevec do not exist before 0.7. On an older build the CREATE statements simply error; check SELECT extversion FROM pg_extension WHERE extname='vector' before adopting them.
• Shortlist too small. A binary shortlist barely larger than k cannot contain enough true neighbors for the rerank to find them, so recall stays low no matter how good the rerank is. Size the shortlist to roughly 5–20× the final k.
• Expecting fp16 to shrink the heap. An expression index on embedding::halfvec shrinks the index, not the fp32 heap column. To reclaim heap bytes you must actually change the column type to halfvec, giving up the exact-rerank source.

Monitoring & Alerting Hooks #

Track two things over time: the storage the quantized representation is actually saving, and the recall it is actually delivering, because a schema migration or an operator-class drift can quietly undo either. Export index sizes and, if you run recall benchmarks in CI, the measured recall@k as gauges.

from prometheus_client import Gauge

INDEX_BYTES = Gauge("pgvector_index_bytes", "Vector index size", ["index"])
RECALL_AT_K = Gauge("pgvector_recall_at_k", "Measured recall@10 vs fp32", ["variant"])

async def sample_storage(pool):
    async with pool.acquire() as conn:
        for r in await conn.fetch(
                "SELECT indexrelid::regclass::text AS name, "
                "pg_relation_size(indexrelid) AS bytes "
                "FROM pg_stat_user_indexes WHERE relname = 'doc_embeddings'"):
            INDEX_BYTES.labels(r["name"]).set(r["bytes"])

On the database side, alert on unexpected index growth — a halfvec index creeping back toward fp32 size means someone rebuilt it against the wrong operator class — and watch the plan-level signal that quantized queries are still using their index. Track scan counts on the vector indexes to catch a silent fallback to sequential scans:

SELECT indexrelid::regclass AS index_name,
       idx_scan,
       pg_size_pretty(pg_relation_size(indexrelid)) AS size
FROM pg_stat_user_indexes
WHERE relname = 'doc_embeddings'
ORDER BY idx_scan DESC;

A vector index whose idx_scan stops incrementing while query volume is steady is the signature of an operator-class or cast mismatch pushing queries onto a sequential scan — page on it, because latency will have degraded even though every result is still correct. Pair recall@k gauges with a CI threshold so a migration that drops recall below your floor fails the build rather than reaching production.

FAQ #

Related #

• Vector Data Type Selection — choosing the base storage type before deciding how to quantize it
• pgvector Storage Overhead Analysis — the per-row byte accounting these storage formulas come from
• Calculating pgvector storage requirements for 10M embeddings — sizing the fp32 versus fp16 versus bit footprint at scale
• Normalizing embeddings before pgvector insertion — why normalization must precede the downcast
• Optimizing m and ef_construction parameters — the build-memory pressure that halved index working sets relieve
• Up: pgvector Architecture & Vector Fundamentals