Vector Databases Explained: Embeddings, ANN Indexes and When You Actually Need One

Vector databases went from niche to mainstream on the back of RAG and semantic search. But "vector database" covers everything from a Postgres extension to a distributed managed service, and plenty of teams adopt one they don't need. This guide explains what vector databases actually do, how their indexes work, and how to choose.

#Key takeaways

• A vector database stores embeddings and answers "find the k most similar vectors to this one" quickly.
• Exact search is simple but slow at scale; production systems use approximate nearest neighbor (ANN) indexes like HNSW or IVF.
• For many applications, pgvector in PostgreSQL is enough. Dedicated vector databases pay off with very large collections, heavy filtering, multi-tenancy or tight latency targets.
• Retrieval quality depends far more on embeddings, chunking and hybrid search than on the database brand.

#What problem does a vector database solve?

Embedding models turn text, images or audio into vectors, which are lists of hundreds or thousands of numbers, such that similar content produces nearby vectors. (If that's new, start with text embeddings explained.)

Once your data is embedded, the core query is nearest neighbor search: given a query vector, return the k stored vectors closest to it by cosine similarity, dot product or Euclidean distance.

A brute-force scan compares the query against every vector. For 10,000 vectors that's instant. For 100 million vectors of 1,024 dimensions, it's about 100 billion multiply-adds per query. That's where specialized indexes come in.

#How approximate nearest neighbor (ANN) search works

ANN indexes trade a small amount of accuracy (recall) for large gains in speed. Instead of guaranteeing the true top-k, they return results that are almost always the same, often 95–99% recall, in milliseconds.

#HNSW (Hierarchical Navigable Small World)

HNSW builds a multi-layer graph. Each vector is a node connected to its near neighbors. Upper layers are sparse "highways" and lower layers are dense local roads. A search starts at the top, greedily hops toward the query, then descends layer by layer to refine.

• Pros: excellent recall/latency trade-off, supports incremental inserts, the default in most systems.
• Cons: memory-hungry, since the graph lives in RAM, and index builds can be slow for huge datasets.
• Key knobs: M (connections per node), ef_construction (build quality), ef_search (query-time accuracy vs speed).

#IVF (Inverted File Index)

IVF clusters vectors with k-means into nlist buckets. At query time, it searches only the nprobe closest buckets.

• Pros: lower memory, fast to build, pairs well with compression.
• Cons: recall depends on tuning nprobe, and clusters need retraining if the data distribution shifts.

#Quantization

Product quantization (PQ) and scalar/binary quantization compress vectors, for example from 32-bit floats to 8-bit integers or single bits. This cuts memory by 4–32× at some accuracy cost. A common pattern is to search compressed vectors, then rescore the top candidates with full-precision vectors.

#Features that matter beyond raw search

When teams compare vector databases, the ANN algorithm is rarely the deciding factor. These are:

• Metadata filtering. "Similar documents where tenant_id = 42 and year ≥ 2024." Filtering naively after the ANN search can return too few results. Good engines integrate filters into the search itself.
• Hybrid search. Combining vector similarity with keyword (BM25) relevance. Essential for product codes, names and error strings. See our RAG guide for why.
• Multi-tenancy. Isolating customers' data efficiently, either through namespaces or partitions.
• Updates and deletes. Some indexes degrade with heavy churn and need periodic rebuilds.
• Operational fit. Backups, replication, monitoring, and whether your team already runs it.

#Your options, compared

#PostgreSQL + pgvector

pgvector adds a vector column type plus HNSW and IVF indexes to Postgres.

sqlCREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id bigserial PRIMARY KEY,
  tenant_id int NOT NULL,
  content text NOT NULL,
  embedding vector(1024)
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

SELECT id, content
FROM chunks
WHERE tenant_id = 42
ORDER BY embedding <=> $1   -- cosine distance
LIMIT 10;

Choose it when: you already run Postgres, have up to tens of millions of vectors, and want transactions, joins and one fewer system to operate. For most startups this is the right default.

#Dedicated vector databases

Purpose-built engines, open-source or managed, offer higher scale, better filtered-search performance, built-in hybrid search, quantization options and horizontal sharding.

Choose one when: you have hundreds of millions of vectors or more, strict latency SLOs under heavy concurrent load, complex filtering at scale, or you want a managed service to own the operational burden.

#Search engines with vector support

Elasticsearch and OpenSearch support dense vectors alongside their mature keyword search, aggregations and filtering.

Choose them when: you already run one for search, or hybrid keyword + semantic ranking is central to your product.

#How to choose: a practical decision path

1. Under ~1M vectors and already on Postgres? Use pgvector. Revisit later.
2. Search is a core product feature with heavy keyword needs? Use a search engine with vector support.
3. Massive scale, many tenants, strict latency, or no appetite for ops? Evaluate dedicated vector databases with your data and queries.
4. Benchmark before committing. Measure recall@k against exact search, p95 latency under realistic concurrency, and cost. Vendor benchmarks rarely match your workload.

#Tuning tips that actually move the needle

• Normalize embeddings if you use cosine similarity, and use the distance metric your embedding model was trained for.
• Measure recall by comparing ANN results against exact search on a sample of queries, then tune ef_search or nprobe until the recall/latency trade-off suits you.
• Keep dimensions sensible. Some embedding models support shorter vectors (Matryoshka-style truncation) with small quality loss and big memory savings.
• Store the text and metadata you need next to the vector to avoid a second lookup on every query.
• Plan for re-embedding. When you switch embedding models, you must re-embed everything. Version your embeddings.

#FAQ

#Is a vector database the same as a vector index?

No. A vector index (HNSW, IVF) is the data structure that speeds up similarity search. A vector database wraps indexes with storage, persistence, filtering, replication, APIs and operations. Libraries like FAISS give you indexes only.

#How much memory does an HNSW index need?

Roughly the raw vector size (count × dimensions × 4 bytes for float32) plus graph overhead that depends on M. For example, 10 million 1,024-dimension float32 vectors are about 41 GB before graph overhead. Quantization can reduce this substantially.

#Can I use a vector database without an LLM?

Yes. Vector search predates LLMs and powers recommendations, image similarity, duplicate detection, anomaly detection and semantic search on its own.

#Cosine similarity or dot product?

Use whatever your embedding model recommends. For normalized vectors, cosine similarity and dot product produce the same ranking, and dot product is slightly cheaper to compute.