Text Embeddings Explained: The Intuition and Math Behind Semantic Search

Embeddings are the quiet workhorse of modern AI. Every RAG system, semantic search bar, "similar items" widget and duplicate detector depends on them. Yet they're often treated as a black box: text goes in, numbers come out. This guide opens the box.

#Key takeaways

• An embedding is a vector of numbers representing the meaning of a piece of text (or an image, audio clip or product).
• Texts with similar meaning land close together in vector space, even with no words in common.
• Similarity is usually measured with cosine similarity.
• Embeddings power semantic search, RAG, clustering, classification, recommendations and deduplication.
• Quality depends on the model, the chunking and matching the embedding model to your domain and language.

#From words to vectors: the intuition

Imagine describing every sentence with a few scores: how much it's about finance, how positive it is, how technical it is. "The stock market crashed" might score high on finance and low on positivity. "Shares plummeted today" would get nearly identical scores, despite sharing no words.

Real embedding models do this with hundreds to thousands of dimensions, and the dimensions aren't human-labeled concepts. They're learned automatically. But the principle holds: meaning becomes position.

text"How do I reset my password?"     → [0.021, -0.113, 0.458, ..., 0.067]
"I forgot my login credentials"   → [0.019, -0.098, 0.441, ..., 0.071]   ← close
"Best pizza in Chicago"           → [-0.302, 0.227, -0.015, ..., 0.190]  ← far

#How embedding models learn meaning

Modern text embedding models are transformer encoders (see how LLMs work) trained with contrastive learning:

1. Take pairs of texts that should be similar: a question and its answer, a title and its article, two paraphrases.
2. Train the model to pull those pairs' vectors together and push unrelated texts' vectors apart.
3. Repeat over enormous numbers of pairs.

After training, the model pools its token representations into a single fixed-size vector per input. Because the training pairs reflect meaning, relevance and question-answer matching, the geometry of the space does too.

#Measuring similarity

The most common measure is cosine similarity, the cosine of the angle between two vectors:

textcos(A, B) = (A · B) / (‖A‖ × ‖B‖)

It ranges from −1 (opposite) to 1 (identical direction). In practice, scores for text embeddings mostly fall in a narrower band, so compare scores relatively (which result ranks highest) rather than treating 0.8 as universally "similar".

pythonimport numpy as np

def cosine(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

query = embed("How do I reset my password?")
docs = {"reset": embed("Steps to change a forgotten password"),
        "billing": embed("Updating your payment method")}

ranked = sorted(docs, key=lambda k: cosine(query, docs[k]), reverse=True)
# ['reset', 'billing']

If vectors are normalized to unit length, cosine similarity equals the dot product, which is faster to compute. Many systems normalize at write time for exactly this reason.

#What you can build with embeddings

#Choosing an embedding model

Consider:

• Retrieval quality on your domain. Public leaderboards (such as MTEB) are a useful shortlist, but test on your own queries. Legal, medical or code search can rank models differently.
• Language support. Multilingual models map different languages into a shared space, useful for cross-language search.
• Dimensions. More dimensions can capture more nuance but cost more memory and compute. Some models support truncating vectors to shorter lengths with modest quality loss.
• Max input length. Text beyond the limit is truncated silently. Chunk accordingly.
• Hosting. API models are easy; open models can run locally for privacy and cost control at volume.
• Asymmetric search support. Some models expect different prefixes or modes for queries and documents. Follow the model's instructions, or quality drops.

#Practical pitfalls

Mixing models. Vectors from different models, or even different versions of one model, aren't comparable. Store the model name and version with every vector and re-embed everything when you switch.

Embedding giant documents. One vector for a 30-page document averages away the details. Chunk into passages, and include titles and headings in each chunk for context.

Expecting exact-match behavior. Embeddings are fuzzy by design. Searching for "ERR_4012" may return other error codes. Combine with keyword search (hybrid retrieval) for identifiers, names and codes.

Ignoring negation and numbers. "Contracts that allow early termination" and "contracts that don't allow early termination" can embed close together. Rerankers and LLM post-processing help with these subtleties.

No evaluation. Build a small set of queries with known relevant documents and measure recall@k before and after any change.

#Storing and searching embeddings at scale

For a few thousand vectors, a NumPy array and brute-force search is fine. Beyond that, you'll want an approximate nearest neighbor index, either in PostgreSQL with pgvector or in a dedicated engine. We compare the options in vector databases explained.

#FAQ

#What's the difference between embeddings and an LLM?

An embedding model outputs a vector that represents meaning. A generative LLM outputs text. They're often built on similar transformer architectures but trained for different objectives. RAG systems use both: embeddings to find context, an LLM to write the answer.

#How many dimensions should an embedding have?

Common sizes range from a few hundred to a few thousand. Higher isn't automatically better. Test retrieval quality at different sizes, weighing the gain against memory and latency costs.

#Can I create embeddings for images or audio?

Yes. Multimodal embedding models map images, and in some cases audio, into vectors, sometimes in the same space as text. That enables searching images with text queries.

#Do I need to re-embed my data when I change models?

Yes. Vectors from different models live in incompatible spaces. Plan re-embedding as a batch job and version your indexes so you can switch over atomically.