The Geometry of Meaning

The remarkable thing about embeddings is that the vector space they create has meaningful structure. Directions in the space correspond to semantic concepts. For example, in a well-trained embedding model, the direction from the vector for man to the vector for woman roughly corresponds to gender. Applying that same direction to the vector for king gives a vector close to queen. This is the classic king minus man plus woman equals queen example, and it demonstrates that embeddings capture relationships between words, not just individual word meanings.

This geometric structure extends to phrases and sentences, not just individual words. The embedding of a question and the embedding of the answer to that question tend to be close in vector space, because they discuss the same topic from different angles. This is what makes embeddings work for retrieval. You embed the user's question and find documents whose embeddings are nearby.

Similarity Metrics

To find the closest vectors to a query vector, you need a way to measure distance or similarity between vectors. Three metrics are commonly used.

Semantic Search and RAG

As covered in the RAG guide, embeddings power the retrieval step of a RAG system. Documents are chunked, each chunk is embedded, and the vectors are stored in a vector database. When a user asks a question, the question is embedded using the same model, and the system finds chunks whose vectors are closest to the question vector. This is semantic search. It finds relevant content based on meaning rather than keyword matching, so a question about refund policies can retrieve a document titled return and exchange guidelines even though no words overlap.

Clustering and Classification

Embeddings let you group similar documents together without labelled training data. A clustering algorithm like k-means applied to document embeddings will group documents by topic, because documents on the same topic have nearby vectors. Similarly, a classification model can use embeddings as input features. Instead of training a model on raw text, you embed the text and train a classifier on the vectors. This is faster and often more accurate than training from scratch, because the embedding model has already learned useful representations of language.

Recommendation

If you embed product descriptions, user reviews, and search queries into the same vector space, you can recommend products by finding items whose embeddings are close to what the user has shown interest in. The same principle applies to content recommendation, where articles or videos are embedded and compared to the user's reading or viewing history.

Deduplication and Entity Resolution

When integrating data from multiple sources, the same entity often appears with slightly different text. Two product listings might describe the same item with different wording. Embedding both descriptions and measuring their similarity lets you identify near-duplicates automatically. This is useful for data cleaning, knowledge graph construction, and entity resolution.

Brute Force vs Approximate Search

The simplest way to find the closest vectors to a query is to compute the similarity between the query and every vector in the database, then return the top results. This is called brute force search, and it is perfectly accurate. For a database of 10,000 vectors, it is fast. For a database of 100 million vectors, it is too slow for real-time queries.

Vector databases solve this with approximate nearest neighbour search. Instead of checking every vector, they use indexing structures that partition the vector space, allowing the search to skip large portions of the database that are unlikely to contain the closest vectors. The result is a small loss of accuracy, meaning the exact nearest neighbours might not always be found, in exchange for a large speed improvement.

Popular Vector Databases

Several vector databases are commonly used, each with different strengths in terms of scale, features, and ease of deployment.

What Makes a Good Embedding

A good embedding captures semantic similarity accurately. Sentences that mean the same thing should have vectors that are close together. Sentences that mean different things should have vectors that are far apart. The embedding should also generalise across topics, languages, and writing styles. An embedding model trained only on news articles may perform poorly on legal text or code.

Evaluation Benchmarks

Embedding models are evaluated on standard benchmarks that test their ability to capture semantic relationships. The MTEB, or Massive Text Embedding Benchmark, evaluates models across several tasks including retrieval, classification, clustering, and semantic textual similarity. MS MARCO evaluates retrieval quality on real web search queries. BEIR provides a collection of datasets for evaluating zero-shot retrieval across domains.

Benchmark scores give a rough sense of model quality, but they do not always reflect performance on your specific data. The best approach is to build a small evaluation set of queries and relevant documents from your domain, embed them with candidate models, and measure which model retrieves the correct documents with the highest accuracy.

Input Length

Embedding models have a maximum input length, typically measured in tokens. A model might accept up to 512 tokens, which is roughly 400 words. If you try to embed a longer document, the model will truncate it, potentially losing important information. This is why RAG systems chunk documents before embedding, as covered in the RAG guide. The chunk size should fit within the embedding model's input limit.

Normalisation

Some embedding models produce normalised vectors, meaning each vector has a length of 1. Others produce unnormalised vectors. If you are using cosine similarity, normalisation does not matter because cosine similarity ignores magnitude. If you are using dot product, normalisation is important because dot product is affected by vector length. Check whether your model produces normalised vectors and configure your vector database accordingly.

Cost and Latency

Embedding text costs money and takes time. For API-based models, you pay per token for embedding. For a knowledge base of one million chunks averaging 200 tokens each, the embedding cost is 200 million tokens. This is a one-time cost unless the data changes. Query-time embedding is cheap because each query is a single short text. Latency at query time is dominated by the vector database search, not the embedding, unless the query is very long.

Multilingual Support

If your application handles multiple languages, choose an embedding model with multilingual support. These models are trained on text in many languages and produce a shared vector space where a question in Hindi and a document in English can be compared directly. Models without multilingual training will produce poor results when the query and the documents are in different languages.

Loss of Information

An embedding compresses a piece of text into a fixed-length vector. This compression loses information. A 500-word document and a 10-word sentence might both produce a 768-dimensional vector. The vector captures the dominant semantic content but discards details, nuance, and structure. This is why chunking matters. A chunk that is too large produces an embedding that averages over too many topics, diluting the signal.

No Understanding of Negation

Embeddings struggle with negation. The sentence I like this product and I do not like this product can produce similar embeddings because the words are nearly identical. The meaning is opposite, but the vector similarity is high. This is a known limitation of embedding-based retrieval and a reason why hybrid search, combining vector search with keyword search, can outperform pure vector search on certain queries.

Domain Sensitivity

Embedding models trained on general text may not capture domain-specific relationships well. Medical text, legal text, and code use specialised vocabulary and sentence structures that differ from general web text. A general-purpose embedding model may produce poor results on these domains. Fine-tuning an embedding model on domain-specific data, or choosing a model pretrained on relevant text, can improve performance.

No Relationship Structure

Embeddings capture similarity, not structure. They can tell you that two texts are related, but they cannot tell you how they are related. For structured relationships, as covered in the knowledge graphs guide, a graph representation is more appropriate. Embeddings and knowledge graphs are complementary. Embeddings handle semantic similarity, and knowledge graphs handle structured relationships.