The phrase "convert your documents into vectors" appears in every RAG tutorial and explains almost nothing. Here's what those numbers actually represent, why similar texts cluster together, and how to pick the embedding model that fits your pipeline.
What a vector actually stores
An embedding model processes a piece of text and outputs a list of numbers, typically 768, 1536, or 3072 of them. That list is the embedding.
Each number is a coordinate in a very high-dimensional space. What matters isn't what any individual number means: it's that texts with similar meaning end up at similar coordinates. "My order hasn't arrived" and "Where is my package?" land near each other in this space. "Where is my package?" and "How do I cancel my subscription?" land far apart.
The model learns these coordinates by training on contrastive pairs: sentences that mean the same thing, and sentences that don't. The training objective is simple — minimize the distance between similar pairs, maximize it for dissimilar ones. The resulting coordinate space is compact enough to fit in memory and good enough to power most production retrieval systems.
One practical consequence: the embedding model is not your LLM. It's a separate, much smaller model optimized specifically for producing meaningful coordinates, not for generating text. OpenAI's text-embedding-3-small has roughly 330M parameters; GPT-4o has around 200B. They're playing different games.
How cosine similarity works
Once you have vectors, you need a way to measure how close two of them are. The standard approach is cosine similarity, which measures the angle between two vectors.
Forget the formula. The intuition: two vectors pointing in roughly the same direction are similar. Two vectors pointing in opposite directions are dissimilar. The score runs from -1 (opposite meaning) to 1 (identical meaning), with 0 meaning unrelated.
Cosine scores above 0.85 are usually strong semantic matches. Below 0.7, relevance gets unreliable. Most RAG systems use a cutoff around 0.75 and return the top 3-5 chunks above it.
This is why semantic caching works: if an incoming question scores 0.92 against a cached question, there's no point calling the LLM again. Same trick, different application.
The reason retrieval systems use cosine similarity instead of keyword matching is precisely the vocabulary gap. "Package not delivered" and "order hasn't arrived" share zero words but near-identical vectors. Full-text search misses the match. Cosine similarity catches it.
Where embeddings live in a RAG pipeline
A RAG pipeline has three stages: ingestion, retrieval, and generation. Embeddings power the first two.
During ingestion, you split your documents into chunks (typically 200-500 tokens), embed each chunk, and store the resulting vectors in a vector database. This is a one-time job, or a scheduled sync if your documents update regularly.
During retrieval, when a user asks a question, you embed the question using the same model, then query the vector database for the N chunks with the highest cosine similarity to the question vector. Those chunks become the context you pass to the LLM for generation.
One constraint that bites early: the embedding model you use during ingestion and retrieval must be identical. If you embed your documents with text-embedding-3-small and later switch to gemini-embedding-001, the vectors live in different dimensional spaces and cannot be compared. Lock your embedding model into your pipeline before you index at scale.
The context window is the other constraint. Retrieval narrows a 50,000-page corpus down to 5 relevant chunks. Without it, you'd need to paste the entire knowledge base into every prompt, which no context window allows at reasonable cost.
Picking an embedding model
The main tradeoffs are cost, embedding dimensions, and maximum input length. Current options with their key specs:
| Model | Cost per 1M tokens | Dimensions | Max input | Notes |
|---|---|---|---|---|
text-embedding-3-small |
$0.02 (OpenAI pricing) | 1,536 | 8,191 tokens | Best default for most pipelines |
text-embedding-3-large |
$0.13 (OpenAI pricing) | 3,072 | 8,191 tokens | ~6x cost, 5-8% better recall |
gemini-embedding-001 |
$0.15 (Google pricing) | 3,072 | 2,048 tokens | Best multilingual coverage |
| Nomic Embed Text | Free (self-hosted) | 768 | 8,192 tokens | Best for on-prem, runs on 4GB RAM |
embed-english-v3.0 |
$0.10 (Cohere pricing) | 1,024 | varies | Separate query/doc modes help precision |
text-embedding-3-small is the right default for most applications: English or major European languages, internal knowledge bases, customer support, doc search. The cost to embed a large corpus is smaller than most people expect. At $0.02 per million tokens, indexing 5 million tokens of content costs $0.10 total; for complete break-even math on large corpora and self-hosting, see embedding costs for RAG in 2026.
The harder decision is which LLM you use for the generation step. See how four current models compare on RAG answer quality for head-to-head numbers.
For teams evaluating retrieval across providers, the LLMTest proxy routes embedding calls alongside generation calls through a single OpenAI-compatible endpoint, so you can swap embedding providers without changing your application code.
FAQ
What's the difference between an embedding and a token? A token is the basic unit of text that LLMs read (roughly 3/4 of a word). An embedding is the vector produced by running text through an embedding model. Tokens are inputs to LLMs; embeddings are outputs of embedding models. They're related: embedding models also convert their input into tokens as part of processing, but they serve different purposes. For a deeper look at how tokens actually work, see how tokens, words, and characters compare.
Do I need a vector database? Not for small corpora. Under 10,000 chunks, you can store embeddings as a JSON array and do cosine similarity in memory at query time. Python's numpy or JavaScript's typed arrays handle this in milliseconds. For larger corpora or many concurrent users, a dedicated vector store (Pinecone, pgvector, Chroma, Weaviate) handles indexing and retrieval without loading all vectors into RAM on every request.
Can I use the same embedding model as my LLM provider? Yes, and it simplifies billing and SDK integration. The downside is a single-vendor outage affecting both embedding and generation. If uptime is critical, consider using different providers for each so a generation outage doesn't also take down retrieval.
What chunk length should I start with? 200-400 tokens is the standard range. Smaller chunks improve precision (each chunk covers one topic) but require more retrieval calls. Larger chunks improve recall but make the retrieved content harder for the LLM to work with cleanly. Most teams start at 300 tokens with a 50-token overlap between consecutive chunks, then adjust based on retrieval quality measurements.
How do I know if my retrieval is good enough? Build a golden set: 20-50 example questions where you know which chunk should surface. Run retrieval against it and measure how often the correct chunk appears in your top-3 results. If the hit rate is below 75%, try smaller chunks, a different embedding model, or query rewriting before touching the generation step. This is more useful than any external benchmark.
Are embeddings the same as fine-tuning? No. Fine-tuning changes the model weights to improve behavior on a specific task. Embeddings are a read-only representation of text; nothing gets modified. Fine-tuning requires a training job. Embedding happens at inference time, costs fractions of a cent, and is reversible. Most teams try RAG with embeddings first and reach for fine-tuning only when behavior itself needs to change rather than just knowledge. For the break-even math on when that switch makes sense, see our fine-tuning explained guide.
Next steps
If you're deciding which generation model to pair with your embedding setup, LLMTest runs head-to-head benchmarks on real prompts so you can pick based on what your pipeline actually produces.