# Choosing an Embedding Model: A Practical Benchmark > A practical framework for picking an embedding model for RAG: dimension versus cost versus recall, why MTEB scores can mislead you, and a script to benchmark candidates on your own data. URL: https://www.matthewswong.com/en/blog/embedding-model-selection-benchmark/ - Author: Matthews Wong (https://www.matthewswong.com/en/about/) - Published: 2026-07-03 - Updated: 2026-07-03 - Category: AI/ML Every retrieval-augmented generation project eventually hits the same fork in the road: which embedding model actually belongs in production. The MTEB leaderboard makes this look like a simple ranking problem, pick whichever model sits at the top and move on. In practice it is closer to a systems decision than a leaderboard lookup, because dimension count, token pricing, latency, and how well a model handles your specific language and domain all pull in different directions. This post walks through the tradeoffs I actually weigh when choosing an embedding model for a client project, why aggregate benchmark scores are a starting point and not a verdict, how multilingual models behave differently on Indonesian text than their English scores suggest, and a small recall at k harness you can run against your own labeled query and document pairs before committing to anything. ## Why the leaderboard number is not the whole story The Massive Text Embedding Benchmark aggregates scores across dozens of datasets and eight task categories, including classification, clustering, and retrieval. A model's overall MTEB score is an average across tasks that may have nothing to do with your use case. A model that excels at sentence similarity or clustering can still underperform on the specific kind of asymmetric query to long document retrieval that RAG actually needs. - Task mismatch: your app needs retrieval quality, but a chunk of the aggregate score comes from classification and clustering tasks that do not resemble a search query matching a paragraph. - Domain mismatch: a model tuned on Wikipedia and web crawl data will not automatically transfer to invoice line items, legal clauses, or ERP field names. - Language mismatch: English retrieval scores tell you almost nothing about how the same model ranks documents when the query and the corpus are in Indonesian. > Do not select an embedding model purely by scrolling to the top of a public leaderboard. A half point difference in an aggregate score can flip completely once you test the model against your own queries and your own documents. ## Dimension size, cost, and what you actually give up Every embedding model publishes a native vector dimension, and most modern models let you truncate that vector to a smaller size without recomputing it, a technique commonly called Matryoshka representation. Smaller vectors mean cheaper storage, faster similarity search, and lower memory pressure on your vector index, but they can quietly cost you recall on harder queries. The table below is a rough comparison of a few models commonly used for RAG in 2026. Low > Good default for English-heavy corpora on a budget Medium > Higher recall ceiling, supports dimension truncation Free, self-hosted > Strong multilingual retrieval, needs a GPU or CPU batch job Free, self-hosted > Dense, sparse, and multi-vector retrieval in one model A useful rule of thumb: OpenAI's own documentation notes that a large embedding model truncated down to a much smaller size can still outperform an older, unshortened model at full size. That means the honest question is rarely which model has the biggest vector, it is which model retains the most useful signal at the dimension size your infrastructure can actually afford to index and query at scale. ## Multilingual performance and the Indonesian gap If any part of your corpus is in Indonesian, English-language leaderboard rankings should be treated as noise, not signal. The Massive Multilingual Text Embedding Benchmark expanded coverage to hundreds of languages precisely because English-only evaluation was hiding large performance gaps. Its own published finding is a good sanity check on the assumption that bigger models always win: the strongest publicly available multilingual model in that evaluation had only a few hundred million parameters, well below many general-purpose embedding models. - Mixed-language corpora, where an Indonesian query needs to retrieve an English technical document or vice versa, stress models differently than monolingual retrieval, and not every multilingual model handles this well. - Tokenization matters more than people expect. Indonesian affixes and reduplication can fragment poorly with tokenizers trained mostly on English and Chinese corpora, which quietly hurts retrieval on short queries. - Instruction-tuned multilingual models, where you prepend a short task description to the query before embedding it, often close a meaningful part of the gap on Indonesian retrieval without needing a bigger model. > If you serve both English and Indonesian users, do not assume one embedding model is best for both. Run the same recall evaluation twice, once per language, and only standardize on a single model if the drop in the weaker language is small enough to accept. ## Benchmarking on your own data instead of trusting a leaderboard The only benchmark that reliably predicts production behavior is one built from your own queries and your own documents. You do not need thousands of labeled pairs to get a directionally useful signal, thirty to fifty realistic query and relevant document pairs pulled from real support tickets, real search logs, or a few hours of manual labeling is enough to separate a genuinely better model from a merely more popular one. The script below computes recall at k, the fraction of queries where the correct document appears in the top k retrieved results, for any embedding model that exposes a batch embedding call. Run this same script against every candidate model with the same query and document pairs, and you get a like-for-like comparison that accounts for your actual chunking strategy, your actual document length, and your actual vocabulary, none of which the public leaderboard can see. It typically takes less than an hour to run against three or four candidate models once the labeled pairs exist. ## A decision checklist before you commit Once you have recall numbers from your own data, a few practical constraints usually narrow the field down to one or two realistic choices. - Confirm the model's maximum input token length covers your largest chunk size, including any metadata you prepend to the chunk before embedding it. - Decide whether a hosted API or a self-hosted model fits your latency and data residency requirements, since sending customer documents to a third-party API is a real constraint for some clients. - Estimate total embedding cost at your expected document volume, not just the per-token price, since re-embedding an entire corpus after a chunking change adds up fast at scale. - Lock in a re-embedding plan before launch. Comparing raw dot products between vectors produced by two different embedding models is meaningless, so switching models later means re-indexing everything, not just the new documents. > A smaller, cheaper model that you actually validated against your own data will beat a bigger, more expensive model you only trusted because of its leaderboard rank. Validation time is never wasted time on this decision. ## What I actually recommend For a purely English, general-purpose RAG project on a tight budget, a small hosted model with truncated dimensions is a sound default. For anything touching Indonesian content, a self-hosted multilingual model evaluated specifically on Indonesian query and document pairs consistently outperforms English-tuned hosted models in my own testing, even when the hosted model wins on the general leaderboard. Whichever you pick, treat the choice as reversible only if you have budgeted time to re-embed the entire corpus, because embeddings from different models are never comparable to each other. ## Sources - Hugging Face: MTEB, the Massive Text Embedding Benchmark - OpenAI API docs: vector embeddings guide - MMTEB: Massive Multilingual Text Embedding Benchmark (arXiv) ## Source links - https://huggingface.co/blog/mteb - https://developers.openai.com/api/docs/guides/embeddings - https://arxiv.org/abs/2502.13595 ## FAQ ### What is the best embedding model for RAG in 2026 There is no single best model, it depends on your language, domain, and budget. A small hosted model with truncated dimensions works well for English-only, cost-sensitive projects, while a self-hosted multilingual model tends to perform better on mixed English and Indonesian corpora. The only reliable way to decide is to run a recall at k test against your own labeled query and document pairs. ### Why should I not just pick the top model on the MTEB leaderboard MTEB aggregates scores across many task types, including classification and clustering, that may have little to do with retrieval quality for your app. A model can rank high overall while underperforming on the specific retrieval pattern RAG needs, and domain or language mismatch can flip the ranking entirely once you test on your own data. ### How much does embedding dimension size affect RAG quality Larger dimensions generally give a higher recall ceiling but cost more to store and query. Many modern models support Matryoshka-style truncation, letting you shrink the vector after generation while retaining most of the useful signal, so the right size is often smaller than the model's native dimension once cost and index size are factored in. ### Do embedding models handle Indonesian text as well as English Not automatically. English retrieval scores on public leaderboards say almost nothing about Indonesian performance, since affixes and reduplication tokenize differently and many models are trained mostly on English and Chinese data. Multilingual benchmarks like MMTEB show the strongest multilingual models are not always the largest ones, so testing on real Indonesian queries is essential. ### Can I switch embedding models after launching a RAG system Yes, but it requires re-embedding the entire corpus, not just new documents, because vectors from different models are not comparable to each other. Budget the re-indexing time and cost as part of the decision up front rather than treating the choice of embedding model as easily reversible later. --- - HTML version: https://www.matthewswong.com/en/blog/embedding-model-selection-benchmark/ - Bahasa Indonesia: https://www.matthewswong.com/id/blog/embedding-model-selection-benchmark/ - Site index for agents: https://www.matthewswong.com/llms.txt - Sitemap: https://www.matthewswong.com/sitemap.xml - API description (OpenAPI): https://www.matthewswong.com/openapi.json