Choosing an Embedding Model: A Practical Benchmark

Every retrieval-augmented generation project eventually hits the same fork in the road: which embedding model actually belongs in production. The MTEB leaderboard makes this look like a simple ranking problem, pick whichever model sits at the top and move on. In practice it is closer to a systems decision than a leaderboard lookup, because dimension count, token pricing, latency, and how well a model handles your specific language and domain all pull in different directions.

This post walks through the tradeoffs I actually weigh when choosing an embedding model for a client project, why aggregate benchmark scores are a starting point and not a verdict, how multilingual models behave differently on Indonesian text than their English scores suggest, and a small recall at k harness you can run against your own labeled query and document pairs before committing to anything.

Why the leaderboard number is not the whole story

The Massive Text Embedding Benchmark aggregates scores across dozens of datasets and eight task categories, including classification, clustering, and retrieval. A model's overall MTEB score is an average across tasks that may have nothing to do with your use case. A model that excels at sentence similarity or clustering can still underperform on the specific kind of asymmetric query to long document retrieval that RAG actually needs.

• Task mismatch: your app needs retrieval quality, but a chunk of the aggregate score comes from classification and clustering tasks that do not resemble a search query matching a paragraph.
• Domain mismatch: a model tuned on Wikipedia and web crawl data will not automatically transfer to invoice line items, legal clauses, or ERP field names.
• Language mismatch: English retrieval scores tell you almost nothing about how the same model ranks documents when the query and the corpus are in Indonesian.

Dimension size, cost, and what you actually give up

Every embedding model publishes a native vector dimension, and most modern models let you truncate that vector to a smaller size without recomputing it, a technique commonly called Matryoshka representation. Smaller vectors mean cheaper storage, faster similarity search, and lower memory pressure on your vector index, but they can quietly cost you recall on harder queries. The table below is a rough comparison of a few models commonly used for RAG in 2026.

A useful rule of thumb: OpenAI's own documentation notes that a large embedding model truncated down to a much smaller size can still outperform an older, unshortened model at full size. That means the honest question is rarely which model has the biggest vector, it is which model retains the most useful signal at the dimension size your infrastructure can actually afford to index and query at scale.

Multilingual performance and the Indonesian gap

If any part of your corpus is in Indonesian, English-language leaderboard rankings should be treated as noise, not signal. The Massive Multilingual Text Embedding Benchmark expanded coverage to hundreds of languages precisely because English-only evaluation was hiding large performance gaps. Its own published finding is a good sanity check on the assumption that bigger models always win: the strongest publicly available multilingual model in that evaluation had only a few hundred million parameters, well below many general-purpose embedding models.

• Mixed-language corpora, where an Indonesian query needs to retrieve an English technical document or vice versa, stress models differently than monolingual retrieval, and not every multilingual model handles this well.
• Tokenization matters more than people expect. Indonesian affixes and reduplication can fragment poorly with tokenizers trained mostly on English and Chinese corpora, which quietly hurts retrieval on short queries.
• Instruction-tuned multilingual models, where you prepend a short task description to the query before embedding it, often close a meaningful part of the gap on Indonesian retrieval without needing a bigger model.

Benchmarking on your own data instead of trusting a leaderboard

The only benchmark that reliably predicts production behavior is one built from your own queries and your own documents. You do not need thousands of labeled pairs to get a directionally useful signal, thirty to fifty realistic query and relevant document pairs pulled from real support tickets, real search logs, or a few hours of manual labeling is enough to separate a genuinely better model from a merely more popular one. The script below computes recall at k, the fraction of queries where the correct document appears in the top k retrieved results, for any embedding model that exposes a batch embedding call.

// Minimal recall@k harness against your own labeled pairs
import { OpenAI } from "openai"

interface EvalPair {
  query: string
  relevantDocId: string
}

const openai = new OpenAI()

async function embed(texts: string[], model: string) {
  const res = await openai.embeddings.create({ model, input: texts })
  return res.data.map((d) => d.embedding)
}

function cosineSim(a: number[], b: number[]) {
  let dot = 0, magA = 0, magB = 0
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i]
    magA += a[i] * a[i]
    magB += b[i] * b[i]
  }
  return dot / (Math.sqrt(magA) * Math.sqrt(magB))
}

async function recallAtK(
  pairs: EvalPair[],
  corpus: { id: string; text: string }[],
  model: string,
  k = 5
) {
  const corpusVecs = await embed(corpus.map((c) => c.text), model)
  const queryVecs = await embed(pairs.map((p) => p.query), model)

  let hits = 0
  pairs.forEach((pair, i) => {
    const scored = corpus.map((doc, j) => ({
      id: doc.id,
      score: cosineSim(queryVecs[i], corpusVecs[j]),
    }))
    scored.sort((a, b) => b.score - a.score)
    const topK = scored.slice(0, k).map((s) => s.id)
    if (topK.includes(pair.relevantDocId)) hits++
  })

  return hits / pairs.length
}

Run this same script against every candidate model with the same query and document pairs, and you get a like-for-like comparison that accounts for your actual chunking strategy, your actual document length, and your actual vocabulary, none of which the public leaderboard can see. It typically takes less than an hour to run against three or four candidate models once the labeled pairs exist.

A decision checklist before you commit

Once you have recall numbers from your own data, a few practical constraints usually narrow the field down to one or two realistic choices.

1. Confirm the model's maximum input token length covers your largest chunk size, including any metadata you prepend to the chunk before embedding it.
2. Decide whether a hosted API or a self-hosted model fits your latency and data residency requirements, since sending customer documents to a third-party API is a real constraint for some clients.
3. Estimate total embedding cost at your expected document volume, not just the per-token price, since re-embedding an entire corpus after a chunking change adds up fast at scale.
4. Lock in a re-embedding plan before launch. Comparing raw dot products between vectors produced by two different embedding models is meaningless, so switching models later means re-indexing everything, not just the new documents.

What I actually recommend

For a purely English, general-purpose RAG project on a tight budget, a small hosted model with truncated dimensions is a sound default. For anything touching Indonesian content, a self-hosted multilingual model evaluated specifically on Indonesian query and document pairs consistently outperforms English-tuned hosted models in my own testing, even when the hosted model wins on the general leaderboard. Whichever you pick, treat the choice as reversible only if you have budgeted time to re-embed the entire corpus, because embeddings from different models are never comparable to each other.