Sahabat AI: Indonesian LLM Developer Guide and Token Costs

The chatbot on this site answers in English and in Indonesian, and the Indonesian answers cost more tokens. Not because they are longer, but because an Indonesian word is worth more tokens than an English one on whichever tokeniser the model happens to use. I had assumed a model trained in Indonesia would fix that. Reading the model cards, it does not.

This post is what I found when I actually looked: what Sahabat AI is, what its model cards say and pointedly do not say, what four tokenisers charge per Indonesian word when you run them over the same text, and how I would choose between a locally trained model and a frontier API for an Indonesian product. Every number below is either quoted from a model card or measured on my own machine, and the measuring script is included so you can disagree with it.

What Sahabat AI actually is

Sahabat AI is a collection of large language models pretrained and instruct-tuned for Indonesia, co-initiated by PT GoTo Gojek Tokopedia and Indosat Ooredoo Hutchison and developed together with AI Singapore. It is not one model. Five repositories are published on Hugging Face covering three sizes: a 70 billion parameter v2 built on Llama 3.1 70B Instruct, and two v1 models at 8 and 9 billion parameters built on Llama 3 8B and Gemma 2 9B, each in a base and an instruct variant. There is also a public chat service at sahabat-ai.com and inside the GoPay app.

The three sizes differ in ways that matter more than the parameter count. The 70B lists six supported languages, adding Batak Toba and Balinese to English, Indonesian, Javanese and Sundanese; the smaller pair lists four. Context length is 8,192 tokens on both v1 models and 128k on the 70B. And each ships under the community licence of the model it was built on, which is not the same thing as an open source licence, whatever the surrounding ecosystem is called.

Continued pre-training does not replace the tokeniser

One sentence on the model cards changed how I thought about this, and it appears on all three in nearly the same words: the model employs the default tokeniser used in its base model. The 70B keeps the Llama 3.1 tokeniser. The 9B keeps the Gemma 2 one. The 8B keeps Llama 3's. Nobody trained a new Indonesian vocabulary.

That is a defensible engineering decision. Swapping a tokeniser means rebuilding the embedding layer and discarding most of what the base model already knows, which is a far larger project than continued pre-training. But it has a consequence people assume away. Continued pre-training changes what a model knows about Indonesia; it does not change how many tokens your Indonesian sentence costs. If you are hoping a local model will cut the token half of your bill, the model card has already told you it will not.

What four tokenisers charge for one Indonesian word

So I measured it. The script below pulls the plain-text Wikipedia article on Indonesia in English, Indonesian, Javanese and Sundanese from the MediaWiki API, truncates each to the first 15,000 characters, and counts tokens with four tokenisers: OpenAI's o200k_base and cl100k_base through tiktoken, plus the tokenizer.json files of Llama 3.1 and Gemma 2 loaded with the Hugging Face tokenizers library. Because Sahabat AI keeps the base tokeniser unchanged, measuring Llama 3.1 is measuring the 70B.

# tokens_per_word.py -- what one Indonesian word actually costs.
# Corpus: first 15,000 characters of the plain-text Wikipedia article
# "Indonesia" in en / id / jv / su, fetched from the MediaWiki API:
#   https://id.wikipedia.org/w/api.php?action=query&prop=extracts
#     &explaintext=1&format=json&titles=Indonesia&redirects=1
import re
import tiktoken
from tokenizers import Tokenizer

# Sahabat-AI ships NO tokenizer of its own. Every model card says it keeps
# the base model's, so loading Llama 3.1 here IS loading the 70B's.
llama = Tokenizer.from_file("llama31-tokenizer.json")   # unsloth/Meta-Llama-3.1-8B-Instruct
gemma = Tokenizer.from_file("gemma2-tokenizer.json")    # unsloth/gemma-2-9b-it

encoders = {
    "o200k_base": tiktoken.get_encoding("o200k_base").encode,
    "cl100k_base": tiktoken.get_encoding("cl100k_base").encode,
    "llama3.1": lambda s: llama.encode(s, add_special_tokens=False).ids,
    "gemma2": lambda s: gemma.encode(s, add_special_tokens=False).ids,
}

for lang in ("en", "id", "jv", "su"):
    text = open(f"full-{lang}.txt", encoding="utf-8").read()[:15000]
    words = len(re.findall(r"[A-Za-z\u00c0-\u00ff'-]+", text))
    for name, encode in encoders.items():
        n = len(encode(text))
        # chars/token is the sanity check: it needs no word count at all,
        # so it survives the fact that these are not parallel translations.
        print(f"{lang} {name} {n / words:.2f} tok/word {len(text) / n:.2f} chars/tok")

The result was not the one I expected. The Llama 3.1 tokeniser, the one the flagship 70B inherits, was the most expensive of the four on Indonesian: 2.34 tokens per word against 1.41 for English, a 66 per cent penalty. OpenAI's current o200k_base did the same job at 1.90, a 37 per cent penalty. And Gemma 2 was cheapest at 1.79, which means the 9 billion parameter Sahabat AI tokenises Indonesian more cheaply than the 70 billion one does.

Vocabulary size explains most of the ordering. Gemma 2 carries 256,000 entries, o200k_base 200,019, Llama 3.1 128,256 and cl100k_base 100,277, and the ranking on Indonesian follows that almost exactly. Two caveats I would want if I were reading this. Wikipedia articles in four languages are not translations of one another, so tokens per word compares each language's own encyclopaedic prose rather than a strict parallel corpus; characters per token, which does not depend on word counts at all, ranks the four tokenisers the same way. And encyclopaedic prose is not chat traffic. Run the script on your own logs before you budget from it.

The failures a token count does not show

None of the above is a reason to pick a locally trained model. If the token bill is your only concern, the table says use a frontier API and stop reading. The reason to consider Sahabat AI is everything a token count cannot see, and there are four of those worth naming.

• Register. Formal written Indonesian and everyday chat Indonesian behave like two different languages. A model that has read mostly news and encyclopaedia text answers a WhatsApp question in the voice of a bank letter, and Indonesian customers hear it immediately.
• Regional languages. Javanese was 3 per cent of the 8B model's 50 billion token continued pre-training mix and Sundanese 1.5 per cent, drawn from 0.40 and 0.20 billion unique tokens repeated 3.8 times. Those counts are in Llama 3 tokens, and Javanese costs about 1.9 tokens per word on that tokeniser, so it is fewer words than the figure first suggests. It is still far more than a frontier model has any commercial reason to hold.
• Code-switching. Indonesian users and developers attach Indonesian affixes to English stems constantly, as in di-deploy, staging-nya, top up-nya. No public benchmark scores this, and both kinds of model fumble it in different ways.
• Local conventions. Rupiah uses full stops as thousand separators, so Rp1.250.000 is one and a quarter million rather than one and a quarter. Addresses nest through kelurahan and kecamatan. NPWP, NIK, QRIS and e-Faktur are entities a model has either seen or has not.

Sahabat AI is not automatically better at all four either. The 9B card describes its instruction tuning as a wide range of synthetic instructions alongside publicly available ones hand-curated with native speakers, at roughly 448,000 Indonesian pairs, 96,000 Javanese and 98,000 Sundanese. Synthetic Indonesian instructions inherit whatever register their generator wrote in, which is exactly the failure mode in the first bullet. The only way to know is to test the register you actually receive.

Build the eval from your own traffic, not a leaderboard

The model cards publish benchmark results, and the benchmarks are real work. IndoMMLU is Indonesian school examination material across humanities, Indonesian language, local languages and cultures, social science and STEM at primary, middle and high school level. SEA-HELM covers question answering, sentiment, toxicity, translation in both directions, summarisation, causal reasoning and natural language inference. Neither of them is your product. One of the instruction-following benchmarks on the 70B card, SEA-MTBench, is judged by gpt-4-1106-preview against a gpt-3.5-turbo-0125 baseline, which is a useful reminder that a leaderboard position is a statement about one judge on one task set at one point in time.

// eval/run.ts -- 200 prompts lifted from real traffic, deduplicated.
// This is the only benchmark that ever settled the argument for me.
import cases from "./cases.id.json";   // id, prompt, expectedIntent, lang

const CANDIDATES = [
  { id: "frontier", ask: askFrontierApi },
  { id: "sahabat-70b", ask: askSelfHosted },   // vLLM on our own GPUs
];

for (const model of CANDIDATES) {
  for (const c of cases) {
    const started = Date.now();
    const out = await model.ask(c.prompt);
    rows.push({
      caseId: c.id,
      lang: c.lang,              // "id", "jv", or "id-en" for code-switched
      latencyMs: Date.now() - started,
      // Tokens, not requests. An Indonesian prompt costs ~1.7x an English
      // one per word, so a per-request average hides where the money went.
      promptTokens: out.usage.prompt_tokens,
      completionTokens: out.usage.completion_tokens,
      answer: out.text,
      // NOT model.id -- the grading sheet must not know who answered.
      candidate: hash(model.id),
    });
  }
}

Two hundred cases is enough to see a real difference and small enough that two people can grade them in an afternoon. Grade blind, with the model name stripped before the sheet reaches the grader, and record tokens and latency in the same row as the quality score. A model that wins on quality and loses on cost is a decision you have to make, not a result you can read off.

Self-hosting is a different cost shape, not a cheaper one

The 70B model card is refreshingly direct about what running it takes: approximately 140 GB of VRAM in FP16 or BF16, with recommended setups of two NVIDIA H100s or four L40s. That is a floor you pay whether or not anyone is chatting. The 9B is the one most Indonesian products should try first, at roughly 18 GB by the same two-bytes-per-parameter arithmetic, but its 8,192 token context rules out anything that stuffs long documents into the prompt.

# The 70B card puts FP16/BF16 at ~140 GB of VRAM and recommends
# 2x NVIDIA H100 or 4x NVIDIA L40s. That is the floor, before any traffic.
vllm serve Sahabat-AI/Llama-Sahabat-AI-v2-70B-IT \
  --tensor-parallel-size 2 \
  --dtype bfloat16 \
  --max-model-len 32768

# Start here instead. Nine billion parameters at two bytes each is ~18 GB,
# and its Gemma 2 tokenizer was the cheapest of the four on Indonesian.
# The catch is the context: 8192 tokens, covering prompt AND completion.
vllm serve GoToCompany/gemma2-9b-cpt-sahabatai-v1-instruct \
  --dtype bfloat16 \
  --max-model-len 8192

That is the honest shape of the trade. An API charges per token and nothing at idle; a self-hosted model charges for the GPU and nothing per token. Which is cheaper depends entirely on how busy you keep it, which is a question about your traffic rather than about the model. What self-hosting buys unconditionally is location. GoTo and Indosat state that the Sahabat AI chat service runs on Indosat's GPU Merdeka sovereign AI cloud, and that the data and GPU infrastructure serving the model stay within Indonesian territory or on the user's own servers. For a regulated Indonesian workload, that sentence can outrank every number in this post.

Which one I would ship, and when

For the Indonesian products I work on today I default to a frontier API and keep an open-weights path warm, rather than the other way round. The reasons are specific and none of them is that the frontier model is smarter. My traffic is spiky enough that a dedicated GPU would idle through most of the day, o200k_base tokenises Indonesian more cheaply than the Llama 3.1 tokeniser the 70B inherits, and nobody on a small team wants to be paged at two in the morning for an inference node.

I would flip that on four conditions, any one of which is sufficient on its own. Data residency is a requirement rather than a preference. The input is Javanese, Sundanese, Batak Toba or Balinese, where a frontier model has no commercial reason to be good and Sahabat AI has at least been trained on some. Volume is steady enough to keep a GPU meaningfully busy. Or the eval says so, which is the only one of the four that actually settles the question, because the other three are inputs to it.

The useful question was never whether a local model beats a frontier one. It is which of them is better at the exact task you ship, in the register your users actually write, at a cost you have measured rather than assumed. Sahabat AI's model cards are unusually honest about what it is and what it needs: the tokeniser is inherited, the regional-language data is thin, the 70B wants 140 GB of VRAM. Read them, then go and build the two-hundred-case eval. It is a day of work, and it ends an argument that otherwise runs forever.