Worth Accepting: Evaluating Mellum at JetBrains
I led evaluations for the first Mellum. Most of the hard problems were not about the model.
Every time you see a cloud completion in your JetBrains IDE, chances are you are interacting with Mellum, JetBrains’ family of open-weight LLMs for code. In 2023, the task was to build the first one: JetBrains’ own Copilot, trained and productioned entirely in-house. It shipped in 2024, small enough to be efficient but good enough to be useful for hundreds of thousands of developers worldwide. Before shipping, we needed to ensure the model is not shabby.
Mellum is definitely not a frontier model. By design, it is a fairly small 4B Llama 2-like model trained on permissively-licensed code. One distinction worth making explicit: Mellum does code completion, not code generation. Code generation turns natural language into code. Code completion turns code into more code. The model uses fill-in-the-middle (FIM): the code before the cursor becomes the prefix, the code after it the suffix, and the model generates what goes in between.
The problem space is narrower than generation, which means we can lean on context (imports, type definitions, surrounding functions) to evaluate quality with higher confidence. I was responsible for model evaluations, which is a good place to sit: close enough to know the subtle nuances, distant enough to see the big picture.
Mellum2 is out now; I was not involved in it. What follows is what we learned building the first one, and most of it should apply to whatever comes next.
Separating Rejection from Selection
The most important strategic decision in our evaluation process was keeping two jobs clearly apart: offline evaluation existed to reject bad configurations, and online evaluation existed to select among the good ones. This is not a new idea. Netflix described it well back in 2013, and we followed their framework deliberately.
Due to the high number of moving parts (context collection method, FIM splitting strategy, prefix and suffix trimming, stop tokens, programming language, library usage, single-line vs. multi-line completion), we needed a multi-step process where each stage was more expensive and more directly connected to real user behaviour than the last. A private offline benchmark ran on every checkpoint: scalable, fast, and easy to automate. LLM-as-a-judge and public benchmarks provided relative comparisons. End-to-end IDE runs closed the offline loop. Finally, A/B testing on real users made the call: the most informative signal, but also the most demanding to run.
Traceability Is Not Optional
Most public benchmarks target generation, not completion, which our small FIM model was never designed for. Those that do target completion are rarely a good fit: wrong language mix, no guarantees about data freshness, and no connection to how IDEs actually collect context that materially impacts the quality. This is one of the strongest sides of Mellum: while we did not innovate much on the model architecture, we tailored the whole pre-training and post-training data pipelines to our in-IDE use case. We needed something we could trust, so we built JetComplete. The investment was significant, but it gave us full control over what mattered: language coverage, data freshness, consistent prefix/suffix/middle splitting, in-IDE preprocessing, and a clean validation/test split.
JetComplete covered the full set of languages Mellum needed to support (Java and Kotlin, JavaScript and TypeScript, Python, PHP, and others) under one set of principles, with a deliberate focus on fresh code to avoid data leaks. Code was processed with in-IDE tools and filtered through an LLM-as-a-judge step, then scored on exact match, chrF, and the KK score, a proprietary relaxed Levenshtein metric.
The pipeline ran on every checkpoint via Kubeflow: download the dataset, split it, run inference across GPU machines, combine the results, compute metrics with confidence intervals, upload reports. Over the course of the project, that meant thousands of checkpoint evaluations. Every number was traceable to the exact step that produced it. That lineage saved us more than once, and made me miss Yandex Nirvana, an excellent job orchestrator, even more.
If the pipeline needs explaining, it will not get adopted by the people. Evaluations should start automatically on each new checkpoint and be scriptable end to end. One thing that is easy to overlook: evaluation is a high-throughput task, not a low-latency one. The goal is to score thousands of completions in batch overnight, not to serve a single result in milliseconds. It is also the most compute-intensive offline stage and routinely the first thing cut when training budgets are under pressure; protecting evaluation compute requires the same deliberate attention as protecting training compute.
Against Benchmaxxing
JetComplete gave us absolute scores per configuration. To additionally compare models against each other, we ran pairwise comparisons using an LLM judge and aggregated the results with the Bradley–Terry win rate model through Evalica, an open-source toolkit I built for exactly this kind of evaluation. The result is a ranked comparison of configurations by win rate.
We also ran public benchmarks: SAFIM, RepoBench-C, and HumanEval-Infilling. The working assumption is that any open-source benchmark is effectively training data for any model developed today, which makes optimising for those numbers self-defeating. We ensured no contamination, but leaned on our internal benchmark and live data instead.
If you are winning on every benchmark, you are losing it; review your protocol immediately. We invested in result analysis and cross-examination methods, and eventually collaborated with Mistral AI to organise a shared task at ASE 2025 focused on exploring the best context collection strategy.
Dogfooding
Each IDE team independently maintained their own set of representative projects as quality gates. Running the model through those projects, inside the actual IDEs rather than a test harness, let us catch divergence between benchmark scores and real product behaviour. In one case, a mistake in the request schema to the inference engine went unnoticed in all prior stages but was caught as soon as the evaluation ran through the real product. JetBrains has a strong dogfooding culture, and it was useful: using your own product is what makes that kind of catch possible.
Metrics and Habits
Our A/B tests covered more than just model swaps; we tested result filtering, context collection strategies, and other configuration changes against production. We measured acceptance rate and total characters accepted, the fraction of model-proposed code that developers actually kept. Running an experiment meant landing the change in a scheduled release train over multiple products, waiting for the data, and then processing the experimental results. The cycle took weeks.
This is also where any new metric has to earn its place with the wider organisation. People usually have strong opinions about what to measure, and those opinions are slow to change. Replacing a metric outright rarely goes well; evolving it incrementally does. Getting teams to adopt a new number requires finding internal champions before the number is even ready, but even then, adding a metric is not enough. Trust in it has to be built separately, and that takes time.
Questioning the Obvious Design
My initial expectation was that training the model on a specific language like Python or Kotlin would outperform a multilingual model on that language. It did not. As our pre-print reports, performance on our private benchmark did not suffer from multilingual training at all. In contrast, the DPO version of Mellum trained on multiple languages outperformed all other configurations in most cases.
The base model actually scored highest on RepoBench-C, a reminder that fine-tuning trades one kind of performance for another. The most natural design assumption (one specialist model per language) was not the best one.
Results
The multilingual + DPO configuration won on our private benchmark and most pairwise comparisons; see the pre-print.
On public benchmarks the picture was more mixed: 52.17% on SAFIM (pass@1), 53.76% on HumanEval-Infilling (pass@1), 0.69 on JetComplete KK. Competitive but not uniformly dominant, which is a healthy sign of non-benchmaxxing. In online evaluation, Java and Go showed the highest acceptance scores; Python sat in the middle.
We ended up with one model for all languages instead of one per language. At 4B I do not think we had saturated Mellum’s capabilities. That is neat.
Developers said the completions were spot-on and saved them time. This is the most impactful form of feedback one could receive.
Beyond Mellum, I was lucky to work on several related problems: overall model description (arXiv 2510.05788), context engineering competition (workshop at ASE 2025), LLM-as-a-Judge tooling (demo at COLING 2025), pairwise comparison limitations (SRW@ACL 2025). The team is now working on mode advanced AI tools for developers; with my freshly minted JetBrains Alumni licence, I am looking forward to using it.
On my last day at the Belgrade office, I brought Medovik and Napoleon cakes. Mellum was already running in every JetBrains IDE, and the completions were worth accepting.