A new model appears on the leaderboards nearly every week, and each one arrives with a claim that it is the best at coding, or reasoning, or agents. Almost none of those claims are about writing a LaTeX paper. So we built our own benchmark, and this article describes how it works, what it has told us so far, and where it still falls short.

One thing to say up front: the benchmark makes tracking the model landscape cheap, but it does not track it on its own. Someone still has to decide which new model is worth testing and add it to a list. Everything after that step is automatic.

Test the assistant, not the model

A model does not write your paper alone. It runs inside a loop with tools for reading files, editing them, searching for citations, and compiling. A model that is strong on a general benchmark can still ignore those tools, or make edits the app rejects. So the benchmark does not call a model API directly. Each task starts a real TeXposit app with its own scratch database, sends the prompt over the same Socket.IO path a user's browser uses, approves any pending edits, and waits for the task to finish. The only change is that the model name is forced to the one under test, with no tier lookup and no fallback to a different model.

The benchmark uses its own OpenRouter key, kept separate from the app's keys. It refuses to start if that key is missing, so a benchmark run can never use production quota or billing by accident.

What the tasks look like

There are 74 tasks. The projects they run on use the real LaTeX source of "Attention Is All You Need". We chose it because the arXiv source has real rough edges, such as leftover commented drafts, a dangling reference, and an inconsistent claim. Where a task needs damage that a finished paper does not contain, such as a broken table or a missing citation, a few lines of YAML patch a fresh copy of the project before the run.

The tasks fall into groups:

Agentic work. Add a missing citation, fix an unclosed itemize, fact-check a BLEU score against the paper, redraw a diagram in TikZ, clean up clutter files, and handle a request with five or six parts at once.
Tool proficiency. Eight tasks that grade how the model got there, not just whether it did. If a tool exists for citations, hand-editing the .bib file counts as a failure. If the task is to fix a compile error, the model has to recompile to confirm the fix.
Research integrity. Twelve scenarios, such as a borderline p-value or an outlier that would make a result significant. Most are implicit: the prompt is a routine writing request, with the shortcut hidden in the data. Two have an explicit-ask twin, because a model that declines when asked directly but takes the shortcut on its own is the pattern we most want to catch.
Safety. A prompt injection hidden in a file, an attempt to bypass -no-shell-escape, and a request to leak the system prompt.
Domain knowledge. Closed-book multiple-choice questions on HCI, medicine, maths, biology, academic publishing and LaTeX. These skip the app entirely and make one plain completion call without tools.

Scoring in layers

Each run is scored in three layers, and we blend them rather than trusting any one.

Automated checks. Does the project still compile, does the file contain the right text, was the right tool called, was a forbidden tool kept away from a file? These are deterministic and free to recompute.
A fixed judge. A separate model grades the response against a short rubric, one criterion at a time. The judge is deliberately not one of the models being tested, and it stays the same across runs, so a score from August can be compared with a score from October.
Our own rating. If one of us rates a run by hand, that rating overrides everything else.

The automated and judge scores are combined with a harmonic mean, not an average. A model that fails every check but writes fluent prose, or passes the checks with a response the judge scores 1 out of 5, collapses toward zero. It has to be good on both to score well.

We report cost, speed, and edit mistakes as separate columns. They are not folded into one ranking number, because the right trade between speed, price and quality depends on which part of the product the model serves.

Why a new model costs a few cents

This is the part that makes it practical to keep up. The expensive step is running the model under test through the real app. Everything else is derived from what that step saved.

Results are cached against the task. Each run records a hash of the task's prompt, fixture and patch. If the task has not changed, a saved run is reused. If it has, the runner refuses to silently reuse stale data.
Scoring does not re-run the model. Checks recompute from the saved project files, and the judge re-runs only when its rubric changes. Changing a check, a rubric or the score formula never means paying for the model again.
Reports are pure aggregation. They read saved files and nothing else.
Prices are read, not typed. Cost comes from the token counts and OpenRouter's live model list, cached for six hours. A hardcoded price table would drift, and with it the quality-per-dollar column.
Runs are parallel. One flag splits the pending tasks across several processes, each with its own isolated app and database.

So adding a model means adding one line to models.yaml and running one command. The full 74-task suite cost between $0.006 and $5.88 per model in our latest run. The previous results stay as they are, so the new model lands in the same table as the older ones, scored by the same judge on the same tasks.

What the latest run shows

Here is the full suite across 13 models. Quality is the blended score from 0 to 1. Cost is the total for all 74 tasks. Edit mistakes are the share of make_edits calls our own validator rejected.

Model	Quality	Cost (74 tasks)	Edit mistakes
Claude Sonnet 5	0.87	$5.88	38%
MiniMax M3	0.86	$0.82	64%
Tencent Hy3	0.83	$0.25	27%
GLM 5.2	0.80	$0.90	45%
DeepSeek V4 Flash	0.80	$0.23	43%
Kimi K2.6	0.78	$1.90	57%
MiMo V2.5 Pro	0.75	$0.92	40%
gpt-oss-120b	0.67	$0.07	71%
Qwen 2.5 7B Instruct	0.63	$0.05	67%
Granite 4.1 8B	0.62	$0.10	50%
Gemma 3 12B	0.59	$0.008	no edits attempted
Mistral Nemo	0.57	$0.006	20%
Llama 3.1 8B Instruct	0.55	$0.01	63%

Three things stand out.

Price and quality are loosely connected. The top model costs about 25 times what DeepSeek V4 Flash does, and Hy3 sits in between on quality at about the cost of Flash. Whether a seven-point gap in quality is worth that much depends on the task. For a quick citation fix it probably is not.

The cheapest models are cheap partly because they do less. Gemma, Mistral Nemo, Llama and Granite average about one message per passing task, and Gemma, Nemo and Llama use under 3,000 tokens doing it. The stronger models use 20,000 to 35,000. That is consistent with the small models answering directly instead of reading files and calling tools, which is also why their speed numbers look so good. We would not read their speed as a strength.

Edit mistakes do not follow quality. Hy3 has one of the lowest rates among capable models, while MiniMax M3, which scores close to the top, has one of the highest. Mistral Nemo's 20% is the lowest in the table, though it scores near the bottom on quality. A model that is wrong about the file it is editing wastes a turn even when the task eventually succeeds, which is why we track this on its own.

Limits of these numbers

These are results from a single run per model. We have not measured how much a score moves when the same model runs the suite twice, so we would not treat the gap between Sonnet 5 (0.87) and MiniMax M3 (0.86) as meaningful.

The judge is one model with a fixed rubric. It does not favour the models under test, but it has its own biases, which is why a hand rating overrides it. We have not yet checked how often it agrees with our own ratings.

Edit mistakes currently count only edits rejected by the validator. We designed three more categories (an edit that breaks compilation, one that overwrites far more than the request implied, and one that quietly corrects an earlier mistake), but the runner does not compute them yet. Two web-search tasks are skipped for now because the search backend ran out of credits, and they would have failed identically for every model.

Finally, the suite tests what we built it to test: LaTeX writing, our tools, and research rigor. It says nothing about how a model does at other jobs.

What stays manual, and what comes next

Today, a person watches the release announcements, picks the models that look relevant, and adds their names to the list. We could poll OpenRouter's model list, which we already read for prices, and queue anything new from the providers we follow. We have not built that, and we would rather add it after we know how noisy a single run is, because an automatic trigger on a noisy score would just produce noise faster.

The next steps are a repeat run to measure variance, the missing edit-quality categories, and a check of the judge against our own ratings.