The Starknet Coding Index (SCI)

How the score is built

An attempt is a submission, not a turn: thinking, extra turns and documentation lookups never reach you, so they are free, while code that arrives broken is not. Runs over the 15-minute model-time budget count as failures. Where we sent no effort parameter, the label is the level OpenRouter documents as that model's default: max for Kimi K3, high for Hy3 and Sonnet 5. The scales are fixed, not relative: adding a new model later never changes an existing score.

Starknet Coding Index (baseline, no assistance)

One number per model for "how good is this LLM at writing Starknet smart contracts today", weighted toward the thing you actually get: working code on the first submission. Each model runs the full task suite alone, at its best thinking variant (labeled in parentheses), within a budget of 10 turns and 15 minutes of model time per task.

Starknet Coding Index, top 15 of the 27 models tested

Behind the score

The same top fifteen, winning variants unpacked, baseline condition. The first chart is the whole distribution behind the effectiveness score: every column covers 100% of that model's runs, split by whether the code worked on submission one, two, three, or later, and topped by a warm-tan band ("never solved" in the legend) for the runs that never worked. Solve rate is everything below that band. Cost and time are the median of a complete pass over the 13-task suite. Each chart ranks best first.

How many submissions until the Cairo code works

Median cost of a full 13-task pass

Median model time for a full 13-task pass

The thinking dial

Same index, baseline runs only: the top eight models, each drawn across its own effort ladder. The ring marks the tier the leaderboard scores.

Things to look for: Muse Spark 1.2's dial splits in two, minimal through medium landing at 66 to 71 and high and xhigh at 80. All three Anthropic models pay for max: Opus 5 peaks at low and gives back 7 points, while Fable 5 and Sonnet 5 hold plateaus that max drops by 5.6 and 10. Kimi K3 is level at off, low and its max default, dipping only at high. Grok 4.6 is the only dial here that climbs the whole way, 84.4 at low to 90.3 at xhigh, and it buys first-try success rather than correctness: 62% to 87% one-shot with every tier already at 100% solved. Gemini 3.7 Flash is the same story with the opposite ending: its one-shot rate climbs just as honestly, 71% to 83%, but the ladder costs it 2.8× the money and 2.6× the wait, so the index hands back nothing and its three tiers finish within 2.9 points of each other.

Fair questions

Head to head: best closed vs best open weights

The ranking's two champions, Opus 5 (low) from Anthropic and Kimi K3 (low) from Moonshot, both solve every task; the gap is in how. The second chart shows it is not about difficulty: Opus 5 first-tries every tier while Kimi sits near 70% across all three. Baseline condition, 26 and 52 runs.

Opus 5 (low) vs Kimi K3 (low): five measures, best baseline configs

First-submission rate by task difficulty, Opus 5 vs Kimi K3

What does the Cairo Coder MCP add? (best config without vs with)

Same index, second question: each model's best configuration without the tool (solid bar) versus its best configuration with it. Each condition picks its own best thinking level, and the labels show it: thirteen of the 27 models win at a different effort with the tool than without, and seven of those thirteen move down the ladder, not up. Documentation substitutes for thinking budget. Four of the thirteen switchers (DeepSeek V4 Flash 0731, Hy3, GLM 5.3 and GLM 5.2) rank below the cut and change effort where no chart on this page shows it.

Cairo Coder documentation lift, top 15 of the 27 models tested, by best score in either condition

Local-inference class

The models you could run yourself. The test is memory rather than parameter count: the published Q4_K_M weight file has to fit the 128 GB of unified memory that today's biggest buyable machines top out at (a DGX Spark, a Strix Halo box, an M5 Max MacBook Pro), minus 16 GB reserved for the OS and a KV cache. Total parameters count, not active ones, because every weight has to be resident even when a sparse model fires only a few experts per token.

Q4_K_M draws the line because it is the default 4-bit quant, the file most people actually download and run, and the sweet spot where memory halves against 8-bit while quality holds. Below 4-bit, degradation stops being minor, and coding feels it first. These scores were measured on the full-precision models, so the class stops at the quant level where the local copy still resembles the model that was scored.

Documentation lift for the models that fit one 128 GB machine

The models

Every model tested, including the twelve below the chart cut. Use it to shortlist by the constraint you actually have: the index with and without the documentation tool, price (per million tokens, as listed on OpenRouter, 2026-08-01, or at first listing for models published since), speed, and whether the weights are open. * weights promised but not yet released. Click a header to sort.

Which quantization for my machine

Pick your memory, read across. Each cell is the best quantization that fits, named as the file you would download from the bartowski and unsloth GGUF repos, and what it weighs in GB. Every figure is a real published file, not arithmetic.

The weights do not get the whole machine, so each column reserves room for the OS, the KV cache and activations: 16 GB at 128, 8 GB at 64, 4 GB at 32, 4 GB at 24. A quant in full black is Q6 or better, where quantization is not the thing holding the model back; a greyed one is a 4-to-5-bit compromise.

A blank means nothing 4-bit or better fits, which is a real answer rather than a missing one: below 4-bit degradation stops being minor and coding feels it first, so no cell recommends going lower. The ten open models not listed need more than 128 GB even at 4-bit: Kimi K3, MiMo-V2.5-Pro, MiniMax M3, Qwen3.8 2.4T A95B, DeepSeek V4 Flash 0731, DeepSeek V4 Pro 0813, Hy3, Inkling, GLM 5.3 and GLM 5.2.

Findings