cnm · bench notes · live · October 2026
Little MoE bench notes
Follow-up to I only wanted to run a little MoE. Measured results only. Medians of 3 unless stated, one agent run per task.
Live notes. The two-bank file for a 16 GB + 8 GB pair has its first GPU numbers (below); its KL covers 2 of 7 texts so far, and no real 8 GB card has run it yet. PR-6 L3b on i7 and the remaining PR-6 tasks on the 8 GB box are paused. This page fills in as results land.
11 / 11 ×2SMOKE tasks solved at home, with flat −0.75 and with graded cap −1
+10.5%decode on an 8 GB card with the graded bias (18.92 vs 17.13 t/s)
49.0 t/sgraded bias at matched strength on 2×16 GB, vs 47.4 flat
0.505KL cost of graded cap −1, vs 0.570 for flat −0.75 at the same strength and 0.904 for a hard mask
The recipe
Recorded agent sessions give a per-(layer, expert) profile, split by phase (explore, edit, verify). The hot set is sized to the GPU cache. A graded bias nudges the router toward it, the upstream SLRU expert cache keeps it resident, and MTP drafts ride on top. All 512 experts stay reachable.
Graded versus flat bias on an 8 GB card
i5: RTX 2070 8 GB, PCIe 3.0, full IQ2_XS 512 experts, expert cache 3.5 GB, pinned memory, -ub 512, no MTP. The graded bias is max(cap, ln(b / b_edge)) on the cold set: experts just below the hot-set edge are barely penalised, the deep tail gets the cap.
other armsbest armno bias
Decode, tokens/s
Prefill 8K, tokens/s
Expert-cache hit rate, %
The graded bias with a −2 cap wins on all three. A hard PB3-64 edge does not beat the bigger PB3-192 set. Caveat: the cap −2 file has a mean penalty of −1.4, stronger than flat −1, so part of the gain may come from more bias; a flat −1.4 control is owed, and quality under it is not measured yet.
Table view
Graded versus flat at matched strength on 2×16 GB
i7: two RTX 4060 Ti 16 GB, full IQ3_S 512 experts, expert cache 17 GB, MTP drafts, -ub 512, pinned memory. Four arms in one session with flags identical to the reference run. The graded cap −1 has a mean penalty of −0.80, matching the flat −0.75, so the difference comes from the grading itself.
flat −0.75 referencegraded cap −1 (chosen)stronger caps
Decode, tokens/s (MTP acceptance in brackets)
Prefill 8K, tokens/s
Graded cap −1 is +3% decode and +4% prefill over flat at the same strength. Prefill rises steadily with the cap. The cap −1.5 decode dip is MTP acceptance (62%): a stronger bias changes the generated text and the draft head guesses it worse. One synthetic prompt, so decode is noisy; prefill is the steadier signal. Choice rule fixed in advance: the fastest graded cap, unless within 3% of cap −1, in which case cap −1. The flat reference repeated the previous day: 47.38 vs 47.98 decode.
Table view
The price of the router bias
A bias changes which experts the router picks, so it changes the output. KL divergence measures how far the next-token distribution moves: full IQ3_S 512 with each bias file against the same model without bias, 7 held-out agent texts, 120,773 scored tokens, llama-perplexity on one 96 GB cloud GPU. Lower means closer to the unbiased model. The hard PB3-192 arm masks every expert outside the hot set, which is what pruning to 192 experts per layer does.
hard mask (= pruning)flat −0.75 referencegraded cap −1 (kit default)other arms
Graded cap −1 on PB3-192, the default in the kit, moves the output less than flat −0.75 at the same mean strength (0.505 vs 0.570, lower on all 7 texts) and less than half as much as the hard mask (0.904). Only flat −0.5 is gentler, and it is also the weakest nudge. A smaller hot set costs more: graded cap −1 on PB3-64 is 0.621. KL is not pass rate: flat −0.75 and graded cap −1 both solve 11 of 11 SMOKE tasks below, and no task result yet shows the gap between them.
Table view
Source: runs/2026-10-28-s19-d-pod/ (REPORT.md, result.json, bias block).
Cheaper cold experts: what each format costs
In a hot/cold split the 256 most-used experts per layer (PB3 top-256) stay in IQ3_S and the other 256 take a smaller format. KL against full IQ3_S 512 on the 3 texts that every variant finished (61,410 scored tokens), all on GPU. Size is the routed-expert bytes only.
besthot/cold splitswhole model IQ2_XS, no split
The all-Q2_0 cold bank is both smaller and closer than the IQ1_S mix: 4.27 GB less and lower KL on every shared text. Our own Q2_0 encoder with an importance matrix comes close to the published Q2_0 bytes but stays behind on every shared text (0.349 vs 0.321); it matters for models that have no published Q2_0. The plain IQ2_XS file is 6.7 GB smaller than the Q2_0 split at almost twice its KL. KL only; no agent tasks run on these files yet.
Table view
Source: runs/2026-10-28-s19-d-pod/ (REPORT.md, result.json, common-3 texts).
Two-bank file on GPUs: first numbers
The two-bank file keeps 144 hot experts per layer in IQ3_S and the cold rest in Q2_0, 71.88 GB. It runs on i7 in two layouts: 16 + 8 GB, with the second card held to 7.6 GiB (peak 14,030 MiB on the first card and 7,366 MiB on the second), and 2×16 GB with MTP on the second card. MTP does not fit the 16 + 8 layout. The engine carries a fix for a crash on the first request that is not released yet. Speed: /completion, 2,000-token prompt, 128 greedy tokens, three servers × three runs interleaved, median, spread ≤0.4%. This is a different tool and prompt from the speed sections above, so the numbers do not compare with them.
| Layout | Bias | Decode t/s | Prefill t/s | Cache hit |
|---|
| 16 + 8 GB, no MTP | off | 12.05 | 102.95 | 64.2% |
| 16 + 8 GB, no MTP | graded cap −1 | 17.87 | 139.70 | 84.3% |
| 2×16 GB, MTP | off | 14.12 | 101.09 | 79.5% |
| 2×16 GB, MTP | graded cap −1 | 30.08 | 137.32 | 93.3% |
The graded bias gives +48% decode on 16 + 8 and doubles decode with MTP on 2×16, through the cache hit rate. KL against full IQ3_S on 2 of the 7 texts: 0.579 / 0.478 without bias, 0.762 / 0.602 with graded cap −1. Caveat: the same plain IQ3_S file scored against an i7 reference and against an i5 reference differs by KL 0.127, with an identical engine build. So the i7 KL values here do not compare with the cloud-GPU KL values above, only with each other. KL is not pass rate, and no agent tasks run on this file yet.
Source: runs/2026-10-28-v05-horse-i7/REPORT.md.
Strata (fork) on i7
Strata (MIT) is a separate MoE inference engine with its own expert cache and a selection prior that steers the router toward a profile. Our local patch to it (PGO-SGB, not published) loads the same PB3 keep sets and the same graded bias. Same i7, same model pack, MTP spec 4, greedy, one 1,628-token ABAP prompt, 128 tokens, median of 3, arms interleaved. These numbers come from Strata's own runner and do not compare with the llama.cpp numbers above.
Gates pass: parity on both GPUs; an all-zero bias table gives the same 128 greedy tokens as no table (the single-card engine is not fully deterministic run to run, so this gate carries a caveat); a −inf bias on the top 3 experts changes routing in all 48 layers.
offgraded cap −1other arms
One card (cache ~100 experts per layer), decode tokens/s (hit rate in brackets)
Two cards (cache ~223 experts per layer), decode tokens/s (hit rate in brackets)
On one card the graded bias lifts a PB3-96 keep set by 16.5% over the same set without bias (41.36 vs 35.49), but Strata's own profile with the selection prior is faster still (58.14). On two cards all arms sit between 61.5 and 65; graded cap −1 on PB3-192 is the fastest at 64.94, +5.5% over off. Bigger keep sets on two cards (graded cap −1): PB3-224 58.69, PB3-256 64.78, PB3-320 64.92; the PB3-224 dip follows MTP acceptance (0.607) on that token path, not the hit rate. One prompt, so ±5% can be token-sequence noise. Approximate KL against off, top-20 plus tail and only over the positions before the texts diverge (about token 15–19): graded 0.029, profile prior 0.033. That is not a quality measurement; pass rate under Strata is not measured, nor more than one prompt or the HTTP path.
Table view
Sources: runs/2026-10-28-strata-i7-gates/REPORT.md (gates), runs/2026-10-28-strata-i7-serve-pb96/REPORT.md + result.json (all speeds and KL shown).
Strata: how far each arm moves the output
KL against Strata with no bias, 2 cards, 8 texts (16,884 positions per arm): ABAP code, an agent session, Russian prose, an English story, and the same UI text in Russian, German, Chinese and English. This is forced-window KL: comparable between arms, not the decode distribution. A self-check of the forced read against real decoding fails (same top token on 183 of 192 positions), so these values rank the arms in one scoring path and are not absolute. Floor: off against off is exactly 0. Negative control (top 3 experts banned): 0.239.
Arms: prior 1.5 is Strata's own profile with its selection prior. Graded, selection only applies graded cap −1 on PB3-192 to which experts are picked. Graded, damped weights also feeds the bias into the expert mixing weights. Residency only keeps PB3-192 resident with no bias.
residency only (control)graded, selection only (kept)other arms
The selection prior moves the output most (0.546, 3.1× the selection-only graded bias), and non-English UI text takes the biggest hit under it. The damped-weights variant is worse than selection only on every text (0.248 vs 0.176). Speed gives no reason to prefer it: damped against selection only is −7.3% (2 cards, PB3-192), −1.4% (2 cards, PB3-256) and +4.7% (1 card, PB3-96), with no consistent sign, on a single prompt. We keep the selection-only graded bias, not damped weights. Pass rate under Strata is not measured.
| Arm | All 8 | ru-ui | de-ui | zh-ui | en-ui | abap | Same top % |
|---|
| prior 1.5 | 0.546 | 0.939 | 0.579 | 0.283 | 0.190 | 0.879 | 71.5 |
| graded, damped weights | 0.248 | 0.271 | 0.278 | 0.213 | 0.148 | 0.461 | 79.4 |
| graded, selection only | 0.176 | 0.166 | 0.172 | 0.133 | 0.107 | 0.404 | 82.8 |
| residency only | 0.064 | 0.026 | 0.030 | 0.025 | 0.033 | 0.249 | 90.5 |
Source: runs/2026-10-29-strata-kl-i7/REPORT.md + result.json (kl_2card, speed).
Agent tasks on the full model at home
Full IQ3_S 512 on i7, flat bias −0.75, MTP, Claude Code harness at xhigh effort, time limits ×5, graded by hidden tests. Bar length is wall time. osg-l0-body-003 is excluded from SMOKE: no model has solved it.
solved, wall time
11 of 11 solved. Two tasks show no time: block-015 and core-body-018 ran while a second agent shared the one-slot server, so only their result counts. Earlier runs on the same tasks: line-018 1,494 s and apc-002 1,489 s on the older standard-offload setup; block-015 900 s on PB3-208 in the cloud. Graded-bias runs of the same tasks are compared below.
Table view
Same tasks: flat −0.75 vs graded cap −1
The same 11 SMOKE tasks, the same server and harness, only the bias file changed. Both arms solved 11 of 11. Grey bars are flat −0.75, teal bars are graded cap −1. Two flat times are missing because those runs shared the server.
flat −0.75graded cap −1
On the 9 tasks with clean times in both arms, total wall time was 5,374 s flat and 4,820 s graded (−10%); median 530 s vs 388 s. Calls to the model, all 11 tasks: 303 flat, 325 graded. The arms trade places task by task: graded is much faster on apc-002 and core-l1-013 and much slower on pia-block-001. One run per task, so single-task differences are mostly the agent taking a different path; the totals are the signal, and still a small sample.
Table view
First field run on Windows: one 16 GB card
A tester ran the packaged kit (cnm-serve v0.3) on an RTX 4080 16 GB with 128 GB RAM under Windows 10, using the full IQ3_S 512 with the graded bias cap −1 and no MTP. The client was the Continue extension in VS Code.
| Setup | Decode t/s | Note |
|---|
| RTX 4080 16 GB, Windows 10, no MTP | 40.6 | from the llama-server log, one reply of 435 tokens at about 3K context |
| Estimate before the run | 20–35 | computed from VRAM, not measured |
| Two RTX 4060 Ti 16 GB (i7), no MTP | ~33 | for comparison; with MTP 48–52 |
A single card beat the estimate and the two-card box without MTP. A faster GPU and a PCIe 4.0 slot (i7 is PCIe 3.0 x8) are the likely reasons, but this is one reply, not a median. Prefill was not measured: that prompt was 92 tokens. GPU "shared memory" showed about 47 GB, which is the pinned host memory holding the experts, as intended. The first Windows run also found a download bug, fixed in v0.3.
Repository tasks (PR-6) on the full model at home
PR-6 tasks are cut from real merged pull requests, from L1 (small) to L3 (large), and are graded by the PR's own hidden tests. Full IQ3_S 512 on i7, graded bias cap −1, MTP, Claude Code harness, time limits ×5.
| Task | Result | Time | Patch lines |
|---|
| L1a-osg678 | solved | 11.5 min | 71 |
| L1b-osg680 | solved | 45 min | 706 |
| L2a-osg693 | solved | 81 min | 352 |
| L2b-abapiti87 | solved | 84 min | 1,521 |
| L3a-minz39 | solved | 120 min | 772 |
| L3b-abapiti95 | paused (run stopped at ~55 min, to be redone) | | |
L2b is the one that separates models. On the same bench, by executor:
| Executor | L2b |
|---|
| Opus (Claude Code) | solved, 20 min |
| Sol 6.1 (Codex) | solved, 7 min |
| Qwen3.8 full, cloud GPU (Claude Code) | solved, 30 min |
| K224 cut on i7 | solved, 99 min |
| Full IQ3_S at home, graded bias cap −1 | solved, 84 min |
| K192p cut on i7 | failed (looped) |
| K224 + MTP + n-gram | failed |
| GLM-5 (Pi), Kimi K2.5, MiniMax M2.5, Qwen 3.7 Plus | failed |
| Luna medium / high (Codex) | failed (high solved it only with runnable examples) |
The home model, with every expert kept and the graded bias, solved the task that half of the cloud models and our own K192p cut fail. L3b is the hardest; so far only Opus, GLM-5.3 in Claude Code and K192p have solved it. Its home run is paused.
8 GB card, own model (i5: IQ2_XS, graded PB3-64 cap −2, 64K context): L1a solved in 45 min. L1b failed after 52 min, but not on the model: Claude Code stopped with "autocompact thrashing". The 64K context refilled to the limit three times right after compaction, while the agent read large files. The 8 GB setup needs a larger context, or the agent needs smaller tool outputs. L2a failed after 41 min with an empty patch. The remaining tasks on this box are paused.
Source: bench lane bench-s14-gate-smoke, runs/L*/<executor>/result.json (grader rc 0, leak_suspect false); other executors from the same lanes on i5; 8 GB L2a: runs/L2a-osg693/custom-i5-iq2xs-soft64c2-claude-20261028-102713/result.json, park log runs/2026-10-28-i5-pr6-soft/park.log; i7 L3b stop: session state log, 2026-10-28 11:00Z.
One task, four runs each
Single runs are noisy, so open-abap-apc-l1-method-002 ran four times per arm: three runs interleaved F G G F F G, plus the first night run. All 8 runs solved it.
flat −0.75graded cap −1| = median
Median wall time 631 s flat vs 434 s graded (−31%). Median calls to the model 23 vs 11.5. The graded agent took fewer, longer turns. The spread is large: the fastest flat run (445 s) beats the slowest graded run (564 s). A rank test on 4 + 4 runs gives p ≈ 0.1 one-sided, so this is a consistent lean toward graded, not proof.
Table view
Other 8 GB findings
| Question | Result |
|---|
| Pinned vs mmap host memory | Pinned decode 17.06 vs 16.21 t/s (+5%), prefill 140.5 vs 112.3 (+25%); A/B interleaved. Huge pages gave nothing measurable. |
| IQ3_S with bias (cache 2.5 GB) | 11.93 → 12.38 (−0.75) → 12.79 t/s (−1); hit 44% → 53%. |
| MTP on the 2070 | Loses: 11.0–14.0 t/s against 17.13 without it. The draft head leaves too little cache; hit drops to 1–11%. |
| Phase-aware preload | Misses after a phase change 42.4% vs 41.7% within a phase. A small cache turns over every request; closed for 8 GB cards. |
Sources: runs/2026-10-27-lru-thp-i5/ (REPORT.md, bias-runs/*/result.json), runs/2026-10-27-tiers-speed-i7/*calib*/result.json, bench lane bench-s14-gate-smoke runs/<task>/custom-fulliq3s-b075-claude-*/result.json; task package 573676f.