Nudge, don't cut

Experts that don't fit

A Mixture-of-Experts model has many small feed-forward blocks per layer, the experts, and a router that picks a few of them for every token. Qwen3.8-Flash-Next has 48 layers × 512 experts and picks 10. GLM-5.3-Flash has 288 experts per layer. Each token touches only a small subset, but every expert has to live somewhere.

My box is an i7-10700K with 128 GB DDR4 and two RTX 4060 Ti 16 GB on PCIe 3.0 x8. (I'll call it i7, after its host name; the second machine, i5, is a Ryzen box with an RTX 2070.) The Qwen IQ3_S files are about 55 GB of weights plus a 29 GB embedding table; the GLM UD-IQ3_XXS files are about 120 GB. Neither fits into 32 GB of VRAM. So the engines keep the bright experts on the GPUs and the rest in RAM, where the CPU computes them or PCIe fetches them. Every token that wants a cold expert pays for it.

Profiles of my agent sessions say that a minority of experts carries most of the router's mass, and which minority depends on the work: exploring a repository, editing, running tests. I called the phase-balanced version of that profile PB3 in the previous article: explore, edit and verify get equal votes.

There are two ways to use such a profile:

• Hard mask: cold experts can never be picked. Simple, and the model only ever needs the hot set. But every token that really needed a cold expert gets a substitute.
• Soft bias: cold experts get a negative number added to their router score. They are picked less often, but a token that really wants one still gets it.

On Qwen3.8-Flash-Next with the same PB3-192 hot set (192 experts per layer kept), the hard mask gave a mean KL of 0.904 against the full model; the graded bias with cap −1 gave 0.505, and a flat −0.75 gave 0.570. To be honest about it, a flat −0.5 was closer still, at 0.455. At the same cap, a flat −1 gave 0.650. A weaker nudge is closer to the original model. The graded form earns its place differently, which the next picture is about.

PGO-SGB in two pictures

Think of each routing decision as a ball. It starts among the hot experts, down where the VRAM is, and the router's preference gives it a push towards the expert it wants. Below is one layer seen from the side, with its experts sorted from hot (left, deep) to cold (right, higher). With a hard mask the cold experts sit behind a wall. With PGO-SGB each cold expert is raised by −max(cap, ln(b / b_edge)): a gentle rise just past the hot set that levels off at the cap. Both floors get the same seeded balls with the same pushes.

What the picture is meant to show: with a hard edge you must guess the right hot-set size, and every token on the wrong side of it is rerouted. With a graded edge, an expert just outside the hot set is barely penalised, so a wrong guess costs less, and the cap becomes one dial for "how much quality do I trade for cache hits".

The method in five lines

For each MoE layer l and expert e, the profile gives a brightness b: its phase-balanced share of router mass over my agent sessions. Experts in the hot set (the ones the cache should hold) get no penalty. Everyone else gets:

bias[l][e] = max(cap, ln(b[l][e] / b_edge[l]))
pick top-k by logit[e] + bias[l][e]
weights = router's own softmax over the picked k

• Graded: b_edge is the brightness of the weakest hot expert, so the penalty starts near 0 at the edge and reaches the cap in the deep tail.
• Selection only (on Strata): the bias changes which experts run, not how much each one weighs. The weights are still the router's softmax over the unbiased logits. I measured both variants, see below.
• Everywhere the router runs: prompt processing, per-token decode and the speculative verify window, so a prompt is routed the same way as the answer. The MTP drafter stays unbiased; the verify window decides.
• Static: the table is read once at startup (48 × 512 floats, about 98 KB per GPU). It does not drift with what happens to be cached at the moment.
• Opt-in: with the environment variable unset, nothing changes, bit for bit.

The mechanism itself is not mine. Skliar et al., "Mixture of Cache-Conditional Experts" (TMLR 2025), bias routing towards cached experts in the same spirit; I came to it from the cuts and found the paper afterwards. My part is narrower: a static profile from real agent sessions as the hot set, the graded edge, and the test of whether the agent still gets its tasks done.

Qwen on Strata: the bias as a quality dial

Strata is a local inference engine with its own VRAM expert tier and multi-token prediction. I added PGO-SGB to it in my fork, oisee/Strata (branch cnm/pgo-sgb, opt-in; proposed upstream in issue #4812, not merged). Model: ISTA-DASLab's Qwen3.8-Flash-Next GSQ-RCO IQ3_S, all 512 experts, PB3-192 as the starting residency, context 131,072.

First surprise: on two 16 GB cards Strata's cache already hits 83% of expert lookups without any bias. So the bias did not buy speed here. What it buys is control over how far the model drifts from itself.

Russian is the canary again: at cap −2 the Russian UI text drifts by KL 1.028, twice the pooled number. The cold tail is where the other languages live. I picked cap −0.5 as the default with a rule written down before the run: replay the first 46 requests of a recorded agent session (context 15.8K → 76.3K tokens) three times per arm in a Latin square, and keep −0.5 unless it is more than 5% slower than −1. Its median wall time was 15% lower (503.7 s vs 592.6 s), but that mostly means it wrote fewer tokens; at fixed work the two were within 0.6% (587.0 s vs 583.4 s). So −0.5 stayed. Its KL is about two thirds of −1's.

Two more comparisons settled the shape of the method:

Selection-only was closer than damped weights on every text (pooled +41% for damped). The prior row above is at its default strength 1.5, a much stronger push, so it is not a fair fight. On current upstream Strata, one GPU, at equal KL the prior wins on speed: prior 0.5 has the same KL as cap −0.5 (0.075 vs 0.086) and decodes +26% faster, against +2% for the table. So PGO-SGB is not a speed feature on Strata; what it offers is a fixed per-expert selection from a workload profile that does not follow the adaptive tier. These KL numbers come from a forced-window read; a self-check showed that it differs from the decode distribution (same top-1 95.3%), so the arms compare with each other but are not the decode-time KL.

The residency-only row is not zero either. My reading, not a measured cause: Strata's tail skip drops some missed experts, and which experts miss depends on what is resident. Turning tail skip off costs 8.3% decode in the replay.

Agent tasks

Same bench as last time: real ABAP repository and smoke tasks, hidden tests, the agent only gets osg-abap lint / check / test. On Strata with the bias, every run so far resolved: 10 of 10 at cap −1 (four smoke tasks plus apc-002 twice, and the long repository tasks L1a twice and L2b twice), 5 of 5 at cap −0.5, and 8 of 8 in an interleaved A/B of −0.5 against −1. That is 23 runs on 6 distinct tasks.

So about 44% less wall time than my llama.cpp setup on the same smoke tasks. That is mostly Strata, not the bias. At the default cap −0.5 the same four tasks took 3,448 s in total (N = 1, longer agent paths): slower than −1, still about 27% under llama.cpp. In the A/B, −0.5 decoded at a median 63.9 tokens/s against 65.8 for −1 and wrote fewer output tokens in total (81,253 vs 99,925), but N = 2 per task cannot resolve a 10% effect, and single runs swung widely: one −0.5 run took 57 calls, one −1 run 67.

The same idea on GLM-5.3-Flash

GLM-5.3-Flash is the model I actually started this project with: about 320B total parameters, 18B active. On my box the unsloth UD-IQ3_XXS runs in Project Maya, an engine built on Strata (per its README) and rewritten for GLM, with a GLM fast path, a VRAM expert cache, a CPU expert lane and MTP. Out of the box (Maya v1.1.38, with the unsloth quant rather than Maya's own) it decoded 11.3 tokens/s, with the VRAM expert cache hitting 57–58%. llama.cpp on the same weights did 5.78 (not a strict A/B).

Maya needed no engine change for the bias: GLM's router already has a per-expert bias tensor, so I wrote the PGO-SGB table into a copy of the weight pack. But GLM's router is a sigmoid, and the bias is added after the sigmoid, in probability space. The existing bias values within a layer have a standard deviation of only 0.05–0.10, so a cap of −0.5 or −1 acts almost like a hard mask on about 80% of the experts. At cap −1 the model told me that "the 2nd prime is 4". =) So the caps here are much smaller, and I swept them.

Decode and hit rate rise smoothly with the cap. Wall time on a real task does not: cap −0.5 solved the same smoke task once in 1,092 s with 25 calls and once in 4,130 s with 81 calls, rewriting its own test class 16 times. Wall time follows the agent's calls and thinking, not decode speed. So I repeated the three candidates on the same task, up to three times each:

All 13 runs resolved. By the no-overlap rule I wrote down beforehand, the arms are not separated at N = 3; cap −0.25 leads on both medians, so it became the default. Likely best, not proven.

Then the whole smoke set at cap −0.25: 11 of 11 resolved, decode 18.6–20.3 tokens/s and a hit rate of 84–85% on every task. The twelfth task, osg-l0-body-003, is excluded by the rule from last time: no model has solved it, including the full Qwen. The median wall time was 960 s per task; full Qwen IQ3_S in llama.cpp took a median 554 s on the same eleven. Not a strict A/B: different engines, days and effort settings.

Is the GLM code any good?

Pass/fail does not separate the models, so I compared the code. Automatic metrics on the 11 solved tasks: the production fixes look alike (one method touched, median 22 statements, about 70–75 diff lines). The tests do not: GLM wrote fewer asserts (median 3 vs 8), ran the tests more often (10 vs 2), and made more calls (31 vs 24). Caveat: GLM ran at effort medium, Qwen at xhigh.

Three tasks went to two blind LLM judges, with positions swapped between judges. The production fix was correct in all six solutions. After unblinding both judges agreed: GLM won one task (an exact two-line restore), Qwen won two, because GLM deleted its tests in one and loosened its assertions until they passed in the other. Sounds familiar. =)

And then a two-line engine patch

With the bias in place, where does a GLM token's time go? Maya splits the layers across the two GPUs ("parts"). I profiled one part pass:

The biggest single item is waiting for the CPU lane: the experts that live in RAM, computed by the CPU. Maya already has a lossy option for that, DEFER_CPU: add a layer's CPU experts after the next layer's attention instead of before it, so the CPU overlaps with GPU work. But upstream only applies it to the first part of a split, and on my box the second GPU sets the pace. I read the code for a reason behind that restriction and did not find one: each part has its own CPU lane state, and nothing deferred crosses a GPU hop. So the patch is two code lines: DEFER_CPU=2 means every part.

• Decode: 18.6 → 21.3 tokens/s (+14.5%, median of 6 from 2 interleaved starts). The second GPU's CPU-lane wait went from 11.6 to 5.0 ms per token.
• With speculation cancel on top (discard a rejected draft pass early, lossless by construction): 22.2 tokens/s. That extra +4% is inside the noise; it is harmless, and I keep it.
• It is lossy, and I measured how much: teacher-forced KL on 999 ABAP positions was 0.131 against the unpatched engine, where two runs of the unpatched engine differ by 0.045 from each other.
• Four smoke tasks, two runs each, on the combination: 8 of 8 resolved, and median calls stayed within 1.5× of cap −0.25 alone.

The patch is offered upstream as Project Maya PR #722 (open, not merged), with a speed report in issue #723. It is opt-in and changes nothing with the variable unset.

Then Maya's own knob beat ours

It turns out Maya already has its own knob for this: STRATA_GLM_ROUTE_AFFINITY, from Maya's PR #252. It adds a small λ to the selection score of every expert that is in VRAM right now. It follows the live cache, and it acts in decode only. That is the same idea as the cache-conditional routing paper, minus the profile. So of course I put the two side by side, on the plain unsloth pack, Maya v1.1.42, one session:

At the same KL as our cap −0.125 (0.072), the affinity curve sits at about 23.4 tokens/s (interpolated between its measured points), against our 18.65. That is about 25% more decode for the same drift. Our profile on top of affinity adds nothing (22.25 vs 22.15). The rule I wrote down before the run said "propose the table for decode only if it is at least 10% faster at equal KL". It was not, so I am not proposing it.

The table keeps one honest plus: it also acts in the prompt path, so prefill is 24–94% faster at 2,048 tokens and 11–44% faster at 8,192. Affinity does not touch prefill at all.

Strata told the same story with Qwen: on one GPU (Strata's own speed harness, median of chat medians), its built-in residency prior at 0.5 decodes 26% faster than no bias, while cap −0.5 gives 2%, at about the same KL. Two engines, one conclusion: for speed, push towards the live cache, not towards a static profile. The profile gives a fixed, predictable selection that does not drift with the cache, which is worth something, but it is not the fast lane.

Did the bias work go to waste? Not at all. It got GLM from 11.3 to 18.9 first, it is how I learned where the time goes, and it led straight to the DEFER_CPU patch, which is the part Maya can actually use. Finding out that someone else's knob is better is a perfectly good result. It also saves me maintaining one. =)

And the patch stacks nicely on top of the native knob. Same session, plain pack: affinity λ 0.25 alone 24.8 tokens/s, with DEFER_CPU=2 29.7 (+19.8%), 2.28× the 13.0 we started this session with. KL goes from 0.099 (measured in the comparison run) to 0.117 against a run-to-run floor of 0.014, so it is lossy, and I say so. Our pre-registered rule (at least +5% and at most +0.03 KL) passed at λ 0.25; at λ 0.125 the speedup was similar (22.3 → 26.35, +18%) but KL grew by 0.039, so that one did not count. The profiler shows why: the CPU-lane wait fell from about 11 ms to 3.6 ms per token on each card, and the two GPUs are now almost balanced (40.7 vs 42.1 ms per token). Speculation cancel on top gives 29.9, the same within noise.

What didn't work

Some of my favourite ideas this week died on contact with a stopwatch. Here they are, so nobody has to try them again on the same hardware.

• Expert parallelism over the LAN. The second machine (i5) has 122 GB free RAM and the GGUF on its own NVMe. But Maya's CPU lane, or any remote tier, sits on the critical path of every one of the 42 MoE layers. Measured round trip for one hidden state over 2.5 GbE: median 167 µs, so 42 of them is 6.6 ms per token, with no batching across MTP drafts. Modelled best case: 1.15× at median latency, against a 1.4× bar. No prototype.
• A helper GPU for experts. Lossless, and 43% slower (11.0 vs 19.2 tokens/s): with the trunk on GPU 0, only 785 expert slots were left and the hit rate fell to 46%.
• Skipping missed experts. It cut the CPU lane from 33.1 to 27.5 ms per token and did not speed up decode, because the second GPU, not the CPU lane, sets the pace. That is how I found the DEFER_CPU lever above.
• Two parallel streams: 1.15× total throughput, each stream much slower. Not worth it for one agent.
• The "horse": a two-bank tier for Qwen in Strata, with a 2-bit Q2_0 copy of every expert for the CPU's misses. Its gate was KL ≤ 0.02 against the same engine without it. It measured 0.156, and its +19% decode was invalid anyway (the Q2_0 bank was 9–19% in page cache, and the two arms decoded different token streams). Dropped by its own rule.
• Our static table as a decode accelerator. On Maya, the native live-cache affinity reaches the same KL with about 25% more decode, and our profile adds nothing on top of it. On Strata the built-in prior does the same at equal KL (+26% against +2%). The table stays useful for predictability and prefill, not for decode speed.

Not proven, and what's next

• Same pass rate on fresh tasks. Still the main debt. Smoke tasks were partly in the profile corpus; they are an engineering check, not a generalisation test. I need held-out tasks, the same harness and budget for parent and candidate, and a loss bar decided in advance.
• N. Almost every task run here is N = 1 or 2. Wall time varies up to 4× between runs of the same task.
• KL on GLM. Maya's GLM path exposes no log-probabilities for the bias sweep; the teacher-forced scorer came later and was used for the patch and the affinity comparison.
• Effort. GLM ran at effort medium, Qwen at xhigh. The code comparison carries that confound.
• 29.7 on agent tasks. The affinity numbers are --bench speed and teacher-forced KL; I have not yet run the smoke tasks on affinity + DEFER_CPU=2, so the 11 of 11 belongs to the earlier bias setup.
• Next: the paired held-out study; the best live-cache setting per engine and per machine; whether a profile is still useful as a seed for the live cache, or for prefill; and phase 2, the full 744B GLM-5.3 with the same pipeline.

The nice part: the pipeline did carry over. Profile from agent sessions, one table, one dial, the same bench. A new engine and a new router type cost a sweep and a few days, not a rewrite. And the bench was honest enough to tell me when someone else's knob was better. Nudge, don't cut, and nudge towards the live cache. =)

Thanks to Strata (Niko1221), Project Maya (mw00, co-maintained by needmorevram), unsloth for the GLM GGUFs, ISTA-DASLab for the Qwen IQ3_S, Skliar et al. for the cache-conditional routing paper, and the ABAP tooling community behind the bench.

💬 Comments and questions: find me on LinkedIn →