I only wanted to run a little MoE

The harness is part of the model I actually use

The motivation was never just a prettier tokens-per-second number. In the earlier work behind that October post, changing the harness around the same GLM-5.3 moved the result from 4/6 solved tasks to 6/6. That was what got me interested; it was before these pruning experiments.

An agent needs to read a repository, choose a tool, edit something, interpret the error, test again, and eventually stop. A model can produce a very reasonable description of all this without doing it. I would like the patch, please. The description can come afterwards. =)

So I followed real agentic ABAP work. The project has both repository repair tasks and native ABAP tasks checked through an ABAP transpiler. They have different definitions and different denominators. I keep them separate.

Cloud machines helped collect full-model trajectories and run experiments. The local serving measurements belong to the home hardware. Mixing those into one impressive speed chart would be easy. It would also answer a question nobody asked.

Looking inside the expert traffic

My main backbone here is Qwen3.8-Flash-Next: 48 main layers, with 512 routed experts per layer and top-10 routing. Each token uses a small selected subset, but the weights for the other experts still have to live somewhere if they remain available.

I started by counting expert selections. Then I measured the mixture coefficients actually applied after routing normalization and scaling. An expert selected often with a tiny coefficient is different from one carrying much of the mixture.

I call the resulting views tomography. Replay a recorded conversation through the target model, collect cumulative snapshots, and subtract them to see activity along the history. Separate reasoning, ordinary text, tool calls, edit arguments and tool results. Exclude the repeated system/tool prefix from the conversation signal.

There is a useful trick here: a recorded GLM conversation can be replayed through Qwen. The observed experts are still Qwen's experts. I am borrowing the text of the work, not matching expert identities between models.

There is also a trap. Replaying a final recorded history is not automatically identical to every live request that produced it. Templates, omitted reasoning, truncation and prefill/decode numerics can matter. Tomography gives me a selection signal; it does not certify that I have captured every important computation.

And “cold in this corpus” means exactly that. It does not mean useless. This distinction will become rather personal when I ask the model to speak Russian.

Making the mask less rectangular

A uniform cut keeps the same number of experts in every layer. Convenient to store. Less obviously the best use of a memory budget.

The “figured” masks allocate a total budget using router shares normalized within each layer. Some layers keep more experts; others keep fewer. Each layer still needs enough survivors for its routing operation. That minimum makes the shape legal; it does not prove the layer has enough capacity.

Then there is PB3: explore, edit, verify. I label assistant turns by their tools, attach tool results to those turns, normalize within session and phase, average sessions, and give the phases equal weight. Long stretches of exploration should not automatically drown out a shorter verification phase.

The labels are approximate. Mixed snapshots share their mass proportionally. The original Qwen implementation uses the pooled signal for the final layer, where the phase signal is missing. These details belong in the experiment description, even if they make the acronym less magical.

PB3 is still a static mask. It is not switching expert banks whenever the agent reaches for a test command. That is a different experiment.

Also, none of this makes nonuniform pruning or phase balancing my invention. The novelty review found substantial prior work, including foreign-agent calibration and per-layer expert counts. I am using these ideas in a particular workload and implementation, then measuring what breaks.

The earlier K192p-implant16 build is a separate uniform path: protected selection and expert replacements, with 192 experts per layer. “implant16” does not mean extra slots added on top. Naming things remains a difficult benchmark. 😅

The weights were right. The exported model was different.

This is my favourite technical part.

A runtime mask leaves the original model physically present and prevents selected experts from being chosen. A physical cut removes the unwanted expert tensors. If the retained weights and their mapping are correct, I would like those implementations of the same mask to agree.

For the real A224 export, KL(mask || cut) was 0.146469. The byte audit found the retained weights intact. Yet the first discrepancy appeared in the first layer's BF16 router when its matrix width changed. Expert selections diverged from the following layer.

Changing the shape of a matrix operation can change its finite-precision result. At a top-k boundary, a small score difference can change which expert runs. After that, later layers receive different inputs. Correctly copied weights are necessary, but they are not the whole execution contract.

The repair preserves the original router width and source expert IDs. Missing experts are masked before selection; selected original IDs map into the compact expert tensors. The expensive expert banks shrink, while the router keeps its original execution shape.

After the repair, the reported KL(mask || cut) was 0.006619. Roughly 0.146 to 0.007, with a residual discrepancy. I cannot call that exact equivalence, and it says nothing by itself about agreement with the unpruned parent.

The evidence points to a shape-sensitive routing problem in this export path. A complete causal study still needs frozen router inputs, score margins, route-flip measurements, forced-route controls and a high-precision reference. I am not declaring a universal upstream llama.cpp bug from my fork.

There is a much more interesting short technical report here than “I discovered that layers can have different sizes.”

MoREL, and what fits

MoREL means Mixture of Ragged Experts per Layer. The GGUF stores an expert-count array; the loader and graph use each layer's count; the cutter physically slices the expert tensors without padding every layer to the widest one. The uniform path remains available.

This lives in my llama.cpp fork, branch cnm/varexp. It is implementation work for the supported paths, not a promise that every MoE architecture now loads unchanged.

Here is the useful, slightly inconvenient results table. Quality and speed often come from different runs. The placement notes are part of the result.

The K192p startup readings describe the original serving configuration, not the ub128 speed run. PB3-208 uses q5_1/q4_0 target KV; PB3-304-q2 uses q8_0/q4_0. Prefill is a separate prompt-processing test. I cannot use those different setups to declare a winner.

The synthetic speed protocol uses three warm samples after warmup, a 2,048-token prompt followed by 512 generated tokens for decode, and 8,192 prompt tokens for prefill. Speculative means MTP plus ngram drafting. In plain decode, the selected PB3-208 configuration measured 36.79 tokens/s and PB3-304-q2 measured 42.26.

File GB are decimal; GPU MiB are device readings. Qwen's roughly 28.8 GB PLE table stays in host RAM. My i7 machine also has 128 GB RAM. A large GGUF beside two small cards is therefore not evidence that the entire model lives in VRAM.

The selected PB3 serving configurations allocate 194,560 tokens of context. That is a fit result, not an evaluation of answer quality throughout that length. Lower KV precision has a cost too: the PB3-208 q5_1/q4_0 check measured mean KL 0.188096 against q8_0/q5_1 on its held-out text.

The completed PB3-304-q2 smoke result combines the earlier subset with the remaining tasks from the same pinned package. Its unresolved task was osg-l0-body-003. It is encouraging that this model works locally. But the source audit also found SMOKE tasks among the A-mask calibration trajectories. This suite is an engineering check, not an independent generalization test. Matching smoke fractions does not establish preserved pass rate.

Faster tokens, longer conversations

What happens after the model fits is more interesting than whether it fits.

I reconstructed the first recorded tree that passes the hidden tests, rather than trusting the agent saying “tests pass.” That gives a first-green boundary and lets me count generation calls and tokens before it. It is a recorded snapshot boundary, not the exact instant of an edit.

On a fixed set of three shared solved smoke tasks, the full Qwen median was 4 calls and 3,627 generated tokens to first green. PB3-208 needed 10 calls and 6,023 tokens; PB3-304-q2 needed 11 and 6,561.

Then they kept going. The median calls at the end were 20 for full Qwen, 58 for PB3-208, and 63 for PB3-304-q2. Other selected cuts were around 50 calls, though the complete table spans a wider range. Those are calls, not elapsed seconds, and the columns are separately calculated medians.

These were selected attempts on different hardware, and the agents took different paths. I cannot blame pruning for the whole difference. They also all solved these three tasks; this comparison tells me nothing about how often a model fails. But it shows why I need both “first green” and “finished successfully.” Faster decode can coexist with much more work.

One local K192p L3a run reached its first passing snapshot at 846 seconds and ended resolved at 3,247 seconds of agent time, excluding setup and final grading. It did not stay green throughout. The replay grades early trees with the final dependencies after minzc/go.mod changed, so that first-green time is conditional on the replay environment. A single clean-looking duration would hide most of that story.

Apparently, Russian was a cold path

I tried chatting with the ABAP-selected cuts. They could appear to understand Russian and then answer in English or mixed fragments. A useful coding specialist is not necessarily the conversational model I started with.

The follow-up probe used 20 fixed prompts per model, with a maximum of 400 tokens including reasoning. In pooled final prose, PB3-208 had 2.51% Cyrillic letters. The uncut DSLab IQ2_XS model had 98.19%.

So the funny version is: I pruned for ABAP and the model forgot Russian; the uncut two-bit model remembered it. =)))

The careful version is less tidy. PB3-208 produced 15 empty finals; the uncut lower-bit model produced 6. A short budget can disappear into reasoning. The full higher-precision profile also had timeouts, and the serving configurations differed. Cyrillic share is a script statistic, not a fluency or correctness score.

The probe supports a visible difference in these outputs. It does not identify “the Russian experts,” prove that Russian knowledge was erased, or isolate pruning from every other serving choice. The ABAP-only corpus leaving other capabilities poorly represented remains a hypothesis worth testing.

Still, I would prefer not to compile a different personality every time I change language.

Keep the experts. Change the profile.

That leads to the next idea: retain all the weights, but use task profiles to choose placement and routing preferences. Coding, Russian conversation, diagrams, finishing the task instead of endlessly discussing it. That last profile has my attention.

A placement-only profile keeps the routing unchanged. A soft profile adds a negative bias to cold router logits while leaving those experts reachable. A hard profile excludes them from selection. Those interventions need separate comparisons.

There is already a bounded result for soft bias. With the same PB3-192 hot set on the uncut IQ3_S model, bias −1 gave mean KL 0.648863 against the unmasked parent, versus 0.906512 for hard masking. Cold experts still accounted for 8.2172% of selections. The mean gives each held-out text equal weight.

That is closer distributional agreement on the tested texts. It is not a measured speedup, a Russian-language recovery result, or a live-agent success score. Keeping cold experts available means paying for their storage and sometimes their execution.

Changing routing mid-conversation also leaves a cache computed under the old routing. Recompute, or explicitly treat the switch as a model change. “Dynamic” does not make old state disappear.

From cutting experts to nudging them

NEW · OCT 27 This is how the cuts led to the soft mask. The cuts worked, in the narrow sense: they fit on two 16 GB cards, ran fast, and solved tasks. But every cut is a bet that the cold experts will never be needed. Then the cuts started thinking in English. So the question changed. Instead of asking which experts I can delete, I asked how I can make the model prefer the experts that are already on the GPU while keeping the rest reachable.

Three pieces came together on October 27:

• Upstream llama.cpp got an expert cache. --moe-cache-mib (PR #29887) keeps all experts in host RAM and caches recently used ones in VRAM. It is an SLRU, not a plain LRU: new experts enter a probation half, and only experts hit again are promoted to the protected half. A miss copies the expert over PCIe.
• The phase-balanced profile picked the hot set. The same PB3-192 brightness that chose the MoREL survivors now chooses which experts the router should prefer. Explore, edit and verify have equal weight, so the experts for editing and testing are in the hot set even though exploration produces most of the tokens.
• A soft router bias does the nudging. A constant negative number is added to the router logits of the cold experts before top-10 selection. Nothing is removed. A cold expert can still win if the router wants it badly enough; it just has to win by a larger margin.

The point is the interaction. A cache only helps if the same experts keep coming back. The bias makes the router come back to the same experts, so the cache hits more and PCIe copies less. The experts the bias favours are the ones the agent uses for real work, because the profile came from real agent sessions.

The mechanism itself is not mine. Skliar et al., “Mixture of Cache-Conditional Experts” (TMLR 2025), bias routing towards cached experts in the same way. I reinvented it and found the paper afterwards. My part is narrower: a static task profile as the hot set, and an evaluation on closed-loop agent tasks rather than perplexity.

What the nudge bought on one machine

All of this runs on the i7 box (two RTX 4060 Ti 16 GB, PCIe 3.0 x8, 128 GB RAM) with the full Qwen3.8-Flash-Next IQ3_S, all 512 experts. Every number is decode tokens/s as a median of 3 samples (2,048-token prompt, 512 generated), with prefill measured on 8,192 tokens, one run at a time. The figure walks through the levers in the order I found them.

Three things surprised me.

The bias made the draft head better. With no bias, MTP accepted 65% of its drafts. With bias −0.5 to −0.75 it accepted about 90%. Decode went from 31.7 to 48.8 tokens/s, and most of that came from acceptance, not from the cache. I can see the effect but have not explained it. Acceptance also depends on the generated text: one arm, −0.66, happened to produce a text the head guessed badly (69% accepted, 39.2 tokens/s). So these are single-prompt numbers, and the next step is a multi-prompt replay.

n-gram drafting hurts once the experts are not all resident. Same cache, same bias: MTP + n-gram gave 29.0 tokens/s, MTP alone gave 47.9. See the explainer below for my reading of why.

Microbatch and pinned memory are cheap wins for prefill. -ub 512 instead of 256 raised prefill from 111 to 185 tokens/s, and decode barely moved. Pinned host memory (--load-mode none --no-repack) added a little more: 51.9 decode / 190 prefill. Huge pages gave nothing measurable.

Did it forget how to act, or Russian, or diagrams?

This is the reason for the soft mask, so here is what is measured and what is not.

Why should it remember Russian? Nothing was removed. When a Russian token needs a cold expert, the router can still pick it, and the cache fetches it. That is slower, not forgotten. The cuts did not have that option.

The quiet hero: a test bench that runs by itself

NEW · OCT 27 None of the numbers above would exist without the testing setup built on the OSG ABAP tooling. That setup is the reason this could be iterated rather than argued about.

• Real tasks, real tests, no SAP system. Repository tasks are cut from merged pull requests (PR-6, L1a to L3b). Native ABAP tasks (SMOKE-12, MEDIUM-69) are holes cut into open-abap-core, abapGit and other projects. They are graded by that code’s own ABAP Unit tests, run through a transpiler. The agent gets osg-abap lint / check / test as its only ABAP tool, exactly what a developer would use.
• Nobody fails just for forgetting syntax. The brief tells every agent the same thing: build and test with osg-abap check and osg-abap test, and before finishing, run osg-abap check and fix what it reports. The check activates the code and returns the syntax and activation errors with their places, for example “line 1: Expected ENDMETHOD” and “line 335: Statement does not exist”. So a model that half-remembers ABAP syntax gets told where it went wrong, the way a developer in the ABAP editor would, and the test measures whether it can solve the task, not whether it can recite the grammar. That is deliberate, and it is the difference from a one-shot benchmark. A one-shot test asks for the right code in one go and scores the text. An agentic benchmark gives the model tools and expects it to act with them: run the check, read the error, look at the code, change it, run the tests again. I want agents that get to a working answer. Getting the syntax right from memory on the first try is not the goal.
• Hidden grading. The agent works alone in a sandbox with an egress allowlist and never sees the held-out tests. The grader runs later in a separate offline sandbox and takes only the target code from the agent’s tree. Leak and contamination checks are recorded with every result.
• Unattended. One command runs a task list against any OpenAI-compatible endpoint: the cloud, a pod or my i7. A time scale stretches the limits for slow home hardware. The SMOKE-12 rest and PR-6 behind this article are running through it right now, with nobody watching.
• It records everything. Every model request goes to traffic.jsonl, and every workspace snapshot goes to git. That makes “first green” reconstructable after the fact, instead of trusting the agent when it says “tests pass”.

That last point closes the loop that this whole article runs on. The recorded agent sessions are the corpus for the expert tomography. The tomography gives the PB3 profile. The profile gives the MoREL cut, the hot set and the bias. Then the same bench grades the result. Bench → traffic → tomography → profile → mask or bias → bench.

Scale, counted on the bench directories of one machine (i5): 521 graded agent runs, 382 resolved and 139 failed, across models, harnesses, cuts and serving configurations. The failures matter as much as the passes. The bench is how I know the cuts loop (58 calls against the full model’s 20), and that osg-l0-body-003 defeats every model, including the full one. Proper credits for the OSG tooling and its authors are coming.

The 8 GB lesson: the hot set has to fit the cache

On i5 (RTX 2070, 8 GB, IQ2_XS, about 3.5 GB of expert cache), the same PB3-192 bias barely helps: 17.1 tokens/s with no bias, 17.5 / 17.6 / 18.2 at −0.5 / −0.75 / −1, with the hit rate going from 61% to 70%. The reason is obvious once written down: PB3-192 is about 13 GB of experts at that quantization, almost four times the cache. Favouring 13 GB of experts does not help a 3.5 GB cache much.

So the hot set should be sized to the machine. The PB3 ladder now has masks from 32 to 320 experts per layer (one generator, same signal), and i5 is measuring PB3-64, which is sized for its cache. That is also where the graded bias from the figure above should matter most. With a hard edge you must guess the right N. With a graded edge, an expert just outside the hot set is barely penalised, so a wrong guess costs less. Results pending.

The little distributed-experts dream

Naturally, the next thought was: what about the other computers? i7, i5, i3. Those are machine names; the one called i5 is actually a Ryzen host. A small LAN full of experts sounds like something I would enjoy far too much.

The proposed design keeps expert weights stationary. The main node owns routing and model state; selected activations travel to workers; expert outputs return and join before the next layer. Moving large expert weights across the network for every token would defeat the idea.

This is still a design with simulations, not a demonstrated distributed speedup. Extra machines add transport, synchronization and arithmetic differences. The design even contains scenarios where local execution wins. More computers are not an optimization proof. Annoying, but fair.

The local tier work has made real progress since that design: a CPU batching defect was repaired, and the real K192p all-CPU-expert endpoint matched the native CPU reference. The tiny Qwen placement matrix passes 6/24 head cells by default—all six at the all-GPU endpoint—and 15/24 with the experimental arithmetic path, including those six endpoint passes. CPU-containing placements still fail qualification. No speed benchmark was run in that repair report.

NEW · OCT 27 A later bug, in the CUDA Stream-K reduction order, was found and fixed, and the two-tier engine then ran. On the same i7 without speculative drafts it decoded 22.3 tokens/s, while upstream’s expert cache with the soft bias decoded 33.5. Profiling put about 14 ms per token into synchronisations and graph splits. I parked my engine and moved the main line to the upstream cache. The distributed version waits behind that.

I need to finish that numerical work before claiming the LAN version is fast. Otherwise I may only have distributed the mistake.

What I can reproduce, and what I still owe

The most concrete starting point is the PB3-208 selected-KV run in REPORT9 (local-only). It names the serving script on i7 and the comparison driver in the report checkout. With those original files and their environment in place, the script entry points are:

# On i7: selected PB3-208 serving setup recorded in REPORT9.
bash ~/src/llama-varexp/artifacts/round9/q5_1-q4_0-ub384-spec-start.sh
# Separately, from the REPORT9 tools checkout: held-out KV comparison.
bash validation/kl-round9.sh

These are report-local paths, not verified public downloads; I have not rerun them for this article. The serving setup uses PB3-208-router.gguf, q5_1/q4_0 target KV, ub384 and context 194,560. The comparator uses the same weights against q8_0/q5_1 KV on texts/heldout-own-body006.txt; its reported mean KL is 0.188096. This entry point covers the KV comparison, not the A224 router-compaction experiment.

REPORT9 also names validation/round9.py; REPORT12 (local-only) names validation/run-round12.py for the later smoke work. Both record task package 573676f648c31ac57b972589168439c31c82e4ec. That pins the tasks, not the engine or model. The reports do not supply the PB3-208 engine/tools revision pins, model hash or exact source-ID keep-map identity needed to turn these entry points into a complete recipe. I still owe those, plus the tokenizer/template, build details and full serving/comparator arguments.

The main MoREL code uses cnm/varexp; soft bias uses cnm/varexp-bias; the tier repair uses cnm/varexp-tiers. Each needs its own pinned recipe. Exact recipes, pins, model hashes and keep maps are still to be published.

Then check the export before scoring the agent: tensor bytes and mappings, runtime mask versus physical cut, repeated-run numerical baseline, and behavior at tool boundaries. Keep the original unpruned comparison separate from mask-to-cut parity. Run the actual maximum-shape request; a successful health check is insufficient.

Next I need the same fresh tasks, the same harness and the same budget for the parent and each candidate—and to decide in advance how much quality I am willing to lose. I need to save the traffic and tree snapshots, and keep the failures and loops in the results. Interrupted runs need their own label. Otherwise I will be back to comparing the runs I happen to like.

What I still owe is that paired quality study, a stronger compaction reproducer, equal-memory comparisons against lower-bit uncut and smaller models, and measured placement/profile-switch costs. PB3 also needs stronger selection baselines and shuffled-phase controls. The work so far is an engineering report, with questions attached.

Thanks to the Qwen, llama.cpp and ISTA-DASLab work this builds on, and to the ABAP tooling community.

TL;DR: PB3-208, PB3-304-q2, K192p-implant16, and the fork.

Next I want to keep the cold experts within reach—and see whether I can get my Russian answers back. =)

💬 Comments are here: discuss on LinkedIn →