LISTEN TO THIS ARTICLE
A mixture-of-experts model can use a small share of its weights to process each token while still needing a large deployment. A dense model can fit an existing serving stack, yet fail the task you need it to perform. Architecture helps explain resource use; it does not settle quality, safety or price.
Use this guide when comparing specific open-weight models for an agent or application. If you buy an API, start with measured task quality and the provider's bill. You may have little control over the architecture underneath it.
What sparse activation actually means
In a sparse MoE layer, a router selects a subset of feed-forward blocks to process a token. Other parts of the network remain shared. The active parameter count describes the weights used for that token; the total parameter count describes the larger collection of weights. The Mixtral paper explains this distinction and how expert parallelism distributes work across devices.
An expert is a learned block of parameters, not a named specialist you can ask to handle legal analysis or code review. Mixtral's routing analysis did not find a simple correspondence between experts and subject areas. Do not turn a routing diagram into a claim about domain competence.
Choose a model using completed tasks, not the architecture printed on its model card.
Neither architecture establishes suitability for a high-stakes use case. Evaluate the particular model, configuration, evidence requirements and human review process.
Separate compute from resident memory
Sparse activation can reduce feed-forward computation per token compared with evaluating all the experts. It does not make inactive weights disappear.
For a conventional fully resident deployment, budget for the entire model's weights across the serving devices, plus runtime overhead and request state. Quantisation changes weight storage. Weight offloading can move some weights to host memory or another storage tier, but transfer costs and the serving implementation then become part of the comparison. Every expert need not fit on a single GPU.
Record peak allocation after loading the model and while serving representative requests. A parameter count multiplied by nominal storage precision is a planning estimate, not a measured memory requirement. Include temporary buffers, communication workspaces and reserved memory.
The useful question is whether the complete configuration fits your hardware with room for requests. A model that loads successfully can still run out of memory under traffic.
Context and concurrency can change the result
The attention key-value cache stores state from previously processed tokens. In ordinary full-attention decoding, longer sequences and more simultaneous requests increase the state to retain. This is separate from expert weight storage.
Cache behaviour depends on the model and runtime. Sliding-window layers can bound growth; static caches reserve capacity differently from dynamic caches; quantised or offloaded caches trade memory against other costs. Hugging Face's cache guide describes these differences.
Run each candidate at short, typical and long input lengths, then increase concurrent requests. Keep the output allowance representative too: generated tokens extend the sequence. Record peak memory, time to first token, completion latency and accepted tasks per unit of time. Include the busiest workload you expect.
Fast token generation can still mean slow answers when requests queue. Conversely, low cost per generated token may hide the cost of hardware waiting for work. Measure at the utilisation you can sustain.
A comparison matrix you can fill in
| Decision | Evidence to collect | What rules out a candidate |
|---|---|---|
| Task quality | Held-out examples with identical acceptance criteria | Critical errors or missing evidence |
| Tool use | Valid arguments, permissions and recovery from failures | Unsafe actions or malformed calls |
| Memory | Peak allocation at target context and concurrency | Exhaustion or unacceptable offloading delay |
| User experience | Completion latency including queues and retries | Exceeding the application's latency budget |
| Economics | Total cost divided by accepted tasks | Exceeding the sustainable budget |
| Maintenance | Runtime, adapter support, licence and upgrade procedure | A dependency the team cannot operate or use |
These are suggested evaluation criteria, not benchmark results. Set pass conditions before looking at which candidate wins.
For agents, include a failed tool call, an ambiguous request and a task that should stop for human input. Scoring only fluent final answers misses behaviour that determines usability.
Fine-tuning is another experiment
Do not assume dense fine-tuning updates every weight: adapter methods deliberately update a subset. Nor does MoE fine-tuning inevitably damage routing. Ask what the selected tooling supports and whether adaptation improves held-out tasks.
Keep training and evaluation examples separate. Check both the base and adapted models on behaviours you must preserve, such as tool formatting and refusal to invent missing evidence. Expert utilisation, if available, is a diagnostic rather than a substitute for task evaluation.
Account for adapter loading, quantisation compatibility and deployment memory. A training run that finishes is only the beginning of the serving decision.
Run a fair pilot
Use an identical task set, tool implementation and acceptance rubric. Record model revision, runtime, quantisation, prompt and sampling configuration. Use supported chat templates and document differences.
Keep raw outcomes as well as aggregate scores. Inspect why each candidate failed before deciding whether a prompt repair is justified. If the harness changes, rerun the comparison.
Leave room for request state when you budget memory for model weights.
Choose the least costly configuration meeting your quality, latency and operational requirements. Reopen the decision when contexts grow or more users arrive together. Continue with agent cost accounting and the evaluation checklist.