LISTEN TO THIS ARTICLE
Test-time compute means spending additional computation while answering a request. That might mean generating alternative solutions, revising a candidate or searching through intermediate steps. The useful question is whether the answer your system actually returns improves enough to justify the added cost and delay.
Start with a task you can score, an unchanged baseline and a spending limit. Keep the generator, selection method and final evaluator separate. A promising answer somewhere in the candidate pool is useful evidence about the generator; it is not yet a successful response to the user.
Separate finding an answer from choosing it
Large Language Monkeys distinguishes coverage from the ability to identify correct samples. Coverage asks whether any attempt solved a problem. In coding evaluations this corresponds to the pass@k idea: did a correct solution appear within the sample budget? Selected-answer accuracy asks whether the system chose a correct answer to return.
You can measure coverage offline using reference answers or held-out tests. Those checks may be unavailable to the deployed selector. Using the answer key to pick a candidate, then reporting that result as deployable accuracy, would hide the problem you need to solve.
The paper finds that selection methods can stop improving while coverage continues to rise in its studied maths tasks. That is a reason to evaluate the selector independently. It does not establish a universal sampling limit for every model or workload.
More candidates help only when your system can choose a better answer.
Choose the method around the task
Parallel sampling generates alternative candidates from the same request. A selector might rank them with a reward model, count equivalent answers or run executable checks. Keep candidate order and tie-breaking rules explicit. Several near-identical answers can share the same mistake.
Sequential revision feeds a candidate and feedback into another attempt. This fits tasks where feedback identifies something actionable, such as a failing test or a missing requirement. Record each revision: an apparently useful correction can also damage an earlier valid result.
Search over intermediate steps explores partial solutions and uses a scoring method to decide which branches receive more work. This creates another evaluation obligation: check the scoring method as well as the final answers.
Scaling LLM Test-Time Compute Optimally studies different allocations of inference effort and finds that effective choices depend on problem difficulty relative to the model. Its difficulty-estimation setup also has costs that are not fully included in the reported analysis. In your service, include routing and difficulty estimation in the bill.
There is no useful universal promise that a particular sample count captures most of the available gain. Choose candidate budgets from your latency and spending constraints, then measure the resulting curves.
Try a small selection failure yourself
The following Python example uses invented correctness labels and invented verifier scores. It calls no model and is not a benchmark. The candidate order is fixed, and the selector can see scores but cannot use the correctness labels.
def measure(tasks, budget):
if not tasks or budget < 1:
raise ValueError("Provide tasks and a positive budget")
if any(len(task) < budget for task in tasks):
raise ValueError("Every task needs the requested candidates")
covered = selected = 0
for task in tasks:
candidates = task[:budget]
covered += any(correct for correct, score in candidates)
chosen = max(candidates, key=lambda item: item[1])
selected += chosen[0]
return {"coverage": covered / len(tasks),
"selected_accuracy": selected / len(tasks)}
tasks = [
[(False, .4), (True, .8), (False, .9)],
[(True, .7), (False, .6), (False, .8)],
]
for budget in (1, 2, 3):
print(budget, measure(tasks, budget))
With the middle budget, the selector finds the correct candidate for each task. With the largest budget, higher-scored wrong answers displace those choices even though correct answers remain available. This deliberately constructed failure shows why the coverage curve cannot substitute for the selected-answer curve.
Use the inline example above. Change the scores, reverse tied candidates and remove a correct answer. Observe what changes before adapting the calculation to recorded model outputs. The example reports observed coverage on fixed candidates; it does not implement a statistical pass@k estimator.
For saved traces, calibration and test splits, and abstention policies, run the answer-selection lab. Start with its synthetic fixture, then substitute independently labelled examples from your own workload.
Build a verifier you can challenge
Use checks that correspond to the intended result. For a code patch, compilation is a different check from passing relevant tests or preserving behaviour outside the edited function. For an extraction task, a valid schema does not establish that the extracted values match the source.
Keep final evaluation independent of candidate selection. If a generator can see every acceptance test, passing those tests gives limited evidence about behaviour on unseen inputs. Retain held-out cases and inspect disagreements between selector scores and final labels.
Let’s Verify Step by Step compares feedback on final outcomes with feedback on intermediate reasoning steps in mathematical problem solving. It supports investigating verifier training and supervision. Its results do not establish that a process reward model will work equally well for your business documents or software changes.
Record false acceptance and false rejection examples. A verifier that confidently rewards a plausible wrong answer needs attention even when its average score looks healthy. Add those cases to a separate regression set without using the final test set to tune the selector.
Measure the answer you return, including the work you spent rejecting alternatives.
Run a comparison you can act on
Take a representative task sample from the work your system receives, with private information removed where necessary. Define the acceptance rule before inspecting the new results. Include routine requests, difficult cases and examples where the system should abstain.
Run the baseline and each candidate strategy on the same task set. Preserve model identifiers, prompts, candidate outputs, selection scores and independent labels. For stochastic generation, repeat the comparison sufficiently to understand whether a difference is stable; a single lucky run is weak evidence.
Record total generation, verification, tool and retry costs alongside end-to-end latency. Compare accepted outcomes per completed task, not token price alone. Use the cost-accounting guide to include failed attempts in that calculation.
Choose the least expensive strategy that meets your acceptance and response-time requirements. Where added candidates improve coverage but not selected accuracy, investigate selection before raising the budget. Where neither improves, inspect the task specification, retrieved evidence and generator before adding more attempts.
Use the agent evaluation checklist to record the decision. Re-run the comparison after a material model, prompt, verifier or task-distribution change. Extra inference is a resource to allocate against observed failures, not a substitute for understanding them.