ARCHON: Architecture Search for Inference-Time Techniques

Chapter 1: The Key Insight

ARCHON's insight is to reframe inference-time system design as a problem that looks exactly like neural architecture search. In NAS, you search over configurations of layers, activation functions, and connections. In ARCHON, you search over configurations of LLM components — generators, critics, rankers, fusers, verifiers — arranged in layers.

The analogy runs deep. In a neural network, each layer transforms a tensor. In ARCHON, each layer transforms a list of strings. Generator layers produce candidate responses. Fuser layers combine multiple candidates into fewer, better ones. Ranker and verifier layers act like non-linearities — they filter the list, keeping only the good candidates.

Like neural networks, ARCHON architectures are deployed off-the-shelf — no weights are learned between components. The "training" happens during architecture search on a small dev set (20% of target data). Once the optimal configuration is found, you run it on new inputs without further tuning.

The framework also reveals a key finding that manual system design misses: the best combination depends on the task. Instruction-following benefits from multiple layers of critiquing and fusing. Math tasks want broad generation followed by quick filtering. Coding tasks need repeated generation with unit-test verification. No single architecture dominates, which is exactly why you need automated search.

What makes this different from prompt engineering?

Frameworks like DSPy optimize what you say to a single model. ARCHON optimizes how you orchestrate multiple models and techniques. The difference matters: even the perfect prompt to GPT-4o can't match the accuracy of a 10-model ensemble with iterative critique-and-fuse, because a single model has a fixed error distribution. Multiple models have different error distributions, and combining them reduces errors that no single prompt can fix.

Think of it this way: prompt engineering is like tuning the knobs on one instrument. ARCHON is like composing for an orchestra — choosing which instruments to include, what each one plays, and how to layer the parts. The search algorithm is the automated composer.

Chapter 2: The Design Space

ARCHON defines six types of LLM components. Each one is a pluggable module that performs a text-to-text operation. Think of them as the "layer types" in ARCHON's neural-network analogy.

1. Generator

Takes the instruction prompt, outputs a candidate response. Can be called in parallel across multiple models (generation ensembling) and each model can be sampled multiple times (repeated sampling). Temperature is set to 0.7 for all components to ensure response diversity. Generators are always the first layer — they are the only component that doesn't require prior candidates as input.

The paper finds that sampling from additional models gives nearly double the benefit of additional samples from the same model (18.5% vs 9.3% improvement). Different models make different errors, so ensembling captures a wider range of correct strategies than temperature-based diversity from a single model.

2. Fuser

Takes the instruction prompt and a set of candidate responses, produces one or more higher-quality fused responses by combining the best elements. Averaging 8.9% improvement across all benchmarks tested. Can be stacked in multiple layers for iterative refinement — MT-Bench gains 10-15 points from 3-4 fusion layers.

3. Critic

Takes the prompt and candidates, produces strengths and weaknesses for each response. These annotations are passed to the next layer (typically a Ranker or Fuser) to inform its decisions. Adds 11.5 percentage points on average when combined with Generator ensemble + Fuser.

4. Ranker

Takes the prompt and candidates, ranks them by quality using pairwise comparisons focused on style and prompt adherence. Filters to the top-K. Improves output quality by 10.8% over random selection, performing within 2.7% of oracle selection.

5. Verifier

Two-stage process: (1) generates reasoning for why a candidate is correct, (2) takes in the original prompt, candidate, and generated reasoning, then produces a binary verdict — [Correct] or [Incorrect]. Only verified responses pass to the next layer. The two-stage design prevents the verifier from rubber-stamping answers: stage 2 can catch flaws in stage 1's reasoning.

Most effective for reasoning tasks, improving MixEval/MATH by 8.4%. However, when used alone (without other techniques), the Verifier lags behind ensemble+fuser by 1.5%, suggesting verification works best as a complement to other techniques, not a replacement.

6. Unit Test Generator + Evaluator

Complementary pair: the Generator produces 5-10 test statements for the instruction, the Evaluator scores candidates against those tests. Devastating for coding — boosted CodeContests Pass@1 by 56% (17.9% to 29.3%).

Every module is text-in, text-out

This uniform interface is what makes ARCHON composable. Any module's output can feed into any other module's input (subject to placement rules). The state flowing through the architecture is always the same type: instruction prompt + list of candidate strings.

Chapter 3: Search Strategy

The design space has 9,576 valid configurations. Exhaustive search is impractical because each evaluation requires running the full architecture on a dev set — dozens of LLM calls per query. ARCHON needs to find the optimum in as few evaluations as possible.

The six search hyperparameters

Where 9,576 comes from

The combinatorial math: 10 choices for top-K generators x 5 for samples per generator x 4 for fusion layers x 5 for fusers per layer x 2 for critic/ranker on/off x 3 for evaluation layer = 6,000. But many configurations are invalid — for example, configurations where the total number of initial generations exceeds the context window of the fusers. After pruning invalid combinations, 9,576 remain. This is large enough that random search is wasteful, but small enough that Bayesian optimization can efficiently navigate it.

Three search methods compared

Random search samples architectures uniformly at random. Simple but wasteful — most of the budget explores bad configurations.

Greedy search optimizes one hyperparameter at a time, holding the rest fixed. Better, but gets stuck in local optima because it never jointly considers interactions between hyperparameters.

Bayesian optimization (ARCHON's default) maintains a Gaussian process surrogate model of the objective function. It starts by evaluating ~230 random architectures to calibrate, then uses an acquisition function to propose the most promising configurations. Each evaluation refines the surrogate, guiding the search toward the global optimum.

To impose compute constraints, ARCHON simply excludes any architecture exceeding the inference call, input token, or output token budget from the search space. This happens before Bayesian optimization even considers them — invalid configurations are never evaluated.

The Bayesian optimization loop in detail

The key advantage: after calibration, the Gaussian process can predict which configurations are promising without actually running them. This is what makes Bayesian optimization 10x more sample-efficient than random search — it concentrates evaluations on the promising region of the search space.

Chapter 4: Architecture Examples

What do discovered ARCHON architectures actually look like? The paper provides detailed specifications for both general-purpose and task-specific configurations. The differences are revealing.

General-purpose all-source architecture

The best general-purpose architecture starts with a broad initial layer of 10 generators (the top performers from Chatbot Arena). Then four successive layers of critique and fusion, using Qwen2-72B for critiquing and Claude 3.5 Sonnet for fusing. Each layer has progressively fewer fusers — 8, then 6, then 4, then 1 — creating a "funneling" effect that distills many candidates into a single refined output.

Task-specific architectures differ dramatically

Instruction-following (MT-Bench, AlpacaEval): Multiple layers of critiquing and fusing with diverse LLMs. The diversity of perspectives matters more than raw generation count. Architecture uses 3-4 Critic+Fuser layers with a mix of open and closed-source models.

Math (MATH benchmark): Broad initial generation (many samples), then quick reduction. The architecture favors many candidates from strong math models with a verifier to filter incorrect reasoning before a single fusion step.

Coding (CodeContests): Repeated generation from a single strong model (GPT-4o, up to 1000 samples), unit test generation and evaluation, then fusion. The architecture is narrow and deep — multiple iterations of generate-critique-fuse over a single response thread, not broad ensembling. When unit-test generation is added with high sampling, CodeContests Pass@1 jumps from 17.9% to 29.3% — and with the full task-specific architecture, it reaches 41.4%.

The "funneling" effect in numbers

The general-purpose architecture starts with 10 generators (each sampled once = 10 candidates). The first fusion layer uses 8 fusers, each processing the full set. The second layer uses 6 fusers operating on the fused outputs. Then 4 fusers, then a single final fuser. At each stage, the number of candidate responses shrinks while the quality increases:

Chapter 5: Pareto Frontier

The fundamental question in inference-time compute is: how much accuracy do you get per token spent? ARCHON pushes the Pareto frontier — achieving higher accuracy at every budget level compared to fixed architectures.

How ARCHON sweeps the frontier

By adding token budget constraints to the architecture search, ARCHON discovers different optimal architectures for different budgets. At low budgets, it might use a single strong model with one fusion layer. At high budgets, it deploys the full funneling architecture with 10 generators and 4 fusion layers. The key insight: the architecture itself adapts to the budget, not just the number of samples.

Concrete numbers from the paper

ARCHON's general-purpose architecture uses 31% fewer tokens than the best baseline (ADAS) while achieving 6.4% better performance. Even the task-specific architectures, which use more tokens, still use 15.1% fewer input tokens and 13.5% fewer output tokens than ADAS.

How budget constraints reshape architectures

At low budgets (<20 inference calls), ARCHON discovers architectures that look like enhanced single-model systems: one strong generator with a Critic and a single Fuser. The architecture search degenerates to model selection plus minimal post-processing.

At medium budgets (20-40 calls), the architectures start incorporating 3-5 generators and 2 fusion layers. The search discovers that diversity at the generation layer is the most efficient way to spend the extra calls.

At high budgets (40+ calls), the full funneling architecture emerges: 10 generators, Critic, Ranker, 3-4 Fuser layers, and a Verifier. The marginal return on each additional layer is still positive but decreasing — the first Fuser layer gives the biggest jump.

Chapter 6: Results

ARCHON was evaluated across seven benchmarks spanning three task categories. Here are the headline numbers from Table 1 of the paper, showing the best all-source task-specific architectures.

Instruction-following

Reasoning

Coding

The CodeContests result is remarkable: ARCHON's task-specific architecture more than doubles GPT-4o's Pass@1 (18.1% to 41.4%) by using high-sample generation with unit-test filtering. Even compared to o1 — which has its own internal chain-of-thought compute — ARCHON achieves 9.9 percentage points higher.

Open-source ARCHON is competitive

Even when restricted to only open-source models (no GPT-4o, no Claude), ARCHON's task-specific architectures achieve a 11.2% average improvement over the best individual open-source LLM. On MT-Bench, the open-source ARCHON reaches 71.1% win rate — beating single-call GPT-4o (44.2%) and approaching the closed-source ARCHON (77.0%). This demonstrates that orchestration can compensate for individual model quality.

Head-to-head: model sources compared

Notably, open-source ARCHON beats closed-source ARCHON on CodeContests (28.9% vs 25.1%). This is because the top open-source coding models (DeepSeek, CodeLlama variants) generate more diverse solution strategies. When both pools are available, the all-source architecture can pick the best of both worlds — open-source generators for diversity, closed-source models for critiquing and fusing.

Generalization to unseen tasks

The general-purpose ARCHON architecture was tested on three benchmarks it was never searched on: GPQA (graduate-level questions), MMLU, and MMLU-Pro. Results:

ARCHON's general-purpose architecture retains 91-94% of the task-specific performance on unseen tasks, while ADAS and AFlow retain only 67-74%. The funneling architecture — broad generation plus iterative critique-and-fuse — is a surprisingly robust general strategy.

Chapter 7: Technique Interactions

The most scientifically interesting part of the paper isn't the final numbers — it's the analysis of how inference-time techniques interact. The authors identify four key trends from exhaustive ablations across seven benchmarks.

T1: Scaling generation always helps

Both repeated sampling (more samples per model) and model ensembling (more models) produce substantial gains: 9.3% and 18.5% respectively. Ensembling from additional models gives nearly double the benefit of additional samples from the same model, because diverse models make different errors.

T2: Stacking technique layers compounds gains

Adding layers of inference-time techniques — particularly Fuser layers — significantly improves performance. Adding a single Fuser as the last layer is always beneficial. The marginal gain varies by task: MT-Bench and AlpacaEval gain 10-15 points from 3-4 fusion layers, while MixEval gains only 1-2 points.

T3: Diversity of techniques matters

Using different types of techniques (Critic before Ranker before Fuser) outperforms using only one type. Critics before Fusers are particularly effective because the critique annotations give the Fuser explicit guidance on what to keep and what to fix. Generation ensembling + critique + fusion improved performance by 18.8% on average over single-model generation.

T4: Verification filters out reasoning errors

For reasoning and coding tasks, adding Verifiers and Unit Test Generators after generation but before fusion filters out flawed responses. This is critical because fusion can propagate errors — if a wrong answer sounds confident, the Fuser might incorporate it. Verification prevents this. On CodeContests, adding unit tests to a high-sample GPT-4o setup boosted Pass@1 from 17.9% to 29.3%.

Concrete interaction examples from the ablations

Notice the pattern: for instruction-following (MT-Bench), the Critic provides the biggest single boost (+7.7 points on top of ensemble+fuser). For reasoning (MixEval), the Verifier matters more. For coding, unit tests with high sampling are transformative. The "full ARCHON" row uses the task-specific architecture from the search, which combines the right techniques for each task.

When interactions go wrong

Not all combinations help. The Verifier module, when used alone after generation (without a Fuser), actually hurts performance on instruction-following tasks. Why? The Verifier is trained to check factual correctness and reasoning chains, but instruction-following quality depends on style, tone, and completeness — dimensions the Verifier doesn't assess. It aggressively filters responses that are stylistically excellent but contain minor reasoning imprecisions, leaving behind dry, technically correct but uninspiring outputs.

Similarly, stacking too many fusion layers on MixEval yields diminishing returns (1-2 points after the first layer). The benchmark's questions are relatively unambiguous, so the first fusion pass captures most of the benefit. Additional layers just paraphrase without improving substance.

These negative and diminishing interactions are exactly why manual system design fails. You can't intuit which combinations help on which tasks without running the experiments — or letting a search algorithm do it for you.

Chapter 8: Cost Analysis

ARCHON's improvements come at a price: multiple LLM API calls per query. The paper is transparent about the costs, giving exact token counts and dollar estimates.

Wall-clock time and dollar costs

ARCHON architectures make multiple successive LLM calls for different operations. The paper estimates approximately 5x more time and money than a single LLM call. For the general all-source architecture (35 inference calls, ~50K input tokens, ~30K output tokens per query), the cost is roughly $0.32 per query at 2024 API prices.

When the cost is justified

The paper argues ARCHON makes economic sense for high-stakes domains: scientific research, competitive programming, complex agentic tasks, medical diagnosis. In these domains, the cost of a wrong answer far exceeds a few cents per query. For casual chat, single-model inference is fine.

Model size matters

ARCHON is most effective with 70B+ parameter models. With 7B models, the framework boosts performance by 7.5% over the best individual 7B model, but the gains are smaller and 7B models are weaker at critiquing and fusing. The Critic and Fuser roles require strong instruction-following capability that smaller models lack.

Interestingly, 7B models can still serve as effective Rankers — pairwise comparison is an easier task than open-ended critique or synthesis. This suggests a practical deployment pattern: use cheap 7B models for ranking and filtering, reserve expensive 70B+ models for generation and fusion. ARCHON's architecture search can discover these heterogeneous allocations automatically.

Latency breakdown

The critical path through an ARCHON architecture determines wall-clock latency. In the general-purpose architecture with 4 fusion layers, the critical path is: Generator (parallel, ~2s) + Critic (~3s) + Fuser Layer 1 (parallel, ~3s) + Critic + Fuser Layer 2 + ... The total is roughly 5x a single LLM call. However, because Generator and Fuser layers run their constituent models in parallel, the latency scales with the number of sequential layers, not the total inference calls. A 35-call architecture with 6 sequential layers takes about the same time as 6 sequential single-model calls.

The cost-quality tradeoff curve

ARCHON provides an explicit knob for this tradeoff via the budget constraint in architecture search. Need results in under $0.10/query? The search discovers a lean 3-call architecture. Have $0.50/query to spend? It discovers a 40-call architecture that squeezes maximum accuracy from that budget. This explicitness is a major advantage over systems like o1, where the compute allocation is opaque and non-configurable.

Chapter 9: Connections

ARCHON sits at the intersection of several research threads in inference-time compute scaling. Here's how it connects to the broader landscape.

Related approaches

The broader picture

ARCHON represents a shift from model-centric to system-centric scaling. Instead of making a single model bigger (training-time compute), you invest compute at inference time to orchestrate multiple models. This is complementary: ARCHON gets better as the individual models get stronger, and training-time improvements compound with inference-time orchestration.

Open questions and future work

Dynamic architectures. ARCHON currently discovers a fixed architecture per task or a single general-purpose architecture. A natural extension is per-query routing — using a lightweight classifier to select the right architecture for each input based on difficulty, domain, or estimated quality of initial generations.

Distillation. Once an ARCHON architecture produces high-quality outputs on a dataset, those outputs could be used as training data to distill the aggregate knowledge into a single model. This eliminates the multi-call overhead at deployment while retaining much of the quality gain.

New technique modules. The framework is open-source and extensible. New inference-time techniques can be added as modules, new models plugged in, and the search re-run. As the field develops new techniques (tree search, debate, reward models, process reward verification), ARCHON's architecture search can automatically discover how they interact with existing methods.

Latency optimization. The current search maximizes accuracy given a token budget. Adding latency as an explicit objective would discover architectures that parallelize more aggressively — for example, running multiple Fusers in parallel rather than sequential layers, or using faster models for intermediate Critic steps.

Scaling to more techniques. The current search space includes 6 technique types. As the field develops new methods — process reward models for step-by-step verification, debate between models, tree-of-thought search, or tool-augmented generation — the search space grows exponentially. Efficient search algorithms that scale to larger spaces (perhaps hierarchical Bayesian optimization or evolutionary strategies) will become essential.

Meta-learning across tasks. Currently, architecture search runs independently for each task. A meta-learning approach could learn priors over which architecture shapes work for which task types, dramatically reducing the search budget for new tasks. The paper's finding that instruction-following tasks prefer wide-then-fuse while coding prefers narrow-deep is a hint that such priors exist and are learnable.

ARCHON's code is available at github.com/ScalingIntelligence/Archon. The framework is implemented in Python, with each LLM component as a modular class. Adding a new technique requires implementing a single process(prompt, candidates) -> candidates method. The architecture search runs on a 20% dev set and typically completes in a few hours depending on the evaluation budget.