Barbarians at the Gate: How AI is Upending Systems Research

Chapter 1: Why Systems Research is the Perfect Target

Not every research domain is equally suited for AI-driven discovery. The authors identify three properties that make systems performance problems a uniquely good fit:

1. Easy to verify

A scheduling algorithm either meets the deadline or it does not. A load balancer either distributes requests evenly or it does not. You can implement the candidate in a real system or simulator, run representative workloads, and measure the result. No subjective judgment. No ambiguous criteria. The evaluator is a deterministic function: code in, numbers out.

Contrast this with, say, AI-driven theorem proving or creative writing, where "is this solution correct/good?" is itself a hard problem. In systems research, verification is essentially free.

2. Correctness is easy to check

Solutions to systems performance problems typically preserve the correctness of the original system. A load balancer must still assign all tasks. A scheduler must still meet deadlines. A transaction reorderer must still execute all transactions. These constraints are binary: either satisfied or not. A quick check in the evaluator catches any violation.

3. The code that evolves is small

The core logic of a scheduler, load balancer, or resource allocator is often a short function — tens to hundreds of lines. This is small enough to fit in an LLM context window and small enough for the LLM to reason about coherently. You are not evolving an entire operating system. You are evolving a policy function.

Which problems are NOT well-suited?

The authors are candid about limits. ADRS does not work well for:

• Distributed protocols — consensus algorithms, failure detectors, network protocols where correctness requires reasoning about interleaving of multiple components
• Problems where semantic equivalence is hard — e.g., verifying that a modified SQL query plan produces the same results as the original
• Problems solvable by exact methods — integer linear programming, constraint solvers, where provably optimal solutions exist
• GPU-heavy or slow evaluations — weight compression or distributed training where a single evaluation takes 12+ hours

A survey of systems venues

The authors surveyed top systems conferences (NSDI, OSDI, SIGMOD, SOSP, VLDB) and found that over one-third of published papers feature performance optimization algorithms as their core contribution. These are all potential ADRS targets. The domains include:

Chapter 2: The ADRS Loop

ADRS automates the Solution and Evaluation stages of the traditional systems research process. The full research pipeline has five stages; ADRS replaces the middle two:

The automated core — stages 3 and 4 — forms a feedback loop. ADRS consists of five components that implement this loop:

The Five Components

These form two nested loops:

• Inner loop (automated): Solution Generator → Evaluator → Storage → Solution Selector → Prompt Generator → repeat. This runs for 100–400 iterations, typically taking 1–5 hours.
• Outer loop (human-in-the-loop): The researcher observes the inner loop's outputs, adjusts the problem formulation, changes the evaluator, adds hints to the prompt, and relaunches. This is the creative oversight layer.

Data flow through one iteration

Let us trace a concrete example. Suppose we are evolving a load-balancing policy for MoE inference. Here is what happens in one iteration:

Notice the richness of the evaluator feedback. It is not just a number — it includes why the score changed, which helps the LLM in the next iteration. This qualitative feedback is one of the paper's key recommendations.

Chapter 3: OpenEvolve — The Engine

The paper uses OpenEvolve as its primary ADRS framework. OpenEvolve is an open-source implementation inspired by Google DeepMind's AlphaEvolve. Let us trace the data flow through a single iteration.

Step 1: Build the prompt

The Prompt Generator assembles:

• The problem description — what the algorithm should do, what metric to optimize
• The current best program (or a sampled parent from the database)
• The evaluator feedback — scores, error messages, trace-level details
• Optional: previous solutions and their scores, so the LLM can learn from the search history
• Optional: solution hints from the researcher

This is the most important design decision. Too little context and the LLM explores blindly. Too much and it gets confused or converges prematurely.

Step 2: Generate a candidate

The Solution Generator sends the prompt to one or more LLMs. The paper recommends an ensemble of two models:

Step 3: Evaluate

The Evaluator runs the candidate in a simulator against predefined workloads. It returns:

• A numeric score (the optimization objective)
• A validity flag (did the solution violate any constraints?)
• Optional: qualitative feedback from an LLM that reads the trace logs
• Optional: per-workload breakdowns so the evolution can see where it is winning and losing

Step 4: Select and iterate

OpenEvolve uses an island-based evolutionary model. The population is divided into 3–5 islands that evolve semi-independently, with periodic migration of top solutions between islands. This prevents premature convergence.

Concrete numbers

The cost is remarkable. For the price of a lunch, you can search a space of algorithms that would take a PhD student months to explore by hand.

What a prompt actually looks like

The paper emphasizes that prompt engineering is critical. A well-structured prompt contains:

# System message (condensed):
"""
You are optimizing a scheduling policy.

PROBLEM: Minimize cost of running deadline-aware jobs
using spot instances. Spot instances are cheap but can
be preempted. On-demand instances are reliable but
expensive.

CURRENT BEST (score: 0.72):
[... full Python function ...]

EVALUATOR FEEDBACK:
- Average cost savings: 12% over greedy
- Failed on traces 7, 14: missed deadline by 2 ticks
- Spot utilization: 68%

PREVIOUS ATTEMPT (score: 0.31):
- Added random delay → missed 40% of deadlines

INSTRUCTIONS: Propose a modification to improve cost
savings while meeting ALL deadlines. Return your
changes as SEARCH/REPLACE blocks.
"""

The key insight: include the failure history. Showing the LLM what did not work is as valuable as showing what did. This prevents the search from revisiting dead ends and helps the LLM understand the constraint landscape.

Chapter 4: Case Study — Spot Instance Scheduling

Cloud providers offer spot instances — spare compute at 60–90% discount. The catch: they can be preempted at any time. Your job gets killed, you lose progress, and you must restart on a new instance (incurring a changeover delay).

The scheduling problem: given a job with a deadline, minimize cost by using spot instances as much as possible, falling back to expensive on-demand instances only when necessary. If you use spots too aggressively, preemptions pile up and you miss the deadline. If you are too conservative, you overpay on on-demand.

The baseline: Uniform Progress

The SOTA comes from Wu et al. (NSDI '24), called Uniform Progress. The idea is simple: track expected progress versus actual progress. If you are behind schedule, switch to on-demand. If you are ahead, stay on spot.

def UniformProgress(has_spot, state, env, task):
    # Calculate uniform progress rate
    progress_rate = task.duration / task.deadline
    expected = env.elapsed_time * progress_rate
    actual = task.progress_made

    # Simple decision: behind schedule?
    if actual < expected:
        # Must use any available resource
        if has_spot:
            return ClusterType.SPOT
        else:
            return ClusterType.ON_DEMAND

    # Default: use spot if available
    if has_spot:
        return ClusterType.SPOT
    else:
        return ClusterType.NONE  # Wait

This is a fixed formula. It does not learn from patterns in spot availability. It treats every spot instance the same, regardless of whether spots have been stable for an hour or getting preempted every 30 seconds.

The evolved policy: Adaptive Strategy

After 400 iterations (~5 hours, <$20), OpenEvolve discovers a fundamentally different approach:

def AdaptiveStrategy(has_spot, state, env, task):
    # Track recent spot availability patterns
    self.window.append(has_spot)
    alpha = window_avg(self.window)
    streak = longest_run(self.window)
    tail = trailing_run(self.window)

    params = get_params(classify_situation(
        alpha, streak, tail))

    # Compare remaining time to safety margin
    need = ticks_needed(task, env)
    slack = ticks_remaining(env)
    safety = safety_margin(task, params)

    if need >= slack:
        lock_on_demand()
        return ClusterType.ON_DEMAND

    # Classify situation: stable, moderate, unstable
    # Adjust behavior based on pattern classification
    if has_spot and looks_stable(streak, tail, params):
        return ClusterType.SPOT
    elif slack < safety * params.wait_margin:
        return ClusterType.NONE  # Wait for stable spot
    else:
        start_rebuilding(params.dwell)
        return ClusterType.ON_DEMAND

Look at the innovations the LLM discovered:

• Sliding window for spot patterns — tracks recent availability to classify the situation as stable, moderate, or unstable
• Selective waiting — when spots are unreliable, the policy waits for stable ones rather than grabbing every available instance (which causes costly changeover delays)
• Dynamic safety margins — safety buffers expand when preemption is frequent and shrink when spots are stable
• Hysteretic switching — avoids rapid flip-flopping between spot and on-demand by requiring sustained stability before switching

Evolution trajectory

The search process reveals how the LLM discovers these strategies incrementally:

• Iterations 1–90: Learns to track spot availability, recognizes patterns
• Iteration 180: Adds adaptive safety margins based on spot stability
• Iteration 240: Adds fine-grained situation classification (stable/moderate/unstable)
• Iteration 350: Learns dynamic safety margins that scale with preemption cost
• Iteration 389: Final policy integrates all mechanisms with lean parameters

Chapter 5: Case Study — MoE Expert Placement

Mixture-of-Experts (MoE) models like DeepSeek-V3 route each token to a subset of experts (specialized sub-networks). During inference, experts are distributed across multiple GPUs. The problem: some experts are "hot" (frequently activated) and others are cold. If hot experts cluster on one GPU, that GPU becomes a bottleneck.

The EPLB baseline

Expert Parallelism Load Balancing (EPLB) from DeepSeek runs in three stages:

1. Distribute expert groups across nodes
2. Create replicas of hot experts
3. Assign replicas to GPUs to minimize imbalance

The initial implementation uses greedy bin-packing: sort experts by load in descending order, assign each to the least-loaded GPU. It is written in pure Python with for-loops and takes ~540 ms per rebalancing step.

What OpenEvolve discovers

After 300 iterations (~5 hours, <$10), the LLM makes two critical breakthroughs:

Breakthrough 1: Replace Python loops with tensor operations

The initial code uses nested Python for-loops to search for the best GPU placement. The evolved code replaces this with PyTorch tensor operations — reshaping, indexing, and assignment via vectorized ops. This alone drops the runtime dramatically.

Breakthrough 2: Zigzag placement pattern

Instead of greedy bin-packing, the evolved code discovers a "snake" or zigzag pattern that alternates high-load and low-load experts across GPU slots:

def EvolvedStrategy(...):
    indices = torch.arange(num_items)
    block_id = indices // len(num_packs)
    idx_in_block = indices % len(num_packs)
    is_even_block = block_id % 2 == 0

    # The "snake" pattern: no loop at all
    # For items in even blocks:
    #   assign them to each pack; while
    # for odd blocks: use a reversed order
    packs_for_sorted_items = torch.where(
        is_even_block,
        idx_in_block,
        num_packs - 1 - idx_in_block
    )
    # Now we have assigned a pack for each item
    update_packs(packs_for_sorted_items)

The zigzag naturally interleaves heavy and light experts. Think of dealing cards: if you sort the deck from highest to lowest, then deal them in a zigzag across players (1,2,3,3,2,1,1,2,3,...), each player ends up with a roughly equal total.

Understanding the imbalance factor

The optimization target is the imbalance factor: the ratio of maximum tokens on any GPU to the average tokens per GPU. Perfect balance gives 1.0. Higher means some GPUs are overloaded.

For a model with E=256 experts across G=8 GPUs, each expert has a different load (number of tokens routed to it). The greedy approach sorts by load [high, ..., low] and assigns in order: GPU0 gets the heaviest, GPU1 the next, etc. But this means GPU0 always gets the heaviest experts in each round.

The zigzag reverses every other round: round 1 goes GPU0→GPU7, round 2 goes GPU7→GPU0. If we have experts sorted by load [100, 90, 80, 70, 60, 50, 40, 30] and 4 GPUs:

Wait — in this toy example, greedy is perfect! The advantage of zigzag appears with replicas. When hot experts are replicated, the assignment becomes more complex: replicas of the same expert must go to different GPUs, and the number of replicas varies. The greedy loop must search for valid placements via nested for-loops (O(E × G)), while the tensor zigzag does it in O(E) with vectorized indexing. Same imbalance, vastly less computation.

Evaluator design

The simulator models a distributed GPU inference engine for MoE (168 LOC). It uses ShareGPT and GSM8K traces. The combined metric is a weighted average of imbalance factor and rebalancing runtime — a single scalar for the evolutionary loop to optimize.

Chapter 6: Case Study — LLM-SQL Queries

Imagine running SQL queries over a large table by invoking an LLM for each row. This is relational analytics with LLMs (Liu et al., MLSys '25). Each row triggers a separate LLM inference call. For a table with 10,000 rows, that is 10,000 forward passes. The cost is prohibitive.

The key insight: prefix cache hit rate

Modern LLM inference engines cache the key-value pairs (KV cache) for token prefixes. If consecutive requests share a long common prefix, the engine can reuse the cached KV pairs and skip recomputation. The metric to maximize is the Prefix Hit Rate (PHR): the fraction of tokens shared between consecutive rows' input sequences.

The trick: if you reorder the rows and columns of the table before feeding them to the LLM, you can dramatically increase prefix sharing. Rows with similar values end up adjacent, and their serialized representations share long prefixes.

The combinatorial challenge: for a table with n rows and m fields, there are n! × (m!)ⁿ possible orderings. Brute-force is impossible. For a modest table of 100 rows and 5 fields, that is 100! × (120)¹⁰⁰ — an astronomically large search space.

The baseline: QuickGreedy (GGR)

The published SOTA uses Greedy Recursive Grouping: recursively group rows by common field values, reorder fields using schema statistics. It achieves good PHR but is slow due to repeated counter operations and deep recursion:

def QuickGreedy(df):
    # 1. Compute value counts for all values
    counts = Counter(df.stack())
    val_len = {v: len(str(v))+2 for v in counts}
    # 2. Pick max-value column
    v_star = argmax(...)
    # 3. Split rows with/without v_star
    G = rows with v_star
    R = rows without v_star
    # 4. Reorder columns in G, recurse on remainder
    # 5. Recurse on G remainder and R
    return concat(G, R)

The evolved policy

After 100 iterations (~1 hour, <$7), OpenEvolve discovers a fundamentally different approach:

def EvolvedPolicy(df):
    # 1. Cache value counts (compute ONCE)
    counts = Counter(df.stack())
    val_len = {v: len(str(v))+2 for v in counts}
    # 2. Larger base cutoff → fewer recursions
    BASE = 400
    v_star = argmax(...)
    if v_star is None: return fixed_reorder(df)
    # 3. Recurse only if |df| > BASE
    if len(df) > BASE:
        top, bottom = split(df)
        return concat(EvolvedPolicy(top),
                       EvolvedPolicy(bottom))
    # 4. Lightweight per-row heuristic:
    # sort by squared value length × continuity
    prev = None
    for row in df:
        if prev is None: ...
        matches = [c for c in df.columns
                   if row[c]==prev[c]]
        matches.sort(key=lambda c:
            val_len[row[c]], reverse=True)
        order = matches + [c for c in df.columns
                           if c not in matches]
    return df

The key innovations:

• Cached value counts: Compute once globally, eliminating redundant counting at each recursion level
• Direct attribute mapping: Replaces slow Pandas lookups with O(N_rows × N_cols) operations
• Local per-row heuristic: Instead of global sorting, reorders fields per-row by maximizing continuity with the previous row, weighted by squared value length
• Higher base threshold: Reduces recursion depth, trading grouping accuracy for speed

Evolution process

The search progresses in phases:

• Iteration ~32: Discovers ordering by frequency × squared length (eliminates redundant value counting)
• Iteration ~72: Adds normalized weights and multi-key sorting for the most informative columns
• Iteration ~97: Final design with optimal speed-accuracy tradeoff, NumPy-based reordering replacing Pandas

Chapter 7: Case Study — Transaction Scheduling

In database systems, conflicts between transactions on shared data cause performance bottlenecks. If transaction A writes to key X and transaction B also writes to key X, they conflict. The scheduler must order transactions to minimize the total number of conflicts, thereby improving throughput.

The problem formulation

Given a set of transactions (each a sequence of read/write operations on keys), find an ordering that minimizes makespan — the total time to execute all transactions. This is equivalent to minimizing the cost of conflicts, because conflicts cause serialization delays.

The paper considers two settings:

The baseline: SMF (Shortest Makespan First)

SMF (Cheng et al., VLDB '24) is a greedy O(n × k) algorithm. At each step, it samples k random unscheduled transactions, picks the one that adds the least incremental cost (conflict-weighted), and appends it to the schedule.

Online setting: rediscovery

In the online setting, OpenEvolve rediscovers SMF from scratch, starting from a random baseline. This is likely due to contamination (the model saw the SMF paper during training), but it is still notable: the evolutionary framework converges to the known SOTA without being told about it.

Offline setting: genuine discovery

In the offline setting, where no prior published solution exists, OpenEvolve finds a novel algorithm that beats SMF by 20–34%. The evolved algorithm combines three techniques:

def get_best_schedule(self, num_seqs=10):
    n = len(self.txns)
    idxs = list(range(n))

    # --- 0. Pre-compute transaction statistics ---
    write_cnt = {t: sum(1 for op in self.txns[t]
                     if op[0] == "w")
                 for t in idxs}

    # --- 1. Sorted sequence (by #writes, length) ---
    base_seq = sorted(idxs, key=lambda t:
        (write_cnt[t], len(self.txns[t])))
    best_seq = base_seq
    best_cost = self.get_opt_seq_cost(best_seq)

    # --- 2. Greedy insertion ---
    improved = True
    while improved:
        improved = False
        for i in range(n):
            txn = best_seq.pop(i)
            # Try every position for this txn
            for pos in range(n):
                candidate = best_seq[:pos]+[txn]+best_seq[pos:]
                cost = self.get_opt_seq_cost(candidate)
                if cost < best_cost: ...

    # --- 3. Pair-swap hill climb ---
    for _ in range(max_iters):
        i, j = random.sample(range(n), 2)
        # Swap and accept if better

    # --- 4. Random restarts (keep best) ---
    for _ in range(num_seqs):
        seq = idxs[:]
        random.shuffle(seq)
        cost = self.get_opt_seq_cost(seq)
        if cost < best_cost: ...

    return best_cost, best_seq

The four-phase structure emerged from evolution:

1. Smart initialization: Sort by writes and length (fewer writes first), creating a conflict-aware starting point
2. Greedy insertion: For each transaction, remove and re-insert at the best position (O(n²) but powerful)
3. Pair-swap hill climbing: Random pairwise swaps to escape local minima
4. Random restarts: Try completely random orderings to diversify the search

Understanding makespan cost

SMF is built around the concept of incremental conflict cost. When you add a transaction to the schedule, you pay a cost proportional to how many already-scheduled transactions conflict with it. The makespan increase from adding transaction t_i at position k is:

where conflict(t_i, t_j) = 1 if t_i and t_j share a key with at least one write. SMF greedily picks the transaction with the lowest incremental cost at each step. The evolved offline algorithm goes further: it also considers the future cost of each placement by running hill climbing and random restarts after the initial greedy pass.

The evaluator uses traces from 5 OLTP benchmarks (Epinions, SmallBank, TPC-C, TAOBench, YCSB) with 500 transactions each. It measures total makespan and provides statistical bounds on the schedule quality.

Chapter 8: Failure Modes & Best Practices

ADRS is not magic. The paper is refreshingly honest about failure modes. Based on analysis of 420 LLM-judged traces, they identify three categories:

1. Runtime Errors

2. Search Failures

3. Algorithm Failures

The evaluation process

The failure distribution comes from an interesting methodology. The authors took 420 individual evolution traces (candidate solutions across all case studies) and had an LLM classify each one into failure categories. This meta-evaluation reveals that the majority of failures are mundane runtime errors, not deep algorithmic issues. This suggests a practical intervention: better sandboxing, automatic syntax checking, and interface validation before sending code to the expensive simulator would dramatically improve evolution efficiency.

Best practices by component

Prompt Generator

• Provide structured specifications — clearly state the problem, evaluation criteria, and constraints
• Start with a suitable base program — a clean, minimal baseline accelerates progress. Buggy or weak baselines waste iterations on trivial fixes (budget exhaustion). Overly strong baselines restrict exploration to micro-optimizations
• Provide hints judiciously — too many hints cause premature convergence, too few leave the search blind. Inject hints when the search gets stuck
• Control the abstraction level — hide high-level library APIs to prevent sub-optimal shortcut solutions

Solution Generator

• Use a two-model ensemble — reasoning model (20%) for exploration + fast model (80%) for exploitation
• Do not use more than two models — three or more models cause mutation drift

Evaluator

• Prevent overfitting — use broad, diverse test sets. Sample different subsets for feedback during search versus final evaluation
• Prevent reward hacking — combine multiple metrics, add adversarial tests, require all constraints to be satisfied

Solution Selector

• Balance exploration-exploitation — greedy selectors converge fast but get stuck; random selectors waste resources. Island-based methods (MAP-Elites) strike the best balance

Chapter 9: Connections & Implications

Full results table

Here is the summary of all 11 tasks investigated in the paper:

What this means for researchers

The authors argue that the human researcher's role shifts from algorithm designer to problem architect:

• Problem formulation becomes the high-leverage skill — defining the right objective, constraints, and evaluation framework
• Evaluator design becomes critical — a good evaluator is the difference between useful evolution and reward hacking
• Interpretation matters more — understanding why the evolved algorithm works, distilling its insights, and connecting them to theory
• Manual algorithm design matters less — the generate-and-test cycle is increasingly automated

Related frameworks

How ADRS compares to AlphaEvolve

This paper is closely related to AlphaEvolve (Novikov et al., Google DeepMind, 2025), but there are key differences:

The ADRS paper's unique contribution is the meta-analysis: rather than showing one spectacular result, it systematically evaluates ADRS across 11 diverse tasks, categorizes failure modes, and distills actionable best practices. It is a roadmap for the field, not just a proof of concept.

Open challenges

• Better evaluators: Build simulators with higher fidelity, generality, and speed. Lightweight VMs and container snapshots can shorten feedback cycles.
• Better search: Current search often gets stuck. Better exploration-exploitation balance, adaptive prompt strategies, and cross-domain transfer of search insights are all open problems.
• Scaling to complex systems: ADRS currently works on isolated functions. Evolving multi-component distributed protocols remains unsolved.
• Standardized benchmarks: No common evaluation suite exists for ADRS frameworks. Building one (like MLGym for AI research) would accelerate the field.

← Back to Veanors