LLM Optimizers via Agent-System Interfaces

Chapter 1: The Key Insight

Here is the central idea: give an LLM a high-level DSL to write mappers in, run the mapper, and feed back structured profiling data — not just a scalar throughput number — so the LLM knows what to fix.

This is the Agent-System Interface (ASI): an abstraction layer between the LLM agent and the parallel runtime system. It has two parts:

The combination is powerful for a surprising reason. The DSL is a brand-new language with zero examples in any LLM's training data. C++ has millions of examples. Yet LLMs generate correct DSL mappers at an 80% success rate on the first try, versus 0% for C++. The abstraction makes the task tractable even without prior exposure.

The full system runs for just 10 iterations. In 10 minutes per application, it finds mappers that surpass expert-written mappers by up to 1.34x — the same mappers that took human experts days to write.

Chapter 2: What Are Mappers?

Before we can optimize mappers, we need to understand what decisions they encode. A mapper is a piece of code that makes three kinds of assignments for every task and data region in a parallel program:

1. Processor Selection

Each task can run on a GPU, a CPU, or the OpenMP runtime. This isn't a one-size-fits-all decision. Small tasks may prefer CPUs because GPU kernel launch overhead dominates. Tasks with huge memory footprints may need CPUs when GPU memory is insufficient. Compute-dense tasks belong on GPUs. The right choice depends on the task's specific characteristics.

2. Memory Placement

Each data region must live somewhere in the memory hierarchy. The options for GPU-targeted tasks include:

• GPU FrameBuffer: Fastest access, but limited capacity
• ZeroCopy memory: CPU-GPU shared, moderate speed, no explicit transfer needed
• CPU System Memory: Plentiful but slow for GPU access

Each choice trades off speed, capacity, and data transfer cost. And this decision is per-task, per-argument — the same data region might need to be in different places for different tasks.

3. Memory Layout

How data is arranged in memory affects cache efficiency dramatically:

• Struct of Arrays (SOA) vs. Array of Structures (AOS): SOA groups the same field across all elements together (better for SIMD). AOS keeps all fields of one element together (better for locality).
• C-order vs. Fortran-order: Row-major vs. column-major. The wrong choice means every memory access is a cache miss.
• Alignment: 64-byte or 128-byte alignment enables vectorized memory operations.

4. Index Mapping

When a parallel for-loop launches thousands of tasks over partitioned data, an index mapping function determines which partition runs on which processor. Think of it as a function from a task index (e.g., grid point [i, j]) to a processor index (e.g., GPU 3 on node 2). A bad mapping means data shuffles between GPUs on every iteration. A good mapping keeps communication local.

This is why expert-written mappers represent the gold standard. A systems researcher who understands both the application and the machine can make informed choices about all these dimensions simultaneously. The question this paper asks: can an LLM match that expertise?

Chapter 3: The Agent-System Interface

This is the core contribution. The Agent-System Interface (ASI) sits between the LLM and the Legion parallel runtime, making it possible for the LLM to generate and iterate on mapper code.

The DSL: From C++ to Clarity

Here's what a mapper looks like in the original C++ API versus the DSL. The same mapping — running task0 on GPUs, placing ghost regions in ZeroCopy memory, using C-order SOA layout with 64-byte alignment, and distributing tasks cyclically across GPUs:

DSL Statement Types

The DSL has four statement types, each controlling one dimension of the mapping decision:

# 1. Processor selection: which processor type runs this task
Task task0 GPU;

# 2. Memory placement: where this data lives
Region * ghost_region GPU ZCMEM

# 3. Memory layout: how data is arranged
Layout * * * C_order SOA Align==64

# 4. Index mapping: custom distribution function
def cyclic(Task task):
    ip = task.ipoint;
    mgpu = Machine(GPU);
    node_idx = ip[0] % mgpu.size[0];
    gpu_idx = ip[0] % mgpu.size[1];
    return mgpu[node_idx, gpu_idx];

IndexTaskMap task4 cyclic

Each statement is independent. The processor choice for task0 doesn't constrain the memory choice for task1. This separation of concerns is the key design principle: it eliminates the tangled dependencies in C++ mapper code and gives the LLM independent knobs to turn.

The Compiler

Behind the scenes, a compiler translates each DSL statement into the low-level C++ system API calls that the Legion runtime actually executes. The LLM never sees the C++. It writes DSL, the compiler handles the rest.

Chapter 4: The Optimization Loop

The optimization process is iterative: the LLM proposes a mapper, runs it, reads the feedback, and proposes a better one. Here's how it works step by step.

Inputs

The LLM agent receives two things before it starts:

• Server specifications: How many CPUs, GPUs per node, total nodes, memory sizes
• Application metadata: Task names and which data arguments each task accesses

These define the search space. The agent knows what processors exist and what tasks need mapping.

The Loop

Modular Code Generation

The agent doesn't generate the entire mapper as one monolithic block. Instead, the task is decomposed into independent segments — one for processor selection, one for memory placement, one for layout, one for index mapping. The agent decides what to generate for each segment separately.

The agent is implemented in the Trace framework, which treats each code segment as an independent optimization variable. This aligns with the DSL's design: each decision dimension is a separate knob the optimizer can turn.

The entire process takes 10 iterations and completes in about 10 minutes per application. For context, expert mapper development takes days.

Chapter 5: Structured Feedback

This is where the paper diverges most sharply from traditional auto-tuning. Instead of just feeding back a throughput number, AutoGuide interprets raw execution output into three layers of feedback.

Layer 1: Raw Execution Output

The runtime produces raw output: either an error message (if the mapper crashed) or performance metrics (execution time, throughput). This is what OpenTuner sees. It's almost useless for guiding the next iteration.

Layer 2: Explain

AutoGuide pattern-matches on the raw output to explain what went wrong. Examples:

Layer 3: Suggest

Based on the explanation, AutoGuide suggests specific mapper modifications:

The Ablation Proves It Matters

The authors tested four feedback configurations on three benchmarks:

• 0-shot / 5-shot (no feedback): LLM generates once with 0 or 5 examples. Worst performance across all benchmarks.
• Execution only: Raw output, no interpretation. Moderate improvement over no feedback.
• Execution + Explain: Error explanations help the LLM understand failures. Better.
• Execution + Explain + Suggest (full AutoGuide): Consistently best performance across all benchmarks.

Chapter 6: Results

The system is evaluated on 9 benchmarks: 3 scientific computing workloads (Circuit, Stencil, PENNANT) and 6 parallel matrix multiplication algorithms (Cannon's, SUMMA, PUMMA, Johnson's, Solomonik's, COSMA). Hardware: 2 Intel 10-core CPUs, 256 GB memory, 4 NVIDIA Tesla P100 GPUs. LLM: GPT-4o.

vs. Expert-Written Mappers

The expert-written mappers are the gold standard — created by domain scientists who spent years mastering both the application and the programming framework. The best mappers found by the LLM optimizer match or surpass every single expert mapper:

The largest gain is on Circuit: 1.34x speedup over the expert mapper. The improvement comes from a subtle memory placement decision — the LLM placed two data collections in GPU FrameBuffer instead of ZeroCopy memory. This trades slightly higher inter-GPU communication for much faster memory access, a net win.

vs. OpenTuner (RL-based)

OpenTuner uses reinforcement learning with scalar throughput feedback. The comparison is stark:

• At 10 iterations: The LLM optimizer (Trace) achieves 11x higher throughput than OpenTuner
• At 1000 iterations: Trace still maintains a 3.8x advantage, despite OpenTuner having 100x more iterations

vs. Random Mappers

Random mappers sample uniformly from the DSL-defined search space. Out of 90 random mappers (10 per benchmark), 74 (82.2%) crash with runtime errors. The ones that survive perform poorly. This confirms that the search space is hostile — most of it is invalid — and any effective method must navigate it intelligently.

Chapter 7: Analysis

What does the LLM actually change when it optimizes a mapper? The case studies reveal specific, interpretable optimization patterns.

Circuit: Memory Placement Insight

The expert mapper placed two data collections in GPU ZeroCopy memory (shared CPU-GPU, no explicit transfer). The LLM discovered that moving them to GPU FrameBuffer (dedicated GPU memory, fastest access) yields 1.34x speedup. Why?

• ZeroCopy avoids transfer overhead but has slower access per operation
• FrameBuffer requires data transfer but each access is much faster
• For these particular data collections, the access frequency is high enough that faster per-access time outweighs the transfer cost

The LLM found this through iteration: it tried FrameBuffer, saw the speedup in the profiling data, and kept it. A human expert chose ZeroCopy as the "safe" default. The LLM wasn't biased by convention.

COSMA: Index Mapping Optimization

The expert mapper for COSMA (a matrix multiplication algorithm) achieved 1.31x less throughput than the LLM's mapper. The difference was in the index mapping function — how matrix subblocks are distributed across GPUs. The LLM discovered a distribution that reduces inter-GPU communication by keeping related subblocks on the same GPU.

Common Patterns the LLM Discovers

• GPU preference: Move compute-heavy tasks to GPU whenever memory allows. This is the most common first optimization.
• FrameBuffer over ZeroCopy: For frequently accessed data, the raw speed of FrameBuffer often outweighs the convenience of ZeroCopy.
• Layout alignment: Adding 64-byte or 128-byte alignment for vectorized access. A free performance win that experts sometimes forget.
• Communication-aware distribution: Index mapping functions that keep communicating tasks on the same node or adjacent GPUs.

DSL Code Generation Success

The authors tested LLM code generation across 10 natural-language mapping strategies:

Chapter 8: The DSL Design

The DSL isn't just "simpler C++." It embodies specific design principles that make it suitable as an LLM-agent interface. Understanding these principles is important because they generalize: if you want an LLM to optimize any complex system, the interface design matters as much as the LLM itself.

Principle 1: Declarative, Not Imperative

The DSL says what to achieve, not how to implement it. Task task0 GPU means "run task0 on a GPU." The compiler handles the 50+ lines of C++ needed to actually implement GPU task scheduling. This means the LLM's output is a specification, and correctness is the compiler's responsibility.

Principle 2: Separation of Concerns

Each DSL statement controls one dimension independently. Processor selection doesn't affect memory placement syntax. Layout decisions don't constrain index mapping. This orthogonality means the LLM can reason about each dimension separately and iterate on one without destabilizing others.

Principle 3: Wildcard Matching

The * wildcard applies a decision to all matching entities. Region * ghost_region GPU ZCMEM applies to all tasks using ghost_region. Layout * * * applies to all tasks, all data, all processors. This gives the LLM a coarse-to-fine control strategy: start with broad defaults, then specialize.

Principle 4: Structured Search Space

Each statement has a finite set of valid options. Processor selection: {CPU, GPU, OMP}. Memory type: {FBMEM, ZCMEM, SYSMEM}. Layout order: {C_order, F_order}. Layout type: {SOA, AOS}. The DSL doesn't allow arbitrary code for these decisions — it constrains the LLM to valid choices. Only index mapping functions allow arbitrary arithmetic.

Principle 5: Expressiveness Where It Matters

The DSL is restricted for most decisions (finite option sets) but expressive for index mapping (arbitrary functions). This matches the problem structure: processor and memory choices are categorical decisions, but the best index mapping might require custom arithmetic. The def statement lets the LLM write real computation:

def block_cyclic(Task task):
    ip = task.ipoint;
    mgpu = Machine(GPU);
    # Block-cyclic distribution:
    # group 4 consecutive indices, then cycle
    block = ip[0] / 4;
    gpu = block % mgpu.size[1];
    node = (block / mgpu.size[1]) % mgpu.size[0];
    return mgpu[node, gpu];

Chapter 9: Connections

This paper sits at the intersection of several active research threads. Each connection reveals a broader pattern about how LLMs can improve systems.

Generative Optimization Family

The core technique — LLM proposes code, gets rich feedback, iterates — appears in several recent systems:

• AlphaEvolve (Google DeepMind, 2025): Uses LLMs to evolve algorithms for fundamental math and computing problems. Same pattern: propose code, evaluate, iterate. The key difference is that AlphaEvolve uses an evolutionary population with parent selection, while this paper uses a single-agent loop with structured feedback.
• Meta-Harness (Lee et al., 2026): Optimizes the code wrapping LLMs (harnesses) by giving a coding agent filesystem access to all prior candidates' code, traces, and scores. Same philosophy of rich feedback over scalar scores, but applied to LLM application code rather than system code.
• Trace (Cheng et al., 2024): The optimization framework this paper uses. Trace treats code generation as an optimization problem where execution traces serve as the optimization signal — generalizing autodiff to discrete program spaces.

LLMs for Systems

• KernelBench: Benchmarks LLMs on writing CUDA kernels. Complementary to this work: KernelBench tests low-level kernel generation, while this paper tests high-level mapping decisions.
• AI for chip design (Mirhoseini et al., 2021): Used RL for chip floorplanning — another mapping problem (logic blocks to chip locations). This paper shows generative optimization with rich feedback can be more efficient than RL for similar problems.
• TVM / Ansor (Zheng et al., 2020): Auto-tuning for tensor programs using learned cost models. Operates at a different level (operator scheduling) but shares the goal of automating performance optimization.

The DSL Pattern

The idea of creating a domain-specific interface for LLM agents appears across domains:

• DSPy (Khattab et al., 2023): A DSL for LLM programs that separates program structure from prompting details. Same principle of abstraction enabling optimization.
• LMQL, Guidance, SGLang: Various structured interfaces for constraining and directing LLM generation. All share the insight that constraining the LLM's output space improves reliability.