Benchmarking Agentic Engineering for Production Inference Serving
SWE-Serve
SWE-Serve turns real SGLang engineering work into coding tasks covering model integration, public APIs, cache systems, and GPU kernels. Each task includes a verifier that checks whether the solution produces the required behavior.
SWE-Serve turns recent SGLang engineering work into coding tasks. Before including a task, we run its tests against both the unchanged repository (no-op) and the frozen reference solution (oracle). The no-op must fail every new-behavior test while passing the regression tests, and the oracle must pass every test. Human reviewers inspect the instruction, tests, and results before including the task.
During evaluation, the agent receives the task instruction and starting repository. It can access the task environment, the model endpoint, and Hugging Face for model artifacts, but not the public web or upstream repositories. Hidden functional and regression tests score what its patch does, not whether its code matches the reference solution.
01
Task construction
How benchmark authors build and validate each task.
01
Production SGLang changes
We select real inference-system work from release notes, roadmap issues, and maintainer suggestions.
02
Build the task
Benchmark authors, assisted by coding agents, prepare the instruction, a reproducible environment, a reference solution, and tests that check the requested behavior.
03
Qualification runs
On the task's assigned hardware, the no-op must fail every new-behavior test while passing the regression tests; the oracle must pass every test.
no-op: no new tests passoracle: all tests pass
04
Review and decision
Agent-assisted adversarial probing challenges the instruction and tests; human reviewers inspect the qualification results and make the final inclusion decision.
02
Evaluation trial
How one agent attempt is run and scored.
01
Pinned task environment
Each attempt starts from its task's pinned SGLang revision. The repository and verifier run in that task's CPU-only or single-H100 environment; model inference is separate.
02
Closed-book run
mini-SWE-agent receives the instruction and starting repository and can inspect, edit, build, and test locally. It can access the task environment, the model endpoint, and Hugging Face for model artifacts, but not the public web or upstream repositories.
public web blocked
03
Hidden functional and regression tests
Hidden functional tests check the requested behavior; regression tests check selected existing behaviors that should remain unchanged. They do not compare the patch with the reference solution.
04
Result
Pass or fail is recorded with cost, output tokens, agent steps, and total wall time.
Pass/fail reflects the verifier only; it does not assess code quality or merge readiness.
Consistent setup. Same mini-SWE-agent harness and 53 tasks, repeated three times. Scores apply only to this setup.
Pass@1. We divide verifier passes by 159 attempts per configuration. Agent-caused missing scores count as failures; ± is the 95% confidence interval across three runs.
Cost. API costs use fixed prices; open-weight model costs are hosted-rate estimates, not GPU operating costs.
