init commit
This commit is contained in:
Junjian Wang
@@ -0,0 +1,417 @@
|
||||
# Harness Engineering: The Complete Guide to Building Systems That Make AI Agents Actually Work (2026)
|
||||
|
||||
## Harness Engineering: The Complete Guide to Building Systems That Make AI Agents Actually Work
|
||||
|
||||
**March 2026** — If 2025 was the year AI agents proved they could write code, 2026 is the year we learned that **the agent isn't the hard part — the harness is.**
|
||||
|
||||
OpenAI's Codex team just built a production application with **over 1 million lines of code** where **zero lines were written by human hands**. The engineers didn't write code. They designed the system that let AI write code reliably. That system — the constraints, feedback loops, documentation, linters, and lifecycle management — is what the industry now calls a **harness**.
|
||||
|
||||
**Harness engineering** is the new discipline of designing these systems. And it's changing what it means to be a software engineer.
|
||||
|
||||
* * *
|
||||
|
||||
## What Is Harness Engineering?
|
||||
|
||||
### The Horse Metaphor
|
||||
|
||||
The term "harness" comes from horse tack — reins, saddle, bit — the complete set of equipment for channeling a powerful but unpredictable animal in the right direction. The metaphor is deliberate:
|
||||
|
||||
- The **horse** is the AI model — powerful, fast, but it doesn't know where to go on its own
|
||||
- The **harness** is the infrastructure — constraints, guardrails, feedback loops that channel the model's power productively
|
||||
- The **rider** is the human engineer — providing direction, not doing the running
|
||||
|
||||
Without a harness, an AI agent is a thoroughbred in an open field. Fast, impressive, and completely useless for getting anything done.
|
||||
|
||||
### The Formal Definition
|
||||
|
||||
**Harness engineering** is the design and implementation of systems that:
|
||||
|
||||
1. **Constrain** what an AI agent can do (architectural boundaries, dependency rules)
|
||||
2. **Inform** the agent about what it should do (context engineering, documentation)
|
||||
3. **Verify** that the agent did it correctly (testing, linting, CI validation)
|
||||
4. **Correct** the agent when it goes wrong (feedback loops, self-repair mechanisms)
|
||||
|
||||
Martin Fowler describes it as *"the tooling and practices we can use to keep AI agents in check"* — but it's more than just safety. A good harness makes agents **more capable**, not just more controlled.
|
||||
|
||||
* * *
|
||||
|
||||
## Why Harness Engineering Matters Now
|
||||
|
||||
### The Model Is Commodity. The Harness Is Moat.
|
||||
|
||||
Here's the uncomfortable truth the AI industry is confronting: **the underlying model matters less than the system around it.**
|
||||
|
||||
LangChain proved this definitively. Their coding agent went from **52.8% to 66.5%** on Terminal Bench 2.0 — jumping from **Top 30 to Top 5** — by changing nothing about the model. They only changed the harness:
|
||||
|
||||
Change
|
||||
|
||||
What They Did
|
||||
|
||||
Impact
|
||||
|
||||
Self-verification loop
|
||||
|
||||
Added pre-completion checklist middleware
|
||||
|
||||
Caught errors before submission
|
||||
|
||||
Context engineering
|
||||
|
||||
Mapped directory structures at startup
|
||||
|
||||
Agent understood codebase from the start
|
||||
|
||||
Loop detection
|
||||
|
||||
Tracked repeated file edits
|
||||
|
||||
Prevented "doom loops"
|
||||
|
||||
Reasoning sandwich
|
||||
|
||||
High reasoning for planning/verification, medium for implementation
|
||||
|
||||
Better quality within time budgets
|
||||
|
||||
**Same model. Different harness. Dramatically better results.**
|
||||
|
||||
### OpenAI's 1 Million Line Proof Point
|
||||
|
||||
OpenAI's experiment is the most compelling evidence yet:
|
||||
|
||||
- **5 months** of development
|
||||
- **1 million+ lines of code** in the final product
|
||||
- **Zero manually written lines** — every line was produced by Codex agents
|
||||
- **Built in ~1/10th the time** it would have taken humans
|
||||
- The product has **internal daily users and external alpha testers**
|
||||
- It **ships, deploys, breaks, and gets fixed** — all by agents within the harness
|
||||
|
||||
The engineers' job? Designing the harness. Specifying intent. Providing feedback. Not writing code.
|
||||
|
||||
* * *
|
||||
|
||||
## The Three Pillars of Harness Engineering
|
||||
|
||||
OpenAI's framework organizes harness engineering into three core categories:
|
||||
|
||||
### 1\. Context Engineering
|
||||
|
||||
Context engineering is about ensuring the agent has the right information at the right time.
|
||||
|
||||
**Static context:**
|
||||
|
||||
- Repository-local documentation (architecture specs, API contracts, style guides)
|
||||
- `AGENTS.md` or `CLAUDE.md` files that encode project-specific rules
|
||||
- Cross-linked design documents validated by linters
|
||||
|
||||
**Dynamic context:**
|
||||
|
||||
- Observability data (logs, metrics, traces) accessible to agents
|
||||
- Directory structure mapping at agent startup
|
||||
- CI/CD pipeline status and test results
|
||||
|
||||
**The critical rule:** From the agent's perspective, anything it can't access in-context doesn't exist. Knowledge in Google Docs, Slack threads, or people's heads is invisible to the system. **The repository must be the single source of truth.**
|
||||
|
||||
### 2\. Architectural Constraints
|
||||
|
||||
This is where harness engineering diverges most sharply from traditional AI prompting. Instead of telling the agent "write good code," you **mechanically enforce what good code looks like.**
|
||||
|
||||
**Dependency layering:**
|
||||
|
||||
```
|
||||
Types → Config → Repo → Service → Runtime → UI
|
||||
|
||||
```
|
||||
|
||||
Each layer can only import from layers to its left. This isn't a suggestion — it's enforced by structural tests and CI validation.
|
||||
|
||||
**Constraint enforcement tools:**
|
||||
|
||||
- **Deterministic linters** — Custom rules that flag violations automatically
|
||||
- **LLM-based auditors** — Agents that review other agents' code for architectural compliance
|
||||
- **Structural tests** — Like ArchUnit, but for AI-generated code
|
||||
- **Pre-commit hooks** — Automated checks before any code is committed
|
||||
|
||||
**Why constraints improve output:** Paradoxically, constraining the solution space makes agents **more productive**, not less. When an agent can generate anything, it wastes tokens exploring dead ends. When the harness defines clear boundaries, the agent converges faster on correct solutions.
|
||||
|
||||
### 3\. Entropy Management ("Garbage Collection")
|
||||
|
||||
This is the most underappreciated component. Over time, AI-generated codebases accumulate entropy — documentation drifts from reality, naming conventions diverge, dead code accumulates.
|
||||
|
||||
Harness engineering addresses this with **periodic cleanup agents:**
|
||||
|
||||
- **Documentation consistency agents** — Verify that docs match current code
|
||||
- **Constraint violation scanners** — Find code that slipped past earlier checks
|
||||
- **Pattern enforcement agents** — Identify and fix deviations from established patterns
|
||||
- **Dependency auditors** — Track and resolve circular or unnecessary dependencies
|
||||
|
||||
These agents run on schedules — daily, weekly, or triggered by specific events — keeping the codebase healthy for both human reviewers and future AI agents.
|
||||
|
||||
* * *
|
||||
|
||||
## Harness Engineering in Practice: How Teams Actually Do It
|
||||
|
||||
### The OpenAI Approach: Zero Human Code
|
||||
|
||||
OpenAI's team structure for harness engineering:
|
||||
|
||||
Role
|
||||
|
||||
Traditional
|
||||
|
||||
Harness Engineering
|
||||
|
||||
Writing code
|
||||
|
||||
Primary job
|
||||
|
||||
Never
|
||||
|
||||
Designing architecture
|
||||
|
||||
Part of the job
|
||||
|
||||
Primary job
|
||||
|
||||
Writing documentation
|
||||
|
||||
Afterthought
|
||||
|
||||
Critical infrastructure
|
||||
|
||||
Reviewing PRs
|
||||
|
||||
Code review
|
||||
|
||||
Reviewing agent output + harness effectiveness
|
||||
|
||||
Debugging
|
||||
|
||||
Reading code
|
||||
|
||||
Analyzing agent behavior patterns
|
||||
|
||||
Testing
|
||||
|
||||
Writing tests
|
||||
|
||||
Designing test strategies agents execute
|
||||
|
||||
### The Stripe Approach: Minions at Scale
|
||||
|
||||
Stripe's internal coding agents, called **Minions**, now produce **over 1,000 merged pull requests per week**:
|
||||
|
||||
1. Developer posts a task in Slack
|
||||
2. Minion writes the code
|
||||
3. Minion passes CI
|
||||
4. Minion opens a PR
|
||||
5. Human reviews and merges
|
||||
|
||||
No developer interaction between step 1 and step 5. The harness handles everything — test execution, CI validation, style compliance, and documentation updates.
|
||||
|
||||
### The LangChain Approach: Middleware-First
|
||||
|
||||
LangChain structures their harness as composable middleware layers:
|
||||
|
||||
```
|
||||
Agent Request
|
||||
→ LocalContextMiddleware (maps codebase)
|
||||
→ LoopDetectionMiddleware (prevents repetition)
|
||||
→ ReasoningSandwichMiddleware (optimizes compute)
|
||||
→ PreCompletionChecklistMiddleware (enforces verification)
|
||||
→ Agent Response
|
||||
|
||||
```
|
||||
|
||||
Each middleware layer adds a specific capability without modifying the core agent logic. This modular approach makes the harness testable and evolvable.
|
||||
|
||||
* * *
|
||||
|
||||
## Building Your First Harness: A Practical Framework
|
||||
|
||||
### Level 1: Basic Harness (Single Developer)
|
||||
|
||||
If you're using Claude Code, Cursor, or Codex for individual projects:
|
||||
|
||||
**What to set up:**
|
||||
|
||||
- `CLAUDE.md` or `.cursorrules` file with project conventions
|
||||
- Pre-commit hooks for linting and formatting
|
||||
- A test suite the agent can run to self-verify
|
||||
- Clear directory structure with consistent naming
|
||||
|
||||
**Time to set up:** 1-2 hours **Impact:** Prevents the most common agent mistakes
|
||||
|
||||
### Level 2: Team Harness (Small Team)
|
||||
|
||||
For teams of 3-10 developers sharing a codebase:
|
||||
|
||||
**Add to Level 1:**
|
||||
|
||||
- `AGENTS.md` with team-wide conventions
|
||||
- Architectural constraints enforced by CI
|
||||
- Shared prompt templates for common tasks
|
||||
- Documentation-as-code validated by linters
|
||||
- Code review checklists specifically for agent-generated PRs
|
||||
|
||||
**Time to set up:** 1-2 days **Impact:** Consistent agent behavior across the team
|
||||
|
||||
### Level 3: Production Harness (Engineering Organization)
|
||||
|
||||
For organizations running dozens of concurrent agents:
|
||||
|
||||
**Add to Level 2:**
|
||||
|
||||
- Custom middleware layers (loop detection, reasoning optimization)
|
||||
- Observability integration (agents read logs and metrics)
|
||||
- Entropy management agents on scheduled runs
|
||||
- Harness versioning and A/B testing
|
||||
- Agent performance monitoring dashboards
|
||||
- Escalation policies for when agents get stuck
|
||||
|
||||
**Time to set up:** 1-2 weeks **Impact:** Agents operate as autonomous contributors
|
||||
|
||||
* * *
|
||||
|
||||
## Common Harness Engineering Mistakes
|
||||
|
||||
### 1\. Over-Engineering the Control Flow
|
||||
|
||||
> *"If you over-engineer the control flow, the next model update will break your system."*
|
||||
|
||||
Models improve rapidly. Capabilities that required complex pipelines in 2024 are now handled by a single context-window prompt. Build your harness to be **rippable** — you should be able to remove "smart" logic when the model gets smart enough to not need it.
|
||||
|
||||
### 2\. Treating the Harness as Static
|
||||
|
||||
The harness needs to evolve with the model. When a new model release improves reasoning, your reasoning-optimization middleware might become counterproductive. Review and update harness components with every major model update.
|
||||
|
||||
### 3\. Ignoring the Documentation Layer
|
||||
|
||||
The most impactful harness improvement is often the simplest: **better documentation**. If your `AGENTS.md` is vague, your agent output will be vague. Invest in precise, machine-readable documentation that serves as the agent's ground truth.
|
||||
|
||||
### 4\. No Feedback Loop
|
||||
|
||||
A harness without feedback is a cage, not a guide. The agent needs to know when it's succeeding and when it's failing. Build in:
|
||||
|
||||
- Self-verification steps before task completion
|
||||
- Test execution as part of the agent workflow
|
||||
- Metrics on agent success rates by task type
|
||||
|
||||
### 5\. Human-Only Documentation
|
||||
|
||||
If your architectural decisions live in people's heads or in Confluence pages the agent can't access, the harness has a gap. **Everything the agent needs must be in the repository.**
|
||||
|
||||
* * *
|
||||
|
||||
## Harness Engineering vs. Related Concepts
|
||||
|
||||
Concept
|
||||
|
||||
Scope
|
||||
|
||||
Focus
|
||||
|
||||
**Prompt Engineering**
|
||||
|
||||
Single interaction
|
||||
|
||||
Crafting effective prompts
|
||||
|
||||
**Context Engineering**
|
||||
|
||||
Model context window
|
||||
|
||||
What information the model sees
|
||||
|
||||
**Harness Engineering**
|
||||
|
||||
Entire agent system
|
||||
|
||||
Environment, constraints, feedback, lifecycle
|
||||
|
||||
**Agent Engineering**
|
||||
|
||||
Agent architecture
|
||||
|
||||
Internal agent design and routing
|
||||
|
||||
**Platform Engineering**
|
||||
|
||||
Infrastructure
|
||||
|
||||
Deployment, scaling, operations
|
||||
|
||||
Harness engineering **includes** context engineering and draws from prompt engineering, but it operates at a higher level — it's about the complete system that makes agents reliable, not just the inputs to a single interaction.
|
||||
|
||||
* * *
|
||||
|
||||
## What This Means for Software Engineers
|
||||
|
||||
### The Job Is Changing
|
||||
|
||||
Harness engineering represents a genuine evolution in what software engineers do:
|
||||
|
||||
Before
|
||||
|
||||
After
|
||||
|
||||
Write code
|
||||
|
||||
Design environments where AI writes code
|
||||
|
||||
Debug code
|
||||
|
||||
Debug agent behavior
|
||||
|
||||
Review code
|
||||
|
||||
Review agent output + harness effectiveness
|
||||
|
||||
Write tests
|
||||
|
||||
Design test strategies
|
||||
|
||||
Maintain docs
|
||||
|
||||
Build documentation as machine-readable infrastructure
|
||||
|
||||
This doesn't mean engineers become less technical. If anything, harness engineering requires **deeper** architectural thinking — you're designing systems that must work without your constant intervention.
|
||||
|
||||
### The Skills That Matter
|
||||
|
||||
Based on what we've seen building AI-powered products at [NxCode](https://www.nxcode.io/):
|
||||
|
||||
1. **Systems thinking** — Understanding how constraints, feedback loops, and documentation interact
|
||||
2. **Architecture design** — Defining boundaries that are enforceable and productive
|
||||
3. **Specification writing** — Articulating intent precisely enough for agents to execute
|
||||
4. **Observability** — Building monitoring that reveals agent behavior patterns
|
||||
5. **Iteration speed** — Rapidly testing and refining harness configurations
|
||||
|
||||
### Our Experience: What Works in Practice
|
||||
|
||||
We've been building AI-powered web applications using multiple agent systems (Claude Code, Codex, Cursor). The patterns that have made the biggest difference for us:
|
||||
|
||||
- **Repository-first documentation**: Every architectural decision, naming convention, and deployment process is in the repo. Nothing lives in Slack or Google Docs.
|
||||
- **Incremental constraint building**: Start with basic linting, add architectural constraints as patterns emerge, don't try to design the perfect harness upfront.
|
||||
- **Agent-specific review checklists**: AI-generated code has different failure modes than human code. Our review process accounts for common agent patterns (over-abstraction, unnecessary error handling, documentation drift).
|
||||
- **Multi-provider harness design**: Our harness works with Claude, GPT, and Gemini models. Provider-agnostic design means we can switch models without rebuilding the entire system.
|
||||
|
||||
* * *
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. **Harness engineering is the new discipline** of designing systems that make AI agents reliable — constraints, feedback loops, documentation, and lifecycle management
|
||||
2. **The model is commodity; the harness is moat** — LangChain jumped from Top 30 to Top 5 on benchmarks by only changing the harness
|
||||
3. **OpenAI built 1M+ lines with zero human code** — proving harness engineering works at production scale
|
||||
4. **Three pillars**: Context engineering, architectural constraints, and entropy management
|
||||
5. **Start simple**: A good `AGENTS.md` and pre-commit hooks are more impactful than complex middleware
|
||||
6. **The engineer's job is evolving** — from writing code to designing environments where AI writes code
|
||||
7. **Build rippable harnesses** — over-engineering breaks when models improve; keep it adaptable
|
||||
|
||||
* * *
|
||||
|
||||
## Related Articles
|
||||
|
||||
- [Best AI for Coding in 2026: 10 Tools Ranked by Real-World Performance](https://www.nxcode.io/resources/news/best-ai-for-coding-2026-complete-ranking)
|
||||
- [OpenAI Frontier Guide: Enterprise AI Agent Platform for Building AI Coworkers (2026)](https://www.nxcode.io/resources/news/openai-frontier-enterprise-ai-agent-platform-guide-2026)
|
||||
- [Cursor Tutorial 2026: Learn AI Coding in 15 Minutes (Beginner Guide)](https://www.nxcode.io/resources/news/cursor-tutorial-beginners-2026)
|
||||
Reference in New Issue
Block a user