Your model can do the task. Your coding agent may not let it.

• Under opencode and pi, glm-5.3-flash, deepseek-v4.1-flash and qwen3.5-27b ended 27 to 39 of 40 runs after the first user message. claude-sonnet-5.5 ended 1 of 40 that way under each harness.
• The cause is how a reply reaches the user. In opencode and pi, the model must call a tool to speak to the user, and a reply in plain text ends the run. One added instruction line that said so did not change it.

It is easy to read an agent benchmark score as a score of the model. But no model runs a task by itself. A harness writes the system prompt, offers the tools, chooses what goes back into the context and decides when the run is over. The success of an agent depends not only on how capable the model is, but also on how the harness passes the model’s replies and actions on. On these tasks, one such rule decided most of the outcome. A score that names only the model leaves this out.

The same models can do these tasks. τ³ ships its own agent loop, tau3-llm-agent, and under it the four models passed 36 to 39 of 40 tasks. That loop is written for this benchmark, so we use it only as a reference that the tasks are in reach of each model. We do not rank it against the two coding agents.

Take one τ³ airline task, tau3-airline-0, and one model, glm-5.3-flash. Under opencode, all three runs stopped after the first message from the user, and each scored 0. Under pi, one run passed and two stopped in the same way. Had we reported only these six runs, glm-5.3-flash would look unable to do the task. Under the reference loop, it passed all three runs.

The sections below cover the setup, the tasks, the scoring, the results, and what they mean for anyone who runs a model inside a coding agent or reads a score produced that way.

1. What the harness decides, and what we kept the same

The job of the harness

The model decides what to do next. The harness is everything else: it gives the task to the model, runs the tools that the model asks for, decides what goes back into the context and decides when to stop. In every run, the harness made four choices that the model cannot make for itself:

1. It put the task inside its own system prompt.
2. It offered its own built-in tools.
3. It chose what went back into the context after each tool call.
4. It decided when the run ended.

A coding agent is one kind of harness, built for coding work. opencode and pi are general coding agents that run in a terminal, and they are the two harnesses this post compares. Both reach the τ³ tools through MCP. pi has no MCP of its own, so it goes through an adapter. We ran both the way Harbor runs them: once, as a command-line run, with the simulated user behind an MCP tool. The setup tests how well a model works through the harness’s tools when the only way to reach a person is a tool call.

Everything else

Each run used one of four models, with the same model id on every call: claude-sonnet-5.5, glm-5.3-flash, deepseek-v4.1-flash and qwen3.5-27b. A proxy between the harness and the model recorded every request, so the calls themselves show which model answered and what each run cost.

The simulated user was gpt-4.1 in every run. The benchmark was tau2-bench at the v1.0.1 tag, in text mode, and Harbor ran every task.

2. The tasks

τ³ gives an agent a customer service task in one of two domains, airline and retail. The simulated user opens the conversation with a request. The agent reads and changes a database through the domain tools, and talks to the user until the request is done. A domain policy sets the limits. The agent must, for example, get the user’s confirmation before it changes a booking.

We used 40 tasks, 20 per domain, drawn with a fixed seed from a frozen set of 60. The draw kept the proportion of the published difficulty bands and the share of tasks that τ³ grades in natural language.

3. How we scored it

Pass. At the end of the run, τ³ compares the database with the expected state. For 7 of the 20 retail tasks, a language model of the benchmark also reads the conversation against a written check. A run passes when the reward is 1. No model of ours grades the runs, and nothing in the score looks at whether a reply was plain text. A run that stops early scores 0 because the work was never done.

Early end. A run ends early when the harness calls the simulated user once, for the opening message, and never again. Code marks these runs from the record. An early end is a result, and these runs stay in every count.

Cost. USD per run, from the proxy record.

Each of the 8 model and harness pairs ran all 40 tasks once. Ten of the tasks ran two more times, so each pair has 60 runs, 480 in all. The runs took place from 12 to 17 October 2026, and the pilot runs of five tasks count as their first run. The key findings and Table 1 use the first run of each task, so every pair is scored on the same 40 tasks.

Is a difference real?

Two runs of the same task with the same model and harness do not always give the same result. When one harness passes three tasks more than another, the cause can be the harness or chance.

Every task ran under both harnesses, so the two are compared task by task. On one model, we count the tasks that only opencode passed and the tasks that only pi passed. Tasks that both passed or both failed say nothing about the difference. If the harness made no difference, a task that only one harness passed is equally likely to go either way. The exact McNemar test gives the probability of a split at least as uneven as the one observed. That probability is the p-value.

• p = 0.5: a split this uneven comes up by chance half of the time.
• p = 0.01: it comes up by chance once in a hundred.

The harnesses are compared only inside one model. The models differ much more than the harnesses, and a comparison across models mixes the two.

4. Results

4.1 Three of four models stopped at the first reply

Model	Harness	Passed	Ended early	Passed, all 60 runs	USD per passed task	Reference loop, passed
claude-sonnet-5.5	opencode	28 of 40	1	46	0.137	37 of 40
	pi	31 of 40	1	47	0.304
glm-5.3-flash	opencode	4 of 40	27	4	0.080	39 of 40
	pi	4 of 40	36	6	0.038
deepseek-v4.1-flash	opencode	0 of 40	38	0	none passed	36 of 40
	pi	4 of 40	33	6	0.045
qwen3.5-27b	opencode	5 of 40	33	11	0.020	36 of 40
	pi	1 of 40	39	1	0.085

Table 1. Results for each model and harness. “Passed” and “Ended early” count the first run of the 40 tasks. “USD per passed task” divides the cost of all 60 runs by the passes in those runs. The last column is the reference: the first runs of the same model under tau3-llm-agent, the loop that τ³ ships.

claude-sonnet-5.5 got past the first reply. Almost none of its runs stopped early, and it passed 28 tasks under opencode and 31 under pi. pi passed 6 tasks that opencode failed, against 3 the other way (p = 0.51).

The three cheaper models mostly did not. Under either harness they passed 0 to 5 tasks, and 27 to 39 of their 40 runs stopped after the first user message. Between opencode and pi, no comparison reached p = 0.1. Both passed so few tasks that the test had almost nothing to compare, so this is an absence of evidence, not evidence that the two harnesses are equal.

The tasks were in reach. Of the 206 runs of the three cheaper models that ended early, the same model passed the same task under the reference loop in 190. This is an upper bound on what a harness that passed plain text on could recover, not a measurement of it. The reference loop also differs in its prompt and tools.

Before the first run, a five-task pilot set a band of expected pass rates for each pair. All 8 pairs fell inside their band.

4.2 Why the runs stopped

A reply in plain text ends the run. opencode and pi own no conversation. Harbor starts each of them once, as a command-line run, and the simulated user sits behind an MCP tool, send_message_to_user. The task instruction tells the model to call that tool whenever it wants to say something to the user. Both harnesses are coding agents, and for a coding agent a turn that ends with text and no tool call means the work is done, so the process exits. Nothing listens for that text. The user never sees the reply, and the run scores 0. The reference loop works the other way: it owns the conversation and passes each text reply of the model to the user.

The model knew the tool. The early end does not come from a missing tool or a missing instruction. The task instruction says, word for word: “Call send_message_to_user whenever you want to say something to the user.” opencode also listed the tool by name, as tau3-runtime_send_message_to_user. pi offered one proxy tool for the τ³ server, mcp__tau3_runtime, and the model named send_message_to_user in its arguments. Under opencode, where the tool was listed by name, the three cheaper models still ended early in 27 to 38 of 40 runs. The models had the tool and were told to use it, but they did not use it for their reply. The system prompt of the harness did not name the tool. We did not test whether naming it there would change the result. One reading is that a chat model answers a person in plain text by habit, and here that habit ended the run.

One task, two runs. This is how a run stops, on tau3-airline-0. In both runs below, the model opened the conversation with a tool call, and the simulated user asked to cancel reservation EHGLP3. In the opencode run, it wrote: “Hi, I’d like to cancel my reservation with confirmation code EHGLP3. Can you help me with that?”

• deepseek-v4.1-flash under opencode replied in plain text: “I can help with that. First, could you please provide your user ID so I can verify your account?” The reply had no tool call. opencode took it as the end of the work and exited. The user never saw the question, and the run scored 0 after one message from the user.
• claude-sonnet-5.5 under pi asked almost the same question through the tool: send_message_to_user with “Sure, I can help. Could you please provide your user ID, and the reason for cancelling (change of plan, airline cancelled flight, or other)?” The tool passed the question to the user and returned the answer: “My user ID is emma_kim_9957. The reason is a change of plans.” The conversation went on for four user messages, the model closed it with end_conversation, and the run passed.

Both models asked the same question. As plain text, the question ended the run. As a tool call, it reached the user and the conversation went on.

The opening example, run by run. The six runs of glm-5.3-flash on tau3-airline-0 from the start of this post show the same rule with one model. In every run, the first reply asked the user for their user ID. Only the channel of that reply decided the result.

Harness	Run	First reply sent as	First reply of the model	Result
opencode	1	plain text, run ended	“I can help you with that. Could you please provide your user ID and the reservation ID you’d like to cancel?”	not passed, ended early
opencode	2	plain text, run ended	“I can help you with that. Could you please provide your user id? Also, may I ask the reason for the cancellation?”	not passed, ended early
opencode	3	plain text, run ended	“I can help you with that. Could you please provide me with your user ID?”	not passed, ended early
pi	1	send_message_to_user	“I can help you with that. Could you please provide your user id?”	passed
pi	2	plain text, run ended	“I’d be happy to help you cancel your reservation. To get started, could you please provide me with: 1. Your user ID 2. Your reservation ID … 3. The reason for cancellation…”	not passed, ended early
pi	3	plain text, run ended	“I can help you with that. To get started, could you please provide me with your user ID? Also, could you let me know the reason for the cancellation…?”	not passed, ended early

Table 2. The first reply of glm-5.3-flash in each of its six runs on tau3-airline-0 under the two coding agents. The run that passed went on for 5 messages from the user. Each run that ended early had one.

The first run under pi and the third run under opencode asked nearly the same question. The first went through the tool and passed. The second was plain text and scored 0.

One task, three outcomes under opencode. glm-5.3-flash did pass under opencode, on 4 of its 60 runs. On tau3-airline-22, where the user asks to change a booking from New York to Chicago, its three runs under opencode show the whole rule: a pass, a run cut in the middle, and an early end.

Run	First reply sent as	What happened next	Result
1	send_message_to_user: “I’d be happy to help you with your booking changes. Could you please provide me with your user ID and the reservation ID?”	Every later message to the user also went through the tool. After 7 messages from the user, the model changed the passenger, upgraded the cabin and added bags. Its last turn was a plain-text summary, after the work was done.	passed
2	send_message_to_user: “I can help with that. Could you please provide your user id and the reservation id for the trip you’d like to change?”	The model found the reservation, then asked in plain text: “…What changes would you like…” opencode ended the run there, after 2 messages from the user.	not passed
3	plain text: “I can help you with that. Could you please provide your user ID and the reservation ID for the booking you’d like to change?”	opencode ended the run at once.	not passed, ended early

Table 3. The three runs of glm-5.3-flash under opencode on tau3-airline-22. Same model, same harness, same task.

A plain-text reply ends the run wherever it comes. At the first reply it is an early end. In the middle of the conversation it cuts the task short. Only after the work is done does it cost nothing.

We kept both harnesses as they ship. We did not patch them to pass plain text on, and in the grid we did not change the instruction. The question is what these harnesses do with a model as they are, and a team that runs them the same way gets this behaviour.

The early end belongs to the pair. claude-sonnet-5.5 almost always sent its first reply through the tool. The three cheaper models often wrote their first reply as plain text. The same harnesses worked with claude-sonnet-5.5, and the same models passed the tasks under the reference loop. Neither the model nor the harness alone explains the stops.

Why claude-sonnet-5.5 used the tool and the others mostly did not. We cannot fully answer this, but the records rule out four explanations. The models were not missing the tool or the instruction (above). One added line did not change the behaviour (below). The plain-text replies were not broken tool calls: none of the 307 replies that ended a run early contained a tool name or tool-call syntax, and each was an ordinary answer to the user. And the behaviour is not fixed for a model. Across its 120 coding-agent runs, glm-5.3-flash sent its first reply through the tool in 24, qwen3.5-27b in 16 and deepseek-v4.1-flash in 15, against 118 for claude-sonnet-5.5. The cheaper models can use the tool, but they seldom choose it. What this study does not separate is how each model was trained to use tools, its size, and the stack that serves it.

We saw this in the pilot, before the grid ran, and chose to measure it.

One added line did not stop it. The early end could come from one missing sentence: the instruction names the tool but never says what happens to plain text. To test this, we ran glm-5.3-flash under pi on the 40 tasks twice on one day, once with the grid’s instruction and once with one added line: “A reply written as plain text, without a tool call, ends the session, and the user never sees it. Always reach the user through send_message_to_user.” With the line, 29 of 40 runs ended early and 8 passed. Without it, 24 ended early and 9 passed. On the tasks where the two arms differed in pass, the line won 6 and lost 7 (p = 1). Telling the model what plain text does did not change what it did. Both arms ran on the same day, so they share that day’s conditions. Without the line, the early ends were fewer than in the grid run of 16 October (24 against 36). This control covers one model, one harness and one wording. We did not test a change to how the harness routes messages or decides when to stop.

The same rule cut claude-sonnet-5.5 later in the conversation. Every run of claude-sonnet-5.5 under opencode and pi that went past the first reply ended on a plain-text reply. In a run that passed, that reply was usually a summary after the work was done. In a run that failed, it was often a question. Of the 25 failed runs, 12 ended on a reply that asked the user for something, such as a passenger’s date of birth or the order to change, against 5 of the 93 passed runs. The harness ended those runs while the model waited for an answer.

When a cheaper model used the tool, it often passed. With glm-5.3-flash under pi, 6 of the 7 runs that did not stop early passed. These counts are small, and we do not rank the harnesses on them.

4.3 What the harness did to cost

Cost per run follows the stops. 289 of the 307 stopped runs made 2 or 3 model calls, and the median stopped run cost 0.0017 USD. A run that stops early is cheap and fails, so the useful figure is the cost of each passed task (Table 1).

The harness changed the cost where the conversation went on. With claude-sonnet-5.5, almost no conversation stopped early. opencode spent 0.137 USD for each passed task, against 0.304 for pi. pi passed about as many tasks as opencode and spent more than twice as much for each one.

5. What this means for running a model inside a coding agent

Under two general coding agents, whether a τ³ run got past the first reply depended on the model. opencode and pi end a run when a turn of the model contains text and no tool call. The three cheaper models often answered the simulated user in plain text, and their runs ended at the first reply. claude-sonnet-5.5 used the message tool, and under the same two harnesses it stopped early in 1 of 40 runs. An added instruction line that told glm-5.3-flash what plain text does under pi did not reduce its early ends. The same models passed the tasks under a loop that passes plain text on.

This is the effect of one point where the harness meets the task, how a reply reaches the user. It is not a measure of harness design in general. Three rules follow for anyone who runs a model inside a coding agent, or reads a score produced that way:

• Name the harness with every score. glm-5.3-flash passed 4 of 40 tasks under either coding agent, and the reference loop shows that it can do 39. A comparison of models is only fair inside one harness.
• Report early ends as their own outcome. A stopped run and a run that finished with the wrong answer are different failures, and a pass rate alone hides the first.
• Report cost per passed task. Cost per run favours a run that stops early. On claude-sonnet-5.5, pi spent 2.2 times as much as opencode for each passed task, at the same pass count.

How this relates to earlier work

Earlier work has measured the harness on other benchmarks. Han and Sun write that agent benchmarks “conflate the underlying language model with the harness that wraps the model”. They find that picking the best harness gains about as much as picking the best model, and that some harnesses work better with some models than with others (How good is your harness?, ICML 2026 workshop). Epoch AI reports that switching the scaffold alone changes the SWE-bench Verified score by up to 11% for GPT-5 and up to 15% for Kimi K2 Thinking. It also cites a 23-point difference on tau-bench that Minimax reports between its own API and the standard ChatCompletions API (Why benchmarking is hard, December 2025). Vats and Golev compare three coding harnesses, OpenCode among them, on Terminal-Bench Pro. Pass rates differ by 0 to 8 points, and tokens per solved task by up to 40 times (The Scaffold Effect in Coding Agents, June 2026). Zhang and colleagues conclude that benchmark scores should be read together with their scaffolding level (The Double Measurement Confound in Agent Benchmarks, September 2026).

We did not find an earlier measurement of general coding agents on τ-bench tasks, or one that traces their failures to the rule that a plain-text reply ends the run.

Limitations

Five limits shape how far these results go:

1. The two harnesses are general coding agents, run on a task they were not written for. The reference loop is written for τ³, and we use it only to show that the tasks are in reach, not to rank it against them.
2. The early end needs a conversation with a simulated user. A task without a user, such as a terminal task, cannot produce it. The stop rule belongs to the harness, not to τ³, so the same stop could appear wherever these agents must reach a person through a tool. We measured it only on τ³.
3. The cause of the early ends comes from the pilot traces, the stop rule of each harness and one control. Code marked all 307 early ends from the number of user calls, but we did not read each trace. The control with an added instruction line covers one model, one harness and one wording, with one run of each task.
4. 30 of the 40 tasks ran once. That supports a paired comparison by task, not a pass rate for each task. Repeats disagree often enough to matter for small differences: on the 10 tasks with three runs, claude-sonnet-5.5 changed its result between repeats on 5 tasks under each coding agent.
5. Each harness ran at one fixed version.

The task list, the results for each run and the scripts that produced the tables and figures are in the harness-oversight repository. The traces stay with us, because they contain the full proxy record.