CUGA and Hermes: the model is only part of the agent

A harness takes over where the model hands off an action: it exposes tools, prepares context, executes requests and routes results or failures back. An IBM Research article published on November 3, 2026 holds the model fixed to examine what changes when that execution system changes. I would treat that boundary as part of the agent's architecture when investigating a failed task.

The CUGA team’s comparison runs CUGA, Hermes and Deep Agents with GLM-5.3-flash on AppWorld. Holding the model constant makes the experiment more informative than a comparison in which every component changes. It does not make the competing configurations identical, or establish a universal ranking of agent products.

What the reported result covers

AppWorld contains tasks involving linked actions across applications. The IBM authors describe it as a proxy for some capabilities needed in enterprise automation: understanding available APIs, retrieving information and changing application state. Consumer application tasks can exercise those capabilities without reproducing the permissions, exceptions and business consequences of a production deployment.

IBM reports 90 tasks across 30 scenarios, with one attempt per task, evaluated through Harbor. Its task completion figures are 81.1% for CUGA, 46.7% for Hermes and 30.0% for Deep Agents. These are source-reported findings from IBM’s experiment. We have not reproduced the runs, and the figures should not travel without their benchmark and setup attached.

The article also considers whether a harness completes every variation within a scenario. That distinction matters to buyers: completing a familiar example and handling related variations reliably are different properties. An isolated success can conceal a brittle path through tools. Equally, a difficult benchmark failure does not describe every useful feature an assistant offers during daily work.

A shared model is not a shared setup

CUGA used task decomposition, checks before execution and automatic reflection. Hermes ran with the Harbor integration’s default configuration and MCP tools. Deep Agents used MCP tools with additional middleware for tool selection, context summaries, task tracking, retries and other execution controls. Those choices are part of the comparison, rather than background details that can be ignored.

The authors explicitly limit their conclusions. The sample comes from a particular benchmark, the configurations differ, and the runs did not measure policy enforcement. The publisher is also the team behind one of the evaluated systems. That relationship does not make the experiment useless; it makes careful attribution and close reading more important.

Our editorial reading is that orchestration can materially affect what a fixed model accomplishes. A much broader claim—that CUGA is the best agent for any workload—would exceed the evidence. The study does not compare every deployment pattern, and it does not test all the properties that could motivate someone to choose Hermes or build with Deep Agents.

Compare the product you actually need

The Hermes repository emphasizes an assistant that develops procedures, retains useful knowledge and works through different communication channels. Those are continuity features. A closed benchmark task cannot fully represent the value of an assistant used across sessions, just as persistent memory does not by itself establish superior execution across business APIs.

LangGraph’s documentation describes a lower-level orchestration runtime for stateful, long-running agents. It supports workflows combining deterministic steps with model decisions. The behavior of a product built on that runtime depends on what its developers assemble. Deep Agents supplies a harness on top of that foundation; the foundation and the finished assistant are different levels of the stack.

This makes the purchasing question concrete. Are you selecting a persistent personal assistant, an engine for business operations across applications, or infrastructure for developing your own workflow? Shared terminology can obscure different requirements. A product can be appropriate for one of those jobs without leading a benchmark designed around another.

Turning a benchmark into a useful evaluation

We recommend collecting representative tasks from the intended operation and recording the complete configuration alongside results. Keep the model constant where practical. Include incomplete tool responses, missing permissions and actions that require checking existing state before making a change. These are proposed evaluation steps, not experiments we performed for this article.

Task completion records the end state. It does not record whether every authorization rule was respected, and IBM explicitly left policy enforcement outside this evaluation. The comparison makes the harness's decisions visible alongside the model's output. The choice still open is which execution and recovery behavior fits the work, and how that system will enforce the policies the study did not test.