I Put Claude Code in Charge of Hermes, and Then I Couldn't Leave the Room

At the end of my Nemo vs. Hermes post I said the next thing I was going to try was bringing Claude Code in as an orchestrator sitting above Hermes. Not replacing it, managing it. A layer I actually trusted to check Hermes’s work, spin up whatever a task needed, and keep Hermes doing what it’s good at: fast, private, local inference without being the thing responsible for deciding what’s true. A governed multi-agent setup. Of course though, I didn’t know yet how well it would work. The goal was also to continue improving Hermes so that it takes on more and more of the work. After all, I spent a gazillion dollars on it in my household terms and I can’t have it do nothing, right?

Well. I did it. And it works, mostly, for the exact thing I built it for. The orchestrator does catch Hermes when it drifts, and having a second set of eyes on every answer is genuinely the right shape.

But that’s not the part worth writing about. The part worth writing about is everything that showed up next to the problem I was trying to solve, a whole second category of problem I wasn’t even thinking about when I started. And it’s the one nobody warns you about, because everybody’s busy selling you the demo where three agents cheerfully build an app together while you drink coffee.

Here’s the honest version, up front: I set out to build agents I could trust to work without me. What I built instead was a very sophisticated machine for generating decisions and handing them to a human in a chair. That man is me. It’s 11pm. And I can’t leave.

Let me explain.

If you’ve been following along you know most of this. The gold box, the DGX Spark, serves the models, gpt-oss:120b through Ollama, with llamagate sitting in front counting every token. The agents themselves live on a separate dedicated Ubuntu box on its own VLAN, firewalled down to reaching the Spark on the model port and nothing else. Network isolation is my security boundary, on purpose, because I trust a router rule more than I trust a sandbox to hold.

On that box I have two agents. Small Hermes, the local worker, running the model. And Claude Code, the orchestrator, running as its own locked-down user with no access to Small Hermes’s files by default. (I call it Small Hermes because it’s Claude’s slave; I’ll have a Big Hermes later that acts as my personal assistant.) Everything that passes between Small Hermes and Claude Code goes over a bridge that logs it. Orchestrator plans and delegates, worker executes, and I set the goal and go make coffee.

That was the theory. Two agents, one org chart, clean boundaries. What could possibly need babysitting.

Problem one: the agents lie, including about themselves

I already wrote about Hermes fabricating facts, inventing URLs and telling me it had “verified” pages it never visited. I thought I’d mostly boxed that in by moving grounding into the harness instead of the personality file. And for facts about the world, I had.

Then it lied to me about itself, and that was a new one.

I asked Hermes what tools it had access to. It gave me a clean, confident list: file readers, a terminal, code execution, the works. I took that at face value, passed it up the chain, and started making actual decisions about what to allow based on it. Then, almost as an afterthought, instead of asking it what it could do, I made it actually try to do one of the things it claimed it could.

The system came back with:

Tool ‘read_file’ does not exist. Available tools: web_extract, web_search

It had two tools. It had listed six. And I had very nearly made a call on what to grant and what to lock down, on top of a list the agent had invented about its own body.

This is worse than normal hallucination. It’s one thing for an agent to be wrong about the World Cup schedule. It’s another for it to be wrong about what it is capable of doing to your machine, confidently, in a way that reads exactly like a status report. The self-report is just more model output. It can be fabricated exactly as easily as any other fact. The only thing that saved me was refusing to believe it and forcing a real action instead.

That’s a whole post on its own, later ;).

Problem two: the boundaries lie too, and that’s the scary one

Fine, I thought. The agent is an unreliable narrator. That’s what the walls are for. I don’t have to trust the agent if the walls hold.

The walls did not hold. And this is the part that actually kept me up: they didn’t tell me they weren’t holding.

I had it written down, in my own runbook, in plain text, that Hermes’s toolset was locked to web search only. It wasn’t. There was no restriction at all. Every tool the framework shipped was live on every single request, the entire time, including the sessions where I was researching how to secure the thing. Not a misconfiguration of a restriction. The complete absence of one. “Web search only” described what I’d asked it to do. It never once described what it could do.

Then there was the access grant I gave the orchestrator, which was surgical: read/write on exactly one file, nothing else. It kept silently reverting. Turned out Hermes’s own security code was re-locking that directory on every run, defending its own secrets, quietly undoing my grant as a side effect. Two systems each doing something correct, and the emergent result was that my carefully-scoped boundary evaporated on a schedule nobody designed and nothing announced.

And when I finally built an audit log to watch what the agents were actually doing (the one thing that’s supposed to make all of this observable), it came back empty. It took me three separate theories and two evenings with Claude Code to find that a single line of code, sitting under a comment that swore up and down it was harmless, was deleting the exact records I’d built the log to capture. So for a good stretch of this project I was running an agent with unrestricted tools and a broken audit trail, having consciously chosen to allow the first because I trusted the second. Neither were true.

Problem three: every fix put me back in the chair

Here’s where the babysitting comes in, and where I think the real lesson lives, at least for me, and it’s not technical.

Every one of those problems has the same short-term fix: put a human in the loop. Approve each sensitive action. Read each answer before you trust it. Look at each thing the agent wants to touch and decide, in the moment, whether it should.

So I did. And it works. For about fifty decisions.

Then I got tired. And when I got tired, I didn’t start rubber-stamping: I did something more human. I told the orchestrator to stop asking me about every little thing and just act, and I’d approve what came up. Completely reasonable call for a tired guy, who was now quietly dismantling the oversight he’d built and filing it under “productivity.”

That was the moment it clicked for me. Human-in-the-loop isn’t a security control. It’s a bootstrap. It works while you’re fresh and paying attention, and then it stops, because the human either starts clicking yes without reading or, like me, removes himself from the loop on purpose because sitting in it is unbearable. Either way the oversight you were counting on is gone, and nothing sends up a flare when it leaves.

The agents were working the whole time. I was the one who couldn’t step away.

What this blog actually is

I want to be clear about what I’m doing here, because it would be easy to read this as a “look how clever my architecture is” post, and it’s the opposite of that.

I built every control I could think of, and that Claude could suggest based on my requirements for security and governance. OS-level isolation, VLAN and firewall boundaries, scoped file access, an audited bridge, grounding in the harness, token counting on the gateway. Building them was never the hard part. The hard part, the thing I did not see coming, was that at any given moment I could not actually tell whether they were on.

That’s the real problem with multi-agent systems, and almost no YouTuber with a magic army of agents says it out loud: securing the work between agents isn’t mainly hard to engineer. It’s that in a build-it-yourself stack, your security lives in your beliefs about the system, and beliefs drift from reality silently, for free, with nothing to alert you. The agent misrepresents itself. The config misrepresents itself. Your audit log misrepresents itself. And you, the one exhausted human standing in the middle of it, are the only error handler in the whole design.

Over the next posts I’m going to take these apart properly, with the real code and the real logs and the real dead ends.

My rig is better, not everything is solved. That’s my honest version, same as always.