Shared Memory Across Agents Is an Identity Problem: The Story of the Badge and the Filing Room
What a journey to shared memory taught me about agent identity.
I’ve written chapters on the individual pieces: the Spark, llamagate, Hermes and grounding, Claude Code as orchestrator. This one is two things at once. It’s the current-state map of the whole thing, for anyone trying to run a multi-agent setup on a single computer, and it’s the story of the one feature I wanted most and found hardest to get.
That feature is shared memory. And the thing I did not expect is that it isn’t a storage problem at all. It’s an identity problem. Two of them, actually, which is the part nobody tells you and the most transferable lesson I’ve hit yet.
For reference, this is the current architecture of my lab:
Everything here runs on one Ubuntu box on its own VLAN, plus a DGX Spark. Read it top to bottom: trust decreases as you go up, and everything Jarvis wants has to travel down through a seam. Four things worth calling out if you’re building something like this:
The Spark isn’t where the agents live. All the agent processes run on agents-machine. The Spark is a separate physical box serving model inference over the network through llamagate. Agents here, tokens there.
Three OS users, ranked by exposure. Jarvis is internet-facing (it parses Telegram), so it gets the least. claude-orch owns the seams and the memory store. Hermes runs the local worker. On one machine, OS users are the isolation. They cost nothing, but it’s only until I have a real enterprise rig with Kubernetes clusters.
Two MCP servers, doing different jobs. ijfw-memory is the memory layer. hermes-bridge is a small FastMCP server exposing one tool, dispatch_to_hermes, which hands work to the local worker over an audited shim and injects relevant fleet memory into every dispatch on the way through. Both are registered at user scope so every session on the box gets them.
Nothing shares a filesystem. Four localhost services, each with a narrow contract, each writing a JSONL audit row per call. Jarvis has zero filesystem access to the memory store. When it needs a memory it asks a service that reads on its behalf and logs the query.
That last one is where this post actually starts.
Why I wanted shared memory
Before this, every agent in my house was an amnesiac stranger.
I’d make a decision in a Claude Code session on my laptop: where something should live, why we picked one approach over another, what we’d already ruled out. Then I’d open a session the next day and it knew nothing. Then I’d ask Jarvis on my phone and it knew nothing. Then I’d dispatch a task to Hermes and it knew nothing.
So in my lab, I was the memory. Every fact that needed to cross from one agent to another crossed through me, retyped or copy-pasted, usually badly. I’d become the transport layer in my own architecture, which is exactly the thing I built the architecture to stop doing.
The specific thing I wanted was simple to state: a fact learned by any agent should be available to every agent, without me carrying it. Decide something in a laptop session, ask Jarvis about it from the car an hour later, get the right answer (important, as I have PTO coming :)).
That’s it. That’s the whole feature.
So I picked IJFW as the memory layer. It was recommended to me by a colleague, Craig Richardson, and it aims to be the “one shared brain, and a full operating layer, for every AI coding tool you use,” so your tools “remember across sessions, plan before they build, check each other’s work for hallucinations, pull in the right specialist for the job, and quietly cut your token bill. All on your machine.” Sounds perfect for my use case. I haven’t learned everything it does yet, but the timing was almost a miracle. Craig suggested it exactly when I was sitting there consulting with Claude about how to go about this. Fits like a glove.
What I didn’t realize while working on it is that it scopes storage per project, and a project is a directory path.
That reads like an implementation detail. It’s the single most consequential decision in the system, and if you don’t make it deliberately it gets made for you.
Here’s what that means in practice. I stored a fact in one session. I then spawned a genuinely separate process, different directory, different PID, zero shared context, and asked it to recall. Nothing. Not because sharing was broken. Because the two processes had resolved to two different scopes and each got its own private diary. Same machine, same store, same tool. Just a polite empty result.
Declared, not derived
The directory thing is IJFW’s design choice, and you could fairly call it a bad one. But don’t stop at “bad tool,” because the shape underneath is universal.
Every shared-state system has a scoping key: the thing that decides which slice of the store you’re actually talking to. A namespace, a tenant ID, a collection, a partition key, a thread ID, a key prefix. There is no memory layer without one. The question that matters isn’t what the key is. It’s whether the key is declared or derived.
Declared. You pass it explicitly and something validates it. A namespace in the API call, a tenant claim in a signed token, a workload identity from an attested runtime. Get it wrong and you get an error, or obviously wrong data.
Derived. It’s inferred from ambient runtime context. The directory you started in, the pod you landed on, the repo you happen to be inside. Get it wrong and you get an empty result that looks exactly like “nothing stored yet.”
That’s the actual rule, and it holds regardless of tooling:
Derived scoping fails silently. Declared scoping fails loudly.
I spent two weeks inside the silent one. So: “shared memory” never means “same store.” It means “same scoping key.” Before you write integration code, or ask your favorite agent to write it for you, answer this. What single scope must every agent resolve to, what binds them to it, and what enforces it? If the answer is that it’s derived from ambient context, you don’t have shared memory. You have a coincidence. And you’ll find out when an agent tells you it has no memories and is technically correct, because it’s been writing to a diary nobody else can see.
The fix in my case was to pin the scope explicitly and decouple it from physical location. An environment variable, IJFW_PROJECT_DIR=/srv/fleet/shared_mem, set everywhere so every agent addresses one store no matter where it runs. That’s a declared key. It’s just declared with the weakest possible primitive, which brings me to the part I got wrong in my own head for a while.
The badge and the filing room
When I first fixed this I described it to myself as “pinning identity.” That was sloppy, and untangling it is the thing I’d most want to hand to anyone building this.
There are two identities in play, and tools like this one smash them together.
Caller identity: “who am I?” Agent A versus agent B. Distinct per agent. This is the thing you authenticate, authorize, and audit against.
Scope identity: “which slice of the store am I addressing?” The namespace, the partition, the drawer. Shared by everyone who’s supposed to see the same data.
IJFW_PROJECT_DIR is the second one only. It isn’t “who is this agent,” it’s “which drawer are we all filing into.” Nobody in my lab is sharing an identity.
Think of it as a badge and a filing room. Everyone in the office has their own badge. They all walk into the same filing room. The badge is not the room. IJFW’s real flaw isn’t that it uses directories. It’s that it has no badge reader at all, and it works out which room you’re in by looking at where your feet happen to be standing.
Once you separate them, my setup makes sense, and so does why it isn’t an enterprise design:
| My lab today | What enterprise needs | |
|---|---|---|
| Caller identity | OS users: claude-orch, jarvis, hermes. Distinct per agent | A certificate or workload ID per agent, issued by a shared CA |
| Scope | IJFW_PROJECT_DIR pointing at one shared folder, same for all | A declared namespace or tenant ID, same for all |
| Enforcement | Filesystem ownership, mode 750, plus a mediated seam | A policy engine evaluating caller identity against the requested scope |
| Audit | JSONL row per seam call | Every read and write an access event tied to the caller’s ID |
So: distinct identities, common scope, something in between deciding who gets in. My version mimics an enterprise design with duct tape, OS permissions standing in for a policy engine, and that’s fine precisely because it’s one box that I own.
Three things make the env var not enterprise-grade, and none of them are “it’s shared.”
It’s asserted, not attested. Anyone who can set an environment variable can claim any scope. No issuer, no signature, no expiry, nothing to check it against. A certificate is issued to you and cryptographically verifiable. Same concept, upgraded from a sticky note to a passport.
It travels in ambient context, so it’s fragile. That’s the entire mess in the next section. A credential carried in the process environment gets dropped by any launch path that doesn’t propagate it.
Nothing checks it against the caller. Filesystem ownership is doing all my authorization work: coarse, binary, filesystem-shaped. Fine for three OS users on one box. Useless the moment you want “the finance agent may read scope X but only write scope Y.”
Scoping is not authorization
That last point deserves its own heading, because it’s the one I expect the industry to get wrong at scale.
A namespace is a filing decision. It answers “where do I put this.” It does not answer “are you allowed to read this.” Nothing in a vector-DB namespace or a Redis key prefix authenticates the caller. If you can name the namespace, you can read it. The isolation is a convention rather than an enforcement.
I got this right in my lab partly by accident. My scoping key is IJFW_PROJECT_DIR, but my authorization is filesystem ownership (claude-orch, mode 750) plus a mediated seam that audits every read. Two independent mechanisms. Jarvis knowing the path buys it exactly nothing, because knowing the path was never what granted access.
Keep those two separate and make sure both exist. The failure I’d bet money on seeing in the next couple of years is a company pointing every agent at one memory namespace, calling it “shared context,” and later discovering the support bot can recall what the finance agent wrote, with no logs to reconstruct it, because a read through a namespace isn’t an access event.
Once every agent has a real identity, you stop needing one memory. You get a shared org scope, a team scope, a per-agent private scope, maybe a per-user scope, with policy deciding which identities reach which. The shared store becomes one namespace among several that an identity is entitled to, rather than the single global bucket.
Which is where this gets harder, not easier
Everything in my lab currently is one machine, where I can lean on filesystem ownership as my enforcement layer. That’s a luxury with an expiry date.
Once this becomes a distributed system, with pods, nodes and workloads that reschedule, derived scoping becomes a certainty. Ambient context is unstable by construction: a pod gets a new name, a new IP, a new node every time it moves. Anything inferred from where a process happens to be running will drift. And the layers multiply. A sidecar reading different config than the main container. An init container that runs before identity is injected. A scheduled job carrying a different service account than the deployment it belongs to. My five hook scripts that ignored the pin while the API respected it are the small version of that problem.
More importantly, the blast radius changes character. On one box, a scoping mistake means an agent can’t find a fact. On a shared cluster, the same mistake means one agent reads another’s memory while both behave “correctly.”
So the direction I’m heading is to keep the shape and replace the primitives. The scope stays declared, but as a real namespace rather than a folder path. The caller identity graduates from an OS user to a certificate the workload is issued and attested for, with every agent getting its own, all trusting the same authority. chmod 750 becomes a policy decision point that evaluates that certificate against the scope being requested. And every read becomes an access event that lands in an audit trail, instead of a lookup nobody logged.
That’s a lot of words for “I’ll have a platform to do it for me.”
So what happened next?
I set the pin. Verified it. It worked. Days later, sessions started siloing again, dropping scaffolding folders inside real git repos, waking up knowing nothing.
The memory tools honored the pin perfectly. But IJFW ships hook scripts, separate shell scripts that do the actual scaffolding, project registration and context-building, and five of them hardcode paths relative to the working directory and never read the pin at all. Basically, the tool I used was not designed for how I was using it.
The generalizable version: a declared scope is a claim about every path by which a process can be born. Interactive shell, login shell, non-interactive, systemd unit, cron, your own dispatch seam, and whatever scripts your vendor ships. And verify it by reading /proc/
This is also the strongest argument for attested identity over ambient config. A credential the platform injects and verifies doesn’t have a line 8 to fall off.
Also, who owns the store is a security decision
When I set up /srv/fleet/shared_mem, the obvious move was to let Jarvis own it. It’s the assistant, memory is for the assistant, done.
Exactly backwards.
Jarvis is the most exposed principal in the house. Internet-facing, parses untrusted input from Telegram, can dispatch sessions. If anything in my lab gets prompt-injected, it’s Jarvis first. The substrate that every other agent trusts and reads from must never be owned by the riskiest thing in the system.
So the store is claude-orch-owned, mode 750, and Jarvis reaches it only through a mediated seam that audits every query. Same logic produced the dispatch rule I’m proudest of. Jarvis can start, stop, list and reattach Claude Code sessions. Jarvis cannot drive one. Only my authenticated Claude app can. The most exposed thing in the house tees up a powerful session and cannot touch it afterward.
If capability and exposure aren’t inversely related in your design, you’ve built a lateral movement path and called it convenience.
Correct plumbing, stale content
Once it all worked, I dispatched a session and asked it about Jarvis. It told me Jarvis hadn’t been built yet. I was messaging it on Jarvis.
The plumbing was perfect. The handoff note in shared memory was a day old and had never been refreshed, because the auto-capture that should have updated it had been writing to the wrong location for days.
And here’s the mental-model error underneath the bug, which cost me more than the bug did. I’d assumed shared memory meant the fleet knew what happened. It doesn’t. It holds what got curated into it.
And when it works
The payoff was worth every hour. One Claude Code session wrote a fact into fleet memory: that I was planning to buy a used HPE ProLiant for the VCF lab. Hours later I asked Jarvis, on my phone, what hardware I was planning to buy. It formed its own recall query, went through the audited seam, and answered correctly. Different agent, different model, different OS user, no shared filesystem access, nobody told it anything.
The lesson: memory is identity infrastructure
Every failure was about who was asking, which drawer they thought they were opening, and whether the system agreed with me about either. A process resolving to the wrong scope. A shell that skipped the line where the scope gets set. A vendor script that didn’t know the scope could be declared. A hook writing into a silo instead of the fleet.
That’s not a database problem. That’s an IAM problem wearing a database costume.
What that means if you’re doing this at work
The reason I care about this beyond my basement is that enterprises are about to wire agents together at scale, and the instinct will be to treat the memory or context layer as a data question. Which vector store, what chunk size, how good is the retrieval. Those are the easy questions.
The hard ones are the identity ones:
Does the tool separate caller identity from scope at all? If there’s one concept doing both jobs, you’re going to end up enforcing access with something outside the tool, the way I ended up enforcing it with chmod. That works on one box. It does not survive a cluster.
Is the scope declared or derived? If the answer is “it uses the working directory,” you’re one deployment change away from silent silos.
What enforces it on every launch path? In my lab it turned out to be one tool whose API respected the scope while five of its own shell scripts ignored it.
Who owns the shared store, and how does it relate to exposure? Once agents share memory, one agent’s context becomes another agent’s input. Two agents with different data entitlements on one store means the lower-privileged one now reads what the higher-privileged one wrote. Nothing in these tools enforces that today. In an enterprise that’s key.
Can you prove any of it right now? Not “we configured it.” Every verification that actually caught something in my build involved going around the tool: reading /proc, diffing a deployed file byte-for-byte, spawning a real second process instead of simulating one. If your verification is “the tool said OK,” you’re not verifying.
What’s the freshness contract? The handoff that told me Jarvis didn’t exist wasn’t corrupted, it was old. In production, agents acting on stale shared context make confidently wrong decisions that look nothing like an outage.
A short preview about what’s coming
There’s a ProLiant on the way for the VCF and Tanzu build, and the reason all of this looks over-engineered for one box is that it isn’t staying on one box. OS users become service accounts with real certificates. Localhost seams become services with policy between them. The env-var scope becomes a declared namespace that doesn’t care where the pod lands. The pipeline registry becomes admission control.
Which tells you what I actually built here. Not an agent lab. A control plane, hand-rolled, badly, one localhost service at a time. Every ugly thing in this post is something a platform is supposed to give you for free.
If you take one thing from this: put a seam in front of anything you don’t own, on day one, while it’s still cheap.
My lab rig is better and nothing is solved. Same as always.
More on this as it develops.