A while ago I wrote about turning an old laptop into an isolated AI coding sandbox. That was a one-user, one-machine setup. Building the cluster-scale version of the same idea taught me something I wish I’d known earlier: you don’t design this architecture up front. You earn it, one problem at a time.

So instead of showing you the finished topology first, this post walks the same path I actually took. Each block exists because the block before it exposed a problem. If you’re building anything similar, your progression will probably rhyme with mine.

Block 1: The Agent Core, Strands SDK Behind an API

The first decision wasn’t about security at all. It was just: get the thing talking.

A UI, an API, and an agent. The Strands SDK handles the conversational loop, so prompt goes in, tool calls happen, LLM round-trips run, and structured output comes out. I had zero interest in rebuilding agent plumbing when battle-tested SDK code existed. The architecture started about as boring as possible:

[ Browser ] ──► [ UI/API Container ] ──(in-process)──► [ Strands Agent ]

At this stage the agent ran in the same container as the API. One process group, shared everything, key on disk, zero ceremony.

And it worked. Which is exactly what made the next problem hurt.

Block 2: Why Does My Agent Have My API Keys?

Once I stepped back and looked at that container, one question refused to go away: this is the component whose entire job is executing code an LLM wrote, and it is also holding my OpenAI key, my Zabbix token, and my Mongo connection string.

Permission separation had to happen. The agent moved to its own container entirely, completely decoupled from master credentials and the host machine runtime:

[ Browser ] ──► [ UI/API Container ]        [ Strands Container ]
                        │                            ▲
                        └────── how to talk? ────────┘

This looks like a small step but it changed the whole security posture. I also decided early that the agent would get no public internet access at all. It lives inside a private Docker bridge network declared internal: true, meaning Docker never even builds an external route for it. That promise immediately created the next question.

Block 3: Two Containers, No Network. How Do They Talk?

The agent was now air-gapped, and the API needed to still reach it. HTTP on localhost felt wrong: anything with network access can poke a TCP port. Then Unix Domain Sockets walked in and never left.

A Unix socket is just a file on disk. It is unreachable from another machine, it is cheap because there is no TCP stack involved, and the property that really sold me: you can mount exactly one socket into exactly two containers, and nothing else in the cluster can see it.

volumes:
  - shared-ipc-sockets:/app/sockets

networks:
  internal-secure-net:
    internal: true   # agent container lives here: no route out, ever

The stateless API worker intercepts the user’s message, packs the connection tracking information with the corresponding sessionId, and pushes it through the socket into the Strands engine. The same channel carries the answer back. This one mechanism, a shared socket volume on an isolated network, became the backbone of the whole design. And it comes back a second time later. Remember it.

Block 4: Running the Code It Writes, Enter gVisor

Now the agent generates Python. Great. Where does that run?

Option one, same machine and hope for the best, is off the table. Option two, a plain Docker container, is closer but still shares the host kernel. A kernel-level escape from inside that container means game over, and assuming the LLM might emit something malicious is not pessimism, it is a failure mode you design for.

The answer was gVisor: run the sandbox under the runsc runtime, which sits between untrusted code and the host kernel and implements an application kernel of its own. Syscalls from inside get intercepted by gVisor’s fake kernel before the real kernel ever sees them.

docker run \
  --runtime=runsc \     # gVisor: an application kernel in front of the host
  --network=none \      # the script needs no internet, ever
  --memory=128m \       # an intentional flood dies instantly
  -v /mnt/efs/scripts/script_uuid:/app/code:ro \
  python /app/code/script.py

Every flag answers a specific threat:

• --runtime=runsc means container escape now means escaping gVisor, not the host kernel.
• --network=none takes the network attack surface from “manageable” down to literally zero, since the script aggregates local metrics and dumps its breakdown straight to stdout.
• --memory=128m hard-caps against intentional or accidental denial-of-service memory floods.
• :ro mounts stop the code from rewriting itself or tampering with its input data mid-run.

The sandbox runs, prints, exits, and gets destroyed immediately. Beautiful, until I asked who was allowed to spawn it.

Block 5: Who Spawns the Sandbox? The Agent Must Not

Here is the failure mode no Docker flag solves: the Strands agent has a “run this script” tool, and the only clean way to implement that tool inside the agent container is to give it access to /var/run/docker.sock, which is effectively root on the host.

Worse, the problem is dual-headed. Given the Docker socket, an LLM agent already drifts between two equally bad behaviors: hallucinating Docker flags that do not exist, and spawning new containers on its own initiative because nothing physically stops it.

So the fix had to be structural rather than prompt-level. I introduced the proxy manager: a dedicated worker container that is the only service in the entire system permitted host mounting rights to the Docker socket. Critically, the UI/API tier itself has no socket access at all, so there is exactly one dumb privileged hop in the entire cluster. The agent does not ask “run this docker command”. It drops a job on a queue. The agent does not reason about containers, does not know Docker exists, and never touches the daemon.

[ Strands Agent ] ──job──► [ RabbitMQ ] ──► [ Proxy Manager ]
                                                  │  (only docker.sock)
                                                  ▼
                                          [ gVisor Sandbox ]

Once the agent cannot spawn containers, hallucinated Docker invocations stop being a prompt-engineering problem and become an impossibility. One design constraint beats ten instructions.

Block 6: Tokens Again, the Proxy Manager Needs the LLM Too

The proxy manager turned out to need more than script execution. It retains the richest knowledge of each job, the metric keys, the schemas, the expected output shape, so it also needs to talk to the LLM for verification and formatting steps.

But the entire point of Block 2 was that nobody internal gets the API key. Hand the worker a key? No. This is where the Unix trick pays its second dividend.

The UI/API container runs an outbound HTTP token proxy: a tiny blind gateway whose whole job is to receive requests from internal containers, wrap the payload with the master key, forward it to the LLM provider, and pass the response back. The key never crosses the socket boundary; only proxied requests do.

# inside agent and worker containers:
- STRANDS_LLM_BASE_URL=http://ui-api-layer:80/v1/proxy/openai

Internal components think they are calling OpenAI. They are calling a blind gateway that adds authentication on the way out. Same pattern as Block 3: a socket-local gateway, one shared namespace, no routable path, applied at the HTTP layer. Once you have one honest choke point, every credential problem becomes “route through the choke point”.

Block 7, Designed But Not Built: The Same Question Asked Twice

At this point the loop worked end to end, and it works everywhere: every equivalent question regenerates the answer from scratch. I can already see the pattern coming. Two users, a day apart, will ask semantically identical questions, and the agent will generate the same script twice and pay full LLM tokens both times. In a multi-tenant service that compounds fast.

So I designed the next block before running into that bill: a script registry indexed by vector embeddings, PostgreSQL with the pgvector extension. The plan:

1. Every generated script gets its intent vector-embedded and cataloged alongside its EFS path.
2. Every new prompt gets embedded too, and searched against the registry before doing anything else.
3. Match found: skip generation and reuse the script. No match: generate, then store.

SELECT script_path FROM scripts
ORDER BY embedding <=> $1   -- cosine distance to the new prompt
LIMIT 1;

The moment I wire this in, every script the service generates becomes a permanent, reusable asset. The cost curve bends the right way: cheaper and faster with each question answered. This is the compounding-loop idea applied at the artifact level: stop re-deriving what you have already built. It ships next on the roadmap.

Block 8, Designed But Not Built: Why Is prod-vm-04 Slow?

The RAG design solves deduplication. It will not solve the harder problem I already know is waiting: answer quality when user prompts correlate multiple data sources at once.

“Right now, why is the load high on prod-vm-04?” is never a question about one thing. It secretly depends on knowing that prod-vm-04 hosts a MongoDB cluster instance, that it is actively tracked by the Zabbix collector zbx-agent-102, and how those two relate. Pure vector search can surface documents mentioning prod-vm-04, but it cannot tell the agent which facts matter and how they connect. That is how an agent hallucinates and correlates the wrong metrics.

So the fourth data plane component is designed: a Neo4j knowledge graph of the real infrastructure. Nodes are servers, databases, instances, and collector configs; edges are the dependencies between them. The agent gets a graph tool, and for anything touching specific entities it will pull the exact relevant subgraph into context:

MATCH (vm {name: 'prod-vm-04'})-[r]-(n) RETURN vm, r, n

Vector RAG pins the fact that a script worked. The graph pins why the reasoning that produced it was sound. Rediscovered relevant relationships, not plausible-sounding documents, are what stop the model from inventing dependencies that do not exist. Both components are already specced into the compose stack; wiring them is the work I have left.

The Full Round Trip (Current State)

Here is the entire machine executing a real prompt end to end. Watch how every block clicks into place in one transaction:

1. Prompt submit. A user types “why is prod-vm-04 load high right now?” into the load-balanced web console.
2. Unix handoff. Any one of the three stateless API nodes takes the message, packs tracking metadata with sessionId: "sess_42" and a fresh job_999, and pushes it through the Unix Domain Socket into the air-gapped Strands engine.
3. RAG check (next up). Strands will interrogate PostgreSQL with a vector embedding search for this exact category of performance question. Today this step short-circuits straight to generation; the pgvector registry is designed and queued.
4. Graph query (next up). Strands will ask the graph tool for topology context. A Cypher query over Neo4j will return explicit node mapping: prod-vm-04 hosting MongoDB instance cluster-shard1, tracked by Zabbix collector zbx-agent-102. Today the agent works against the prompt alone.
5. Generate code. Since the Strands container is barred from the public web, the LLM tokens pass through the token proxy inside the API container, which wraps the request with the secret key. The LLM returns a dual layout: an automation block (script.py) and a retrieval blueprint (blueprint.json).
6. Save artifacts. The logic package lands on EFS under /mnt/efs/scripts/script_uuid_102/. Once the pgvector registry ships, the intent metadata also gets vector-embedded into PostgreSQL here for the next user’s reuse.
7. Push execution job. Strands stamps sessionId and jobId onto a metadata envelope and sends it into RabbitMQ, then completely flushes this turn out of its own active memory. Stateless by design.
8. Spawn box. A free proxy worker pulls the job, extracts the required Zabbix metric keys and MongoDB schemas according to blueprint.json, authenticates with internal vault tokens, flushes the live diagnostic inputs to /tmp/data.json, and uses its local Docker socket to spawn a temporary container.
9. Run script. The sandbox comes up strictly bound: --runtime=runsc, --network=none, code and data both mounted read-only from EFS and /tmp/data.json. It runs python /app/code/script.py, aggregates the metrics with zero network access, dumps the analytical breakdown to stdout, and exits.
10. Kill box. The proxy worker captures exit code and buffered stdout, destroys the temporary container immediately, and wipes the intermediate files off its disk.
11. Publish result. The worker reports the execution payload back to the Redis cluster channel response:job_999.
12. Redis event. Redis broadcasts cluster-wide. Whichever API node happens to hold the open HTTP connection for sess_42 picks up the signal, re-hydrates the Strands session, and feeds the script diagnostics into the model to compose a natural-language answer.
13. Flush response. The answer text goes back through the Unix socket to the API gateway and straight into the user’s browser. The user never sees a terminal, and never sees the plumbing.

job_999 threading through all thirteen steps is what makes the whole dance work across horizontally scaled nodes. Identity travels with the work instead of living on the machine.

The Full Compose Skeleton

For completeness, the orchestration layout that models this:

version: '3.8'

services:
  # --- TIER 1: INTERNET-FACING APPS ---
  ui-api-layer:
    image: company/enterprise-ui-api:latest
    restart: unless-stopped
    ports: ["80:80", "443:443"]
    environment:
      - SECRET_ZABBIX_TOKEN=zbx-prod-token-rotated
      - SECRET_MONGO_CONN_STR=mongodb://root:vault@mongo-cluster:27017/
      - SECRET_OPENAI_KEY=sk-proj-vault-rotated
      - REDIS_BACKBONE_URL=redis://central-message-broker:6379
    volumes:
      - shared-ipc-sockets:/app/sockets
      - shared-efs-volume:/mnt/efs:ro
    networks: [public-ingress-net, internal-secure-net]

  # --- TIER 2: AIR-GAPPED AI CORE ---
  strands-agent-core:
    image: company/isolated-strands-engine:latest
    restart: unless-stopped
    environment:
      - STRANDS_LLM_BASE_URL=http://ui-api-layer:80/v1/proxy/openai
      - REDIS_BACKBONE_URL=redis://central-message-broker:6379
      - POSTGRES_VECTOR_DB_URL=postgresql://db_user:pass@postgres-vector:5432/rag_db
    volumes:
      - shared-ipc-sockets:/app/sockets
      - shared-efs-volume:/mnt/efs
    networks: [internal-secure-net]   # no public network access, ever

  # --- TIER 3: BROKERS AND STORAGE ---
  central-message-broker:
    image: redis:7.2-alpine
    restart: unless-stopped
    command: redis-server --appendonly yes
    networks: [internal-secure-net]

  rabbitmq-cluster:
    image: rabbitmq:3.12-management-alpine
    restart: unless-stopped
    networks: [internal-secure-net]

  postgres-vector:
    image: ankane/pgvector:latest
    restart: unless-stopped
    environment:
      - POSTGRES_USER=db_user
      - POSTGRES_PASSWORD=pass
      - POSTGRES_DB=rag_db
    networks: [internal-secure-net]

  neo4j-graph-db:
    image: neo4j:5.12-community
    restart: unless-stopped
    environment:
      - NEO4J_AUTH=neo4j/secure_graph_password
    networks: [internal-secure-net]

  # --- TIER 4: WORKERS AND ORCHESTRATION ---
  proxy-manager-worker:
    image: company/sandbox-worker-manager:latest
    restart: unless-stopped
    environment:
      - REDIS_BACKBONE_URL=redis://central-message-broker:6379
      - RABBITMQ_URL=amqp://rabbitmq-cluster:5672
    volumes:
      # critical risk boundary: the ONLY container granted host daemon access
      - /var/run/docker.sock:/var/run/docker.sock
      - shared-efs-volume:/mnt/efs:ro
    networks: [internal-secure-net]

networks:
  public-ingress-net:
    driver: bridge
  internal-secure-net:
    driver: bridge
    internal: true   # forces absolute air-gapped isolation!

volumes:
  shared-ipc-sockets:
  shared-efs-volume:

The line that carries the entire security model is this one:

internal: true   # forces absolute air-gapped isolation!

One property on one network definition guarantees that nothing behind it, the brain, every broker, every worker, can even route a packet to the internet. The token proxy, gVisor, and read-only mounts are all layered on top of it.

Key Takeaways

• Isolation is a sequence, not a property. Same container, then split containers, then Unix socket split, then sandbox. Each step was driven by one concrete problem the previous step created. If you draw the whole diagram first, you cargo-cult more than you understand.
• Two reps of the same trick are a signal. Unix Domain Sockets solved inter-container isolation and then credential containment. When a mechanism solves two problems cleanly, keep it as a primitive, not an accident.
• The docker.sock is the crown jewel. One mount, in one dedicated proxy layer, and nothing else. Hallucinated Docker invocations vanish the moment the agent physically cannot spawn containers.
• RAG throttles spend, graphs sharpen truth. pgvector stops you paying twice for the same work. Neo4j stops the model from correlating things that are not related. They answer different questions and you need both. Designed, specced, next on the queue.
• Every script is compound interest, once the registry exists. It is worth designing for reuse before the first duplicate arrives, because retrofitting a registry onto an already-running service means backfilling embeddings for everything you shipped without one.

The security instinct itself came from the old laptop sandbox. At cluster scale, the only thing that changed is that every risk had to be answered with structure instead of vigilance.

---