The Day My MCP Server Went Rogue (And How I Fixed It)

TL;DR

I built an MCP server. It worked perfectly — until one stress test nearly froze my machine.
That moment forced me to build production guardrails: authentication, rate limits, policy-based access, and structured auditing.

---

It Started With a “Simple Test”

One evening, I was testing my MCP server locally. It had a few standard tools: read_file, write_file, and summarize. Connected to Claude Desktop, everything seemed smooth.

So I tried one more thing:

“Read all 50 files in this directory and summarize each one.”

I leaned back, expecting clean output.
Instead, my terminal exploded.

Within seconds:

• Activity Monitor showed CPU bars pinned in the red.
• My trackpad became sluggish and unresponsive.
• The chassis felt like it could fry an egg.

The model had entered a retry loop: read_file → error → retry → read_file with no execution brakes.
It wasn’t a model bug. It was an infrastructure gap.

---

What I Had Built (Without Realizing)

At that moment, I realized my “cool project” was a high-powered engine with no brakes:

• No Auth: Any caller/process could attempt tool execution.
• No Throttling: Calls executed as fast as the model requested.
• No Boundaries: Sensitive operations had weak execution controls.

If you’re moving from demo to production, logic alone is not enough.
You need guardrails by design.

---

The 4 Pillars of a Production MCP Server

I stopped treating this as a tool demo and started treating it as a gateway system.

1) Identity (Authentication)

Every request should prove identity, even in local/dev flows.

In my repo, this is enforced with:

• MCP_CLIENT_ID
• MCP_API_KEY

on every request path before tool execution.

2) Throttling (Rate Limits)

LLMs are fast. Retry loops are faster.
I implemented sliding-window limits per client and per tool. Example from policy config:

• baseline default: 100 req/min
• sensitive tool: write_file: 5/min

3) Governance (Policy-Based Access)

Permissions are data-driven, not hardcoded inside tools.
I moved rules into config/policy.yaml so access control stays explicit and reviewable:

• who can call which tools
• at what limits
• under what policy profile

4) Observability (Structured Auditing)

When incidents happen, plain logs are not enough.
Every access attempt (allowed/blocked) is emitted as structured JSON, including decision context, so debugging is fast and reliable.

---

Quick Threat Model (What This Assumes)

Before production, I made these assumptions explicit:

1. Trust boundary: local env vars are trusted only as far as the parent process is trusted.
2. Secret handling: clients.yaml is fine for demo/local, but prod should use a secret manager (Vault/ASM, etc.).
3. Defense-in-depth: even with leaked credentials, rate limits and policy constraints reduce blast radius.

---

Architecture Rule That Changed Everything

Keep tools dumb. Keep guardrails centralized.

My read_file tool should only do file reading.
Auth, rate limiting, policy checks, and auditing belong in the middleware pipeline before sensitive logic runs.

That single separation makes the system easier to secure, test, and evolve.

---

If You’re Shipping MCP Soon

Don’t wait for your terminal to start screaming.

I open-sourced this as a reference implementation:
👉 mcp-guardrails If you’re deploying MCP internally, clone it for safe defaults and adapt the pattern to your environment.

---

Found this useful? I’m building (and stress-testing) these systems in public — more posts soon.