MCP server review

An MCP server lets an AI assistant search, read and act on your product on behalf of a user. The caller is a language model acting on text it has read, for a user who may not be watching every step. This means that a server that might be perfectly reasonable as an ordinary API can be a security hole when called by an AI assistant.

I built the CLI and the MCP server for caffeine.ai, whose MCP server is listed in Anthropic’s Claude directory.

When it helps

The usual time for a review is just before launch, or before submitting to an assistant’s directory, when teams want an independent assessment before customers connect. A review can also be useful at the design stage, helping to shape the tools and their boundaries before implementation. Another natural point is after a period of rapid growth, to take stock of what an assistant can now do with the server.

Lessons from building one

The protocol is the easy part. Having built Caffeine’s CLI, I knew the systems the MCP server would connect to. Supporting assistants brought a different set of requirements. A few examples give a sense of the work involved:

Signing in from a browser. Caffeine’s sign-in was designed for its CLI, running on the user’s own machine, whereas claude.ai runs in a browser. So the server offers standard OAuth to the assistant and uses Caffeine’s device sign-in behind it. The assistant never holds a Caffeine token, only one that is valid at the MCP server, for one user, and that can be revoked instantly.

Retries that look like theft. Each refresh token works once. If a used token turns up again, someone has a copy, so the server ends the whole session, just to err on the side of caution. But a genuine client retrying on a bad network sends the same token twice, and a design that doesn’t allow for that locks out the very users it’s meant to protect.

Connections that stay open. MCP over HTTP keeps connections open while a tool works, and a gateway’s default timeouts will cut a long tool call off part-way through. Deploys need the same care, so that in-flight calls can finish before a server stops.

A review looks for problems like these in your server.

What I look at

• How users sign in, whether the server checks that every token was issued for it, and whether it ever passes one on to another service
• Whether each tool does one specific job, or whether a generic tool allows a persuaded model to do anything that the server can do
• Which tools can change or delete data. Read-only should be enforced in the server, not just labelled, and writes should be safe to repeat when a client retries.
• What a tool result containing instructions could make an assistant do through your server
• What kind of data is returned by the tools, and whether credentials or personal data reaches the conversation or the logs
• Rate limits, result sizes, timeouts, and error messages an assistant can act on
• The current revision of the MCP specification, which changes quickly, and what a directory listing is likely to need

With access to the code, the review covers both the code and how the server behaves when a real assistant uses it.

What you get

A written report, with each finding ranked by severity and by the effort to fix it, and a walkthrough with your team. A finding looks something like this (an illustrative example, not from a real client):

High. archive_project is offered to connections that were only granted read access, because the permission is checked when the tool list is built but not when the tool is called. An assistant that has read a hostile issue description can archive projects. Fix: check the write permission in the tool handler. Effort: about half a day.

The review is at a fixed price, agreed before it starts. A review of a single server usually takes about a week from start to report, and a review of a design takes less time. Help with the changes afterwards is available, but the review doesn’t depend on it.

If you need an MCP server built rather than reviewed, see AI integration and MCP.