Improving the quality of server entries published to the Official MCP Registry

[BobDickinson]

Pre-submission Checklist

• I have verified this would not be more appropriate as a feature request in a specific repository
• I have searched existing discussions to avoid duplicates

Your Idea

My interest in the Official MCP Registry is from the standpoint of someone building MCP client applications. As such, one of the most exciting aspect of the Official MPC Registry is the configuration language available to MCP server publishers that allows clients to generate high quality user interfaces for MCP servers (almost as if each server has its own custom UX), and then allow users to configure servers through that UX to generate MCP server configurations. In theory the user finds a server, picks a package or remote, fills in the generated config UX, and deploys a properly configured MCP server (without having to go the server repo, read through the docs, and then probably end up copying and pasting some sample JSON and getting it to work through trial and error).

As a proof-of-concept I built an MPC Server Registry browser, which includes UX code to configure servers, and much to my dismay, discovered that the vast majority of currently published servers either provide no configuration information at all, or they provide configuration information that either makes it impossible to configure the server via the UX or creates a misconfigured server.

In what follows, my goal is not to shame anyone or call anyone out about their published servers. I assume everyone wants to make great servers and create correct metadata, and to the extent that there are some challenges, we should work together to help get everyone up to speed (via evangelism, documentation, tooling, etc). FWIW, the server.json Format Specification document is a pretty solid start.

Schema

There is a server.schema.json file that is the definitive schema for server.json data, but it is not automatically applied to servers when they are published. Here are some schema validation stats for the current set of published servers:

Issue	Count
Servers with current schema version	535
Servers with older schema versions (fine)	96
Servers with empty repository URL (schema validation)	88
Package version "latest" (prohibited by schema)	15
Incorrect env var format (same server)	4
Invalid name	1
Invalid status	1

The empty repository url condition is actually caused by mcp-publisher and an issue has been opened for that: #613

So schema compliance is actually not too bad, and would be easily remedied by fixing the publisher bug and requiring servers to pass schema validation when published.

Recommendation: If there is an official schema and we're going to publish servers that indicate (on a per-server basis) that they comply with that schema, we must validate the server against the schema when publishing. That has to be part of the contract between the publisher, the registry, and registry clients.

Beyond Schema

As I dove into testing servers against my UX, I noticed many other issues. I started recording and categorizing these issues, then eventually decided to write a linter to automate the process and collect some statistics across all published servers: mcp-registry-validator. This is available as a CLI app and API, and is integratied into the server test mode of MCP Server Registry Browser.

The full list of linter rules can be found here if interested.

Here were the issues found in the set of published servers:

Rule	Instances
prefer-config-for-remote	160
require-config-for-package	151
no-value-with-irrelevant-properties	137
no-template-variables-missing	128
no-secret-template	128
no-secret-static-value	128
require-args-leading-dashes	49
prefer-dynamic-port	16
require-valid-choices-format	4

Some of the issues found are semantic errors that couldn't have been caught by schema validation (value, default, or choices that don't match the specified format, for example). Some of the issues were more in the area of best practices (packages and remotes should have config, sse and streamable package transports should have configurable ports, etc). And some were in a gray area (named arguments without a - or -- prefix, which while legal in the schema, were used in error 100% of the time in published servers).

A large portion of the errors across several rules were caused by a single error common to 128 remotes that makes them impossible to configure (an Authorization header with a fixed value containing a token, and no token variable to resolve it). The intent of this configuration is pretty clear and it would be an easy fix, and I'm assuming this is all the same publisher.

The bottom line of all of this is that about half of the servers have no configuration, and of the ones that do have configuraiton, only about a third of those produce a valid UX that in turn produces a valid MCP server configuration.

Recommendation: We should provide semantic validation (beyond schema validation) to prevent logical errors and we should provide best-practices feedback to improve the quality of published server entries.

Interactive Server Configuration Tester

I believe that having an easy way to validate your server.json, and to see the UX that it generates and the server config that that UX produces, will help server publishers create better server definitions.

To that end, I made a ToolCatalog Registry project that supports discovering, viewing, and configuring MCP servers from the current Official Registry. I added a Server Configuraiton Tester that can be found in the top nav bar, labelled: Developers: Test your server.json. This allows server publishers to paste in their server.json and test it in the same user interface. They can see the user interface generated from their configuration and interact with it to validate that it produces the expected MCP server configuration. The Server Configuration Tester also provides a Vaidate function that performs JSON parsing, schema validation, and applies server.json linter rules to surface potential issues with the configuration. Server publishers can edit their server.json interactively to address validation issues and tune the generated UX.

This is implemented as a static Next.js website published via a GitHub action and hosted via GitHub pages (very lightweight). It polls the Official Registry once per day and generates a static JSON file that's used by the app. I'd be happy to move this into the main repo if there was interest (perhaps with some guidance). Alternatively, I could separate out the server test mode into its own app (with no dependency on the registry data) that might scale better longer term and still support the server publisher use case.

Also, this website uses my validator package (a pure Javascript npm package). In a perfect world the website would use the validation code from the main repo. An option would be to build an npm package around the main project validation code (likely via WASM) so we have one source of truth for validation available for Go and Javascript. I'd be happy to take a stab at that.

Improved Validation

We have existing semantic validation in the project that catches some serious schema errors. It fails fast, returning the first error encountered (with no indication as to the server.json element involved or the specific schema constraint or rule that triggered it, other than what is in the error message). This has been reasonably effective in maintaining the level of schema compliance we have today (which, while not 100%, is pretty decent).

The ideal state would be:

• Validate servers against the schema
• Perform ad-hoc semantic validation for logical errors that cannot be enforced by the schema
  • We have some of this now in our existing validator implementation, but need to augment it
  • After adding schema validation, we should remove ad-hoc validations that duplicate schema validation
• Validate against linter rules representing best practices (for usability, security, etc)
• Return all validation issues with rich metadata to support addressing the issues
• Servers must pass schema and semantic validation to be published. Publishers may suppress or ignore linter issues.

I have implemented a PR which begins to implement improved validation. It is low-risk, low impact, and backward compatible. I did not feel comfortable going all the way with the longer-term plan until I got some validation of the direction (including the implementation) and the plan itself.

Phase 1 (PR inbound)

Internal Validation improvements

• Schema validation has been implemented and is available as an option in validation methods (defaults to off)
• Existing validators track the JSON path (context) of the attribute they're evaluating, passing it to any child validators
• Existing validators are exhaustive (return all issues, not failing fast on the first error)
• Existing validators return rich validation results, including:
  • type (schema, semantic, linter)
  • severity (error, warn, info)
  • path (JSON path to server.json element)
  • reference (to the schema element or rule that triggered the result)
  • message (currently the same as previous error message)

A new validate command has been added to mcp-publisher so publishers can evaluate their server.json before publishing

• Includes new schema validation and previous semantic validation
• Shows full details of all issues encountered during validation

Phase 2 (fast follow, assuming we move forward)

• Require passing schema validation (in addition to improved semantic validation) in order to publish
• Port all of my linter rules (that apply) to the project validation implementation
• Remove existing validation code that is redundant to schema validation
• Update validation client code and tests to handle the new rich return types natively

Later

Assess interest in adding server validation client app to the project
Assess interest in creating an npm package wrapping our validation code

In conclusion

I'd appreciate feedback and encourage discussion on any of the above, including things I didn't cover or alternative paths to get to more and better server configurations.

Scope

• Protocol Specification
• SDK Features
• Documentation
• Developer Experience
• Other