Skip to main content
Use the Scorecard MCP (Model Context Protocol) server to create evaluation testsets and metrics through natural language in Claude. Instead of writing code or clicking through UIs, just tell Claude what you need and it will use the Scorecard API to set everything up.
This quickstart uses Claude Desktop, but it also works with other MCP clients, like Cursor and Claude Code.

​
Steps

1

Setup Scorecard account

Create a Scorecard account if you don’t have one already.
2

Install the Scorecard MCP server in Claude

Open Claude and add the Scorecard MCP server using the remote configuration:
  1. Open Claude Desktop settings and click the “Connectors” tab
  2. Click “Add custom connector” and paste the URL: https://mcp.scorecard.io/mcp
  3. Click “Add”, then “Connect”
  4. Complete the OAuth authentication flow to connect your Scorecard account
The remote MCP server requires no local dependencies or API key management. Authentication happens securely through your browser.
Once connected, Claude has two tools: search_docs to look up the Scorecard SDK, and execute to run code against it. Together they cover the whole Scorecard API, so you only need to describe what you want in natural language.
3

Create a project

Simply ask Claude to create a new Scorecard project:
Create a new Scorecard project called “Customer Support Bot Evaluation” for testing my AI customer support assistant.
Behind the scenes, Claude writes a short TypeScript snippet against the Scorecard SDK and runs it with the execute tool. Expand the tool call in the conversation to see the code it ran.
4

Create a testset with testcases

Now create a testset to hold your evaluation test cases. Describe the structure you need:
Create a testset called “Support Scenarios” in this project. The testcases should have:
  • Input fields: “customerMessage” (the customer’s question) and “category” (support category like billing, technical, or product)
  • Expected output field: “idealResponse” (what a great response from the agent looks like)
  • Metadata field: “difficulty” (easy, medium, or hard)
Then add 5 testcases covering different support scenarios.
Claude can usually guess which inputs and output fields you want, but it’s better to tell it what your field names are.
5

Create evaluation metrics

Define metrics to evaluate your AI system. Describe what “good” looks like:
Create two AI-scored metrics for this project:
  1. “Response Accuracy” (integer) - Measures how well the response answers the customer’s question compared to the ideal response
  2. “Tone Appropriateness” (boolean) - Checks if the response uses professional, empathetic language appropriate for customer support
If Claude is unsure how to call the SDK, it runs search_docs first to look up the right method and parameters, then tries again. A failed first attempt followed by a corrected one is normal.
6

View your setup in Scorecard

Open your Scorecard project in the web UI to see everything that was created!Everything is now ready to score records. You can score records against your metrics through the UI, SDK, or continue using Claude with MCP.
7

Next: Score records

You can score records from the Records page or the SDK.You can also continue the conversation to analyze and iterate on your metrics and scores.
Explain the latest scoring results for this project.
Update the Response Accuracy metric to be stricter about factual details.
Add 5 more testcases covering edge cases like angry customers and off-topic questions.
The MCP server gives Claude access to the full Scorecard API, so you can manage your entire evaluation workflow conversationally.

​
Tips for Using the MCP Server

Be specific about data structures: When creating testsets, clearly describe the field names, types, and which fields are inputs vs expected outputs. This helps Claude set up the schema correctly.
Describe evaluation criteria: When creating metrics, explain what makes a “good” output in detail. Claude will translate this into effective evaluation guidelines.
Ask for recommendations: Claude can suggest metrics, testcase scenarios, and evaluation strategies based on your use case. Just ask “What metrics should I use for evaluating a RAG system?”
Iterate conversationally: Made a mistake? Just ask Claude to fix it: “Make that metric stricter about factual details” or “Add a new field called ‘priority’ to the testset”

​
Troubleshooting

The Scorecard MCP server works with most MCP clients, including Claude Desktop, Cursor, and Claude Code. Make sure you’ve added the remote server URL correctly (https://mcp.scorecard.io/mcp) and completed the OAuth flow. See the MCP Server documentation for more installation instructions.
Make sure you’ve added the remote server URL correctly (https://mcp.scorecard.io/mcp) and completed the OAuth flow. Restart Claude if needed.
The MCP server uses OAuth tokens that may expire. Try disconnecting and reconnecting the MCP server in Claude settings to refresh authentication.
Verify the MCP server is connected and enabled in Claude settings. You should see “Scorecard” listed in your active MCP servers.
That is expected. The server exposes search_docs and execute, and Claude reaches every Scorecard API operation by writing code for execute. See How it works.
The local npx scorecard-ai-mcp server runs code in a Deno sandbox on your machine. Install Deno and make sure it is on your PATH, then restart your client. The hosted server at https://mcp.scorecard.io/mcp has no local requirements.
For local installation with your Scorecard API key, see the MCP Server documentation. Use npx -y scorecard-ai-mcp@latest with environment variables.

​
Learn More

Ready to go deeper? Check out these resources:

MCP Server Features

Complete guide to the Scorecard MCP server capabilities and architecture

Metrics Guide

Deep dive into creating and managing evaluation metrics

Testsets

Learn about testset schemas, field mappings, and organization