MCP Server Integration

Use AI assistants as your evaluation companion with Scorecard’s Model Context Protocol server

Overview

Scorecard’s MCP (Model Context Protocol) server lets you manage projects, create testsets, configure metrics, run evaluations, and analyze results through natural language in any MCP-compatible client.

How it works

The server uses Code Mode: instead of one tool per API endpoint, it exposes two tools and lets your assistant write code against the Scorecard TypeScript SDK.

Tool	What it does
search_docs	Searches the SDK documentation for the right method and parameters.
execute	Runs TypeScript your assistant writes against a pre-authenticated SDK client in a sandbox, and returns only what the code returns or logs.

A typical execute call looks like this:

async function run(client) {
  const results = [];
  for await (const project of client.projects.list()) {
    const runs = await client.runs.list(project.id, { limit: 1 });
    results.push({ project: project.name, latestRun: runs.data[0]?.id ?? null });
  }
  return results;
}

Because filtering and aggregation happen inside the sandbox, only the final result comes back to the conversation. A question that spans many projects, runs, or large records completes in a single call instead of paging multi-megabyte responses through your assistant’s context.

Each execute call is independent: variables do not persist between calls. Individual API requests time out after 30 seconds and a whole execute call after about 5 minutes. The sandbox can only reach the Scorecard API, with no filesystem or other network access.

Setting Up the MCP Server

Claude Code

Add the Scorecard remote MCP server with a single command:

claude mcp add --transport http scorecard https://mcp.scorecard.io/mcp

Complete the OAuth authentication flow in your browser when prompted. Verify the connection:

claude mcp list

You should see scorecard: https://mcp.scorecard.io/mcp (HTTP) - ✓ Connected.

Claude Desktop

Go to Claude Desktop settings and click the “Connectors” tab. Click “Add custom connector” and paste the URL: https://mcp.scorecard.io/mcp. Click “Add”, then “Connect” to login to Scorecard.

Other clients

Any client that supports remote MCP servers over Streamable HTTP can use https://mcp.scorecard.io/mcp. If your client cannot complete an OAuth flow, pass a Scorecard API key as a bearer token instead:

{
  "mcpServers": {
    "scorecard": {
      "url": "https://mcp.scorecard.io/mcp",
      "headers": {
        "Authorization": "Bearer ak_MyAPIKey"
      }
    }
  }
}

Local configuration

You can also run the same server locally via npx:

export SCORECARD_API_KEY="your_api_key"
npx -y scorecard-ai-mcp@latest

For clients with a configuration JSON:

{
  "mcpServers": {
    "scorecard_ai": {
      "command": "npx",
      "args": ["-y", "scorecard-ai-mcp"],
      "env": {
        "SCORECARD_API_KEY": "ak_MyAPIKey"
      }
    }
  }
}

The local server runs execute code in a Deno sandbox on your machine, so Deno must be installed and on your PATH. Documentation search works offline from an index bundled with the package.

To restrict what the assistant can do, limit the execute tool with flags. For example, a read-only server:

"args": ["-y", "scorecard-ai-mcp", "--code-allow-http-gets"]

You can also allow or block specific SDK methods with --code-allowed-methods and --code-blocked-methods, which take regular expressions matched against method names such as records.delete.

Examples

Create a project and testset

Create a new Scorecard project called "Support Bot Eval". Then create a testset
called "Support Scenarios" with 10 testcases. Each testcase should have:
- inputs: "customerMessage" and "category" (billing, technical, or product)
- expected: "idealResponse"

Create metrics

Create two metrics in the "Support Bot Eval" project:
1. "Response Accuracy" (integer 1-5) - How well does the response answer the question?
2. "Tone" (boolean) - Is the response professional and empathetic?

Analyze results

Show me the latest run results for the "Support Bot Eval" project.
Which testcases scored lowest on Response Accuracy?

Compare across projects

For every project in my organization, find the most recent run and report the
pass rate on each boolean metric. Rank the projects from best to worst.

Review human feedback

For the latest run in the "Support Bot Eval" project, list the annotations on
each record and summarize the thumbs-down ratings. What do reviewers complain
about most?

Generate testcases from a codebase

In Claude Code, you can combine file access with the MCP server:

Read the API routes in src/api/ and generate 20 testcases covering
the edge cases for each endpoint. Add them to the "API Tests" testset
in project 1234.

Iterate on metrics

The "Response Accuracy" metric is too lenient — update the prompt template
to penalize responses that miss key details from the ideal response.

Technical Details

• Built on the Model Context Protocol standard using the Streamable HTTP transport
• Compatible with any MCP client (Claude Code, Claude Desktop, Cursor, and more)
• Secured with OAuth authentication, or a Scorecard API key passed as a bearer token
• The hosted server runs the scorecard-ai-mcp package (3.4.0), whose source lives in the scorecard-node repository
• Anything the SDK can do is available: see the API reference for the full surface

Migrating from the per-endpoint tools

Earlier versions of the server exposed one tool per API operation, such as list_projects or create_testsets. Those tools no longer exist. Prompts written in natural language keep working unchanged, because the assistant now reaches the same operations through execute. If you referenced tool names directly, the table below shows where each one went.

Old tool → SDK call mapping

Old tool	SDK call inside execute
list_projects, create_projects	client.projects.list(), client.projects.create({...})
list_testsets, get_testsets, create_testsets, update_testsets, delete_testsets	client.testsets.list(projectId), .get(id), .create(projectId, {...}), .update(id, {...}), .delete(id)
list_testcases, get_testcases, create_testcases, update_testcases, delete_testcases	client.testcases.list(testsetId), .get(id), .create(testsetId, {...}), .update(id, {...}), .delete({ ids })
list_systems, get_systems, upsert_systems, update_systems, delete_systems	client.systems.list(projectId), .get(id), .upsert(projectId, {...}), .update(id, {...}), .delete(id)
get_systems_versions, upsert_systems_versions	client.systems.versions.get(id), client.systems.versions.upsert(systemId, {...})
list_metrics, get_metrics, create_metrics, update_metrics, delete_metrics	client.metrics.list(projectId), .get(id), .create(projectId, {...}), .update(id, {...}), .delete(id)
list_runs, get_runs, create_runs	client.runs.list(projectId), .get(id), .create(projectId, {...})
list_records, create_records, delete_records	client.records.list(runId), .create(runId, {...}), .delete(id)
upsert_scores	client.scores.upsert(metricConfigId, { recordId, ... })
list_annotations	client.records.annotations.list(recordId)
search_docs	Unchanged

Record tags, record assignees, and session attachments were never exposed as tools but are available through client.records.tags, client.records.assignees, and client.attachments.