Test a Jev decision from a coding agent.

JevBench evaluates a question against labeled examples. The agent can inspect disagreements and write the next question. JevBench does not rewrite the question.

What it does

Four tools share the same account, decision, $9 pass, and 500 successful evaluations as the website: jevbench_status, jevbench_prepare, jevbench_run, and jevbench_result.

Create a key

Sign in, open your account, and create a key. The raw key is shown once. JevBench stores only its hash. Set it as JEVBENCH_API_KEY in the MCP client. Do not commit it.

Configure the server

The server is mcp/stdio.mjs in this repository. It speaks MCP over stdin and calls https://jevbench.app. That protocol was verified against production. The snippets below match each product’s current local-server config. Keep the API key outside the repository.

Cursor

In .cursor/mcp.json, outside the repository if the key is present:

{
  "mcpServers": {
    "jevbench": {
      "command": "node",
      "args": ["/absolute/path/to/jevbench/mcp/stdio.mjs"],
      "env": { "JEVBENCH_API_KEY": "jvb_…" }
    }
  }
}

Claude Code

claude mcp add jevbench --env JEVBENCH_API_KEY=jvb_… -- node /absolute/path/to/jevbench/mcp/stdio.mjs

Codex

In ~/.codex/config.toml:

[mcp_servers.jevbench]
command = "node"
args = ["/absolute/path/to/jevbench/mcp/stdio.mjs"]

[mcp_servers.jevbench.env]
JEVBENCH_API_KEY = "jvb_…"

Test a decision

Ask the agent to call jevbench_prepare with the question and 10–50 examples that include both YES and NO. Then jevbench_run with type baseline. Poll jevbench_result with the returned run_id until status is complete or partial. A completed result is evidence on this sample. It is not a production-readiness claim.

When payment is required

An unpaid run returns PAYMENT_REQUIRED, the decision id, and a Checkout URL for the $9 one-time pass. The person opens that URL and pays. The agent keeps calling jevbench_status. The pass exists only after the verified Stripe webhook settles. A loaded success page is not payment.

What consumes an evaluation

One successful Jev example evaluation consumes 1. A failed row consumes 0. Reading a result consumes 0. Moving a saved decision line consumes 0. Running the same examples again, as a revision or a fresh check, consumes one evaluation per success. The pass covers 500 successes for one decision.

Revoke a key

Revoke it on the account page. The next tool call is rejected. Create a new key and replace JEVBENCH_API_KEY.