API / MCP

Your decision. Any supported model.

Use the same models, input, and criteria in the Playground, API, and MCP. Test candidates on the same examples before changing your workflow.

Find models

GET /api/v1/models is public and free. It refreshes from OpenRouter's Decisions catalog hourly, without a new deployment. It returns model IDs, pinned builds, context limits, provider prices, and whether the catalog is live or using its bundled fallback.

curl https://jevbench.app/api/v1/models

Text-capable decision models with a context of at least 32K, input pricing at or below $0.20 per million tokens, no output-token charge, and no per-request fee are eligible. Experimental and expired entries are excluded. A listing establishes API compatibility, not accuracy on your data.

The playground shows a decision and the request that produced it. POST /api/v1/call sends that request. MCP sends it with jevbench_call. One successful call uses one of 500. $9 one time adds those 500. There is no subscription, and there is not a second free allowance for the API.

Creating an API key is free. Connecting MCP is free. The key is not a purchase. A purchase already on the account keeps its 500. This call spends from that same balance.

Call it from your app

The key is a bearer token. Send a model from the catalog, a text or JSON state, and up to 8 typed questions. The response includes the requested model, actual served build, probabilities, latency in milliseconds, and provider-reported usage and cost when available. Provider cost is separate from JevBench credits.

curl -X POST https://jevbench.app/api/v1/call \
  -H "Authorization: Bearer $JEVBENCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d @request.json
{
  "model": "perplexity/pplx-decider-v1.2",
  "state": {
    "text": "Our cart emails keep landing in spam. Looking for another tool."
  },
  "questions": {
    "buyer": {
      "type": "noul",
      "instructions": "Is the author actively seeking a replacement product?",
      "criteria": {
        "true": "They ask for a replacement for their current tool.",
        "false": "They do not ask for a replacement."
      }
    }
  }
}

Or ask a coding agent

Connect your MCP client to https://jevbench.app/mcp using Streamable HTTP. Set its Authorization header to Bearer YOUR_JEVBENCH_API_KEY. Ask for jevbench_models first, then jevbench_call. Model discovery is free; evaluation uses the same balance as the API. Full structured answers remain available to the agent.

For clients using a local process, mcp/stdio.mjs exposes the same tools and calls the public API. The configurations below are for that local option.

.cursor/mcp.json

{
  "mcpServers": {
    "jevbench": {
      "command": "node",
      "args": ["/absolute/path/to/jevbench/mcp/stdio.mjs"],
      "env": { "JEVBENCH_API_KEY": "jvb_…" }
    }
  }
}

Labeled checks

The older labeled workflow is still on the same key and the same 500. Its tools are jevbench_status, jevbench_prepare, jevbench_run, and jevbench_result. Preparing is free. One decision can run a free early look of 10 successful examples. Those successes count toward the 500. A failed row uses none. Reading a result uses none.

Create a key

Sign in, open your account, and create a key. That key is free. The raw key is shown once. JevBench stores only its hash. Set it as JEVBENCH_API_KEY. Do not commit it, and do not put it in browser code.

Prepare a labeled check

Send PUT /api/v1/decision with the question and 10–50 labeled examples, including both YES and NO. The block below is the shape, not a complete set.

curl -X PUT https://jevbench.app/api/v1/decision \
  -H "Authorization: Bearer $JEVBENCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "question": "Does this person currently have an email problem?",
    "examples": [
      {
        "input": "Our emails keep landing in spam.",
        "expected": true
      },
      {
        "input": "Thanks, the last invoice looks right.",
        "expected": false
      }
    ]
  }'

Then POST /api/v1/runs with {"type":"baseline","model":"typesafe/jev-1.14"}. The free early look scores up to 10 examples. The selected build is pinned. Revisions and fresh validation inherit that model so thresholds remain comparable. Omit model on the baseline to use the existing server default. Results include the run model and actual served builds. Use separate /api/v1/call requests to explore other models on identical inputs.

Test a decision

Ask the agent to call jevbench_prepare with the question and 10–50 examples that include both YES and NO. Then jevbench_run with type baseline. The free early look scores up to 10 examples. Poll jevbench_result with the returned run_id. Reading the result does not spend an evaluation. A completed early look is evidence on that sample. It is not a production cutoff.

When payment is required

After the early look, a revision, a fresh check, or the rest of a larger set returns PAYMENT_REQUIRED. The message says the free early look used its examples, Jev has not evaluated the rest, and $9 unlocks the rest of this decision workflow. It includes a Checkout URL. The person opens that URL and pays. Opening it is not payment. The pass exists only after the verified Stripe webhook settles.

What consumes an evaluation

One successful POST /api/v1/call consumes 1 of the 500. One successful labeled example also consumes 1, including successes from the free early look. A failed call or row consumes 0. Reading a result consumes 0. The pass is not unlimited.

Revoke a key

Revoke it on the account page. The next tool call is rejected. Create a new key and replace JEVBENCH_API_KEY.