How to choose a threshold for Jev decisions
Jev gives you a probability for a YES/NO question. Your application still needs to decide when that probability is high enough to act. The minimum score that triggers YES is your threshold.
Start with a decision you can label
For a fictional email SaaS, ask: “Is this person currently experiencing an email marketing problem our product can solve?” Label “Our emails keep landing in spam” as YES and “We just launched our marketing SaaS” as NO.
Gather at least 50 examples if you can. Include ordinary negatives, borderline cases, and real positives. Label them before looking at Jev’s scores, so its answers do not influence yours.
Watch two kinds of mistakes
- Wrong alert (false positive): Jev flags a person who does not have the problem. A sales team wastes time reaching out.
- Missed positive (false negative): Jev rejects a person who really needs help. A useful conversation never starts.
Lower cutoffs catch more positives and can let in more noise. Higher cutoffs reduce wrong alerts and can miss more positives. Precision itself does not have to improve at every higher cutoff; inspect your data.
Read the counts before the percentages
Imagine 100 examples: 50 YES and 50 NO. At a cutoff of 0.60, Jev accepts 50 items. Of those, 44 are correct and 6 are wrong. It misses the other 6 true positives.
| Result | Count |
|---|---|
| Correct positives | 44 |
| Wrong alerts | 6 |
| Missed positives | 6 |
| Correct rejections | 44 |
Precision is 88%: 44 of the 50 accepted items are right. Recall is 88%: Jev found 44 of the 50 real positives. These values describe this example dataset, not a benchmark of Jev.
Choose the tradeoff your product needs
JevBench’s balanced recommendation maximizes F1, a combined precision/recall score. It is a useful starting point when wrong alerts and missed positives both matter. It is not a business cost model.
If a wrong alert is expensive, inspect the “Fewer wrong alerts” option and check how many positives it misses. If missing a positive is expensive, inspect “Catch more positives” and check how much review work the extra alerts create.
Do not choose by accuracy alone. A model that always says NO is 95% accurate on data that is 95% negative, but it finds none of the positives.
Inspect the confident mistakes
A score near 0.50 may be a borderline example. A score of 0.95 on a clear NO example may reveal a flaw in the question, missing context, or an incorrect label. Read these inputs closely before adjusting the cutoff again.
Validate on fresh examples before shipping
You chose the cutoff using this dataset, so performance on it can be optimistic. Freeze the question and threshold, then test a separate set of examples you did not use for tuning. Small datasets and missing classes cannot establish dependable performance.
Keep the model version with your results. If the question, input format, or model changes, rerun the test. JevBench keeps each run’s dataset snapshot and the served model version for that purpose.
The official OpenRouter Jev tutorial documents the binary probability returned by the Decisions API.