jev-benchmarks
Probability-aware evaluation for typed decision models: calibration, selective risk, latency, reproducible.
Benchmarks and evalsOutput verification
Home / Benchmarks, evals and calibration
Benchmarks, evals and calibration
27 entries
Useful for: Benchmarks and evals
24jevcal
Stop guessing thresholds: calibrate, threshold, and drift-check against an LLM teacher.
Benchmarks and evalsOutput verification
jev-rerank-bench
Jev against Cohere Rerank, ZeroEntropy, and a chat baseline on 14 datasets, raw responses included.
Benchmarks and evalsOutput verification
jev-search-rerank-eval
Does a Jev rerank beat embedding search? 9,831 graded pairs, with the judge-circularity bias measured.
Benchmarks and evalsSearch reranking
jev-sec-bench
Blind benchmarks for prompt injection and vulnerable-code detection.
Benchmarks and evalsOutput verification
jev-phishing-bench
Jev against Claude Haiku on 2,000 phishing emails: accuracy, calibration, latency, cost.
Benchmarks and evalsOutput verification
jev-spam-eval
Zero-shot spam filtering with Noul questions against TF-IDF baselines.
Benchmarks and evalsPlaygrounds and demos
jev-agent-failure-benchmark
Jev against a strong LLM on the Who and When agent-failure-attribution benchmark.
Benchmarks and evalsOutput verification
jev-korean-benchmark
Korean understanding and medical text, with runtime and cost evidence.
Benchmarks and evalsOutput verification
jev-report
Independent Chinese research report: 52 pages, 50 reproducible tests, 143 traceable data rows.
Benchmarks and evalsOutput verification
jev-benchmark
Two benchmarks, chess and predator identification, one inside Jev's lane and one outside, both with results.
Benchmarks and evalsPlaygrounds and demos
jev-dspy-lab
Reproducible calibration and selective-risk benchmarks for Jev decisions in DSPy.
Benchmarks and evalsOutput verification
jev-eval-agent
Personal-assistant agent with 100 mocked tools, measuring how many steps a Jev-gated agent needs.
Benchmarks and evalsTool gating and guardrails
jev-synergy-screening
Choice and Noul questions scored against ASReview SYNERGY gold labels for abstract screening.
Benchmarks and evalsOutput verification
jev-orderby-bench
Measures whether ORDER BY over a Jev probability is defensible: pairwise inversion, Score ordinality against a human grade, calibration, and wording invariants under a pre-registered gate; passes on 20 Newsgroups topics, fails four of six conditions on Amazon ESCI product relevance, and shows that a 40-row batched state through a DuckDB extension fails the ranking gate one row per request passes.
Benchmarks and evalsOutput verification
cultivar
Pinecone's skill-testing CLI, with a Jev grading backend it reports at about 30 times cheaper than the LLM grader.
Benchmarks and evalsOutput verification
jev-eval by 4esv
Jev against GPT-5.6 Terra on three labeled tasks: equal on the easy ones, 6.7 points lower on 77-way routing, 5 times faster, 41 to 50 times cheaper.
Benchmarks and evalsOutput verification
jev-benchmark by themsquared
Tool-call risk classification with the run-to-run variance reported; every wrong answer came with hedged confidence.
Benchmarks and evalsOutput verification
jev-research-eval
Reproducible harness over a pinned jev-ultrafast commit, with baseline and stress suites.
Benchmarks and evalsOutput verification
jev-playground by hegargarcia
Jev against other models in games with explicit states, legal actions, and a measurable outcome.
Benchmarks and evalsOutput verification
JevArena
Hosted arena that puts Jev against an opponent judge you connect and takes your vote before revealing which was which, with latency, cost provenance, and self-reported confidence; a vote records preference, not verified correctness.
Benchmarks and evalsPlaygrounds and demos
JevBench
Benchmark for Jev-class decision models: bounded rubric in, typed answer with a probability per option out, with cascade and committee experiments reported separately.
Benchmarks and evalsOutput verification
jev-align
Finds the examples a Jev function is least sure about, asks you to label them, and improves the function with GEPA.
Benchmarks and evalsClassification and extraction
jev-fanout-bench
Measures what a Jev request is billed and how its answers hold up under batching, translation, and rewording, from 3,455 billed requests with public raw logs.
Benchmarks and evalsOutput verification
Useful for: Output verification
2jev-harness
Confidence gates, shadow mode, recipes, and evals; reports Claude CLI at 48.9 s against Jev at 1.3 s on the same row-filter job.
Output verificationBenchmarks and evals
jevals
Runs a trace's agent, quality, and security evals as one batch of typed Jev questions instead of separate LLM-judge calls.
Output verificationBenchmarks and evals
Useful for: Playgrounds and demos
1jev-behavior-study
Controlled prompt experiments on jev-1.13.0, raw results and offline verification.
Playgrounds and demosBenchmarks and evals