Evals
An eval is a comparison. A suite holds items and the reference answers people actually gave; you send a prediction per item; Jean scores yours against theirs and keeps the answers.
Suites
GET /v1/suites lists the ones your organisation may submit to. Each says:
| Field | What it tells you |
|---|---|
suite_id | What to pass as suite_id when submitting. |
version | Bumped whenever the items or the reference answers change. |
metric | What the score measures — accuracy, recall@10, and so on. |
n_items | How many items the suite holds. |
per_item_feedback | Whether a result may say which items you got right. |
hfm_sha | The scorer commit that will produce your number. |
Held-out suites
On a held-out suite, a result gives you the aggregate number and nothing else: no per_item
array, and never a reference answer. That is not a limitation to work around — it is the only
reason the score is worth quoting. A suite you can read the answers to measures nothing.
On the suites that do allow per-item feedback, you get {"item_id": ..., "correct": true} per
item. Still no answers.
What the number means
A result carries value, the metric it is in, and the counts behind it:
n— items actually scored, meaning the ones your submission matched.n_submitted— how many predictions you sent.n_unmatched— predictions for items the suite does not contain.
If n is smaller than n_items, you did not answer the whole suite, and the value covers only
what you answered. Compare two runs only when n and suite_version agree.
Versions
Every result is stamped with hfm_sha, the exact scorer commit that computed it, and with
suite_id and suite_version. Two numbers are comparable when all three match and not
otherwise. A result that reads hfm_sha: "unpinned" came from the placeholder scorer that
exists so the job path can be exercised end to end; it is a number, but it is not a measurement.
Limits
Submissions are metered per organisation, per calendar month.
| Plan | Jobs at once | Eval items a month | Requests a minute |
|---|---|---|---|
| Free | 2 | 20,000 | 60 |
| Dev | 5 | 200,000 | 300 |
| Team | 20 | 2,000,000 | 1,000 |
GET /v1/usage reports where you are against them. Going over returns 429
with error.details naming which limit and when it resets.
Inline predictions are capped at 5 MB of JSON per request. Beyond that,
reserve an upload, PUT the bytes, and submit with input_ref instead.
The public board
Industry results are published separately, on the user simulation leaderboard. Ask us if you want your system run against it.