Jean Technologies Icon
Jean Technologies Text

Open evaluations

How well today’s models simulate a human being.

Overall

Mean score across the SOUL tasks each system was scored on, out of 100.

  1. Alyosha84.6
  2. GPT-6 (Astra)80.9
  3. Gemini 3.1 Pro80.6
  4. Claude Opus 579.7
  5. Gemini 3.8 Flash78.6
  6. Qwen 3.8 Flash76.7
  7. GLM 5.375.9
  8. OSim-4B75.7

Published boards

tau-USI scores how closely a model matches what one real person did in a conversation. SOUL-Index runs models across 23 tasks.

Talks like a person

tau-USI

/ · published table

SystemUSI
Human (inter-annotator)Mar 202692.7
DeepSeek-V3.1best general modelMar 202676.0
OSim-8BOdysSimJun 202675.4
OSim-4B-MidOdysSimJun 202672.6
OSim-Inst-8BOdysSimJun 202671.4
CoSER-8BMar 202667.2
OSim-8B-MidOdysSimJun 202667.1
OSim-4BOdysSimJun 202666.8
OSim-Inst-4BOdysSimJun 202664.1
UserLM-8BMar 202662.0
HumanLike-7BMar 202659.8
HumanLM-opinionMar 202646.9

Behaves like a person across 23 tasks

SOUL-Index (OdysSim)

/ · published table

SystemAverage
Claude Opus 4.7best general modelJun 202665.5
Ditto-v2-8BJun 202666.5
OSim-8B-Inst-PostJun 202666.0
OSim-8B-InstJun 202665.7
OSim-8B-Inst-Post (no distillation)Jun 202665.3
OSim-8BJun 202664.6
OSim-8B-PostJun 202663.8
OSim-4BJun 202662.6
OSim-4B-PostJun 202660.5
HumanLMJun 202648.7
OSim-8B-Inst-MidJun 202643.1
OSim-8B-MidJun 202641.1
SotopiaRL-7BJun 202639.7

Task by task

A social situation, then a question about it. Can the model read the room?

31 scored · 3 unreadable · line: the paper’s own OSim-8B score, 60.0

/

  • Gemini 3.1 Proreproduced +0.0 · parse 99% · 100 items · Sep 202679.0
  • Gemini 3.8 Flashprompted · parse 100% · 50 items · Sep 202678.0
  • Claude Opus 5prompted · parse 100% · 50 items · Sep 202678.0
  • Claude Fable 5.1prompted · parse 100% · 50 items · Sep 202678.0
  • GPT-6 (Astra)prompted · parse 100% · 50 items · Sep 202678.0
  • Grok 4.6prompted · parse 100% · 50 items · Sep 202674.0
  • GPT-5.5reproduced +2.0 · parse 100% · 100 items · Sep 202671.0
  • Qwen 3.8 Flashprompted · parse 100% · 100 items · Sep 202670.0
  • Claude Sonnet 5prompted · parse 100% · 50 items · Sep 202670.0
  • GLM 5.3 Flash (half-item arm)prompted · parse 100% · 50 items · Sep 202668.0
  • GLM 5.3prompted · parse 96% · 50 items · Sep 202668.0
  • DeepSeek Flashprompted · parse 100% · 50 items · Sep 202664.0
  • Kimi K2.6prompted · parse 96% · 50 items · Sep 202664.0
  • Ditto-v2-8Btrained · parse 100% · 100 items · Sep 202664.0
  • GLM 5.3 Flashprompted · parse 99% · 100 items · Sep 202663.0
  • OSim-8Breproduced +2.0 · parse 100% · 100 items · Sep 202662.0
  • Kimi K3prompted · parse 100% · 50 items · Sep 202662.0
  • OSim-4Btrained · parse 100% · 100 items · Sep 202662.0
  • OSim-Inst-4Btrained · parse 99% · 100 items · Sep 202662.0
  • OSim-Inst-8Btrained · parse 100% · 100 items · Sep 202661.0
0255075100

31 scored of 73 models in the registry ·