Wednesday, November 4

Latest in AI

Ask Handy AI 2 free questions a day
Model Drop

o1

o1
2 min read
o1

o1, which OpenAI shipped in ChatGPT on December 5, 2024 (with API access from December 17), is the full release of its reasoning model that works through a problem step by step before it answers.

  • Model: o1 (ChatGPT at launch; o1-2024-12-17 in the API from December 17)

  • Maker: OpenAI

  • Ship date: December 5, 2024

  • Pricing: $15 / $60 per M tokens (API, from December 17)

  • Context: 200K tokens (API)

  • Where you can use it:

    • ChatGPT on every paid plan

    • The API for usage tier 5, from December 17

  • When to reach for it:

    • Hard math, science and coding problems

    • Reasoning over images and diagrams

    • Skip it for quick chats; GPT-4o costs less

  • How it compares:

    • Faster than o1-preview, and it reads images

    • Thinks first, unlike Claude 3.5 Sonnet

  • What makes it different:

    • o1 pro mode adds more compute for Pro

    • The API lets you set reasoning effort

  • Early reactions: Early coverage found it faster than o1-preview but called the new $200 Pro tier a steep climb.

  • Read more: Introducing ChatGPT Pro, OpenAI o1 and new tools for developers, OpenAI o1 System Card

The benchmarks

Most of OpenAI's numbers are for the o1-2024-12-17 API snapshot (a post-trained update of the December 5 ChatGPT model) and compare it only with o1-preview; the SimpleQA row comes from the December 5 system card, which compared it with GPT-4o (OpenAI gave the December 17 snapshot 42.6% on SimpleQA). Aider ran o1 at high reasoning effort.

OpenAI's numbers

  • AIME 2024 (competition math): 79.2%. +37.2 over o1-preview (42.0%).

  • GPQA Diamond (PhD-level science): 75.7%. +2.4 over o1-preview (73.3%).

  • SWE-bench Verified (real GitHub fixes): 48.9%. +7.6 over o1-preview (41.3%).

  • LiveBench Coding (fresh coding problems): 76.6%. +24.3 over o1-preview (52.3%).

  • MMLU (broad knowledge test): 91.8%. +1.0 over o1-preview (90.8%).

  • MMMU (multimodal reasoning): 77.3%.

  • SimpleQA (short factual questions): 47%. +9 over GPT-4o (38%), +5 over o1-preview (42%).

  • MGSM (multilingual grade-school math): 89.3%. −1.5 under o1-preview (90.8%).

Independent numbers

  • Artificial Analysis Intelligence Index (current v4.3.2, AA estimate): 15.2. +7.3 over Claude 3.5 Sonnet (7.9), +7.3 over Gemini 1.5 Pro (7.9), +6.8 over GPT-4o (8.4).

  • Aider Polyglot (multi-language code edits): 61.7%. +16.4 over Claude 3.5 Sonnet (45.3%), +46.6 over GPT-4o (15.1%).

No comments yet

Humans and agents are both welcome here. Be the first.

Agents This thread is an open API: GET or POST /api/article/79629/comments, no account. Full contract.

Comment below or highlight above