Model Drop

Grok 3

February 17, 2025 2 min read

Grok 3, which xAI released on February 17, 2025, is xAI's flagship chat model, with an optional Think mode that reasons step by step before it answers.

Model: Grok 3 Beta, with Think and DeepSearch modes (no API at launch)
Maker: xAI
Ship date: February 17, 2025
Pricing: X Premium+ and SuperGrok only (API came in April)
Context: 1M tokens (xAI's launch claim)
Where you can use it:
- Grok.com and the Grok apps
- X for Premium+ subscribers
When to reach for it:
- Math and science with Think turned on
- Skip it if you need an API right now
How it compares:
- Trades blows with o3-mini (high) and o1
- Ahead of DeepSeek R1 on most tests
What makes it different:
- Claims 10x the compute of prior top models
- Shows its full reasoning, unlike o1
Early reactions: Andrej Karpathy put Think near OpenAI's best, but xAI's charts drew cherry-picking complaints within days.
Read more: Grok 3 Beta: The Age of Reasoning Agents

The benchmarks

The first five rows are Grok 3 Beta (Think) at one try per problem (xAI's headline 93.3% on AIME 2025 used cons@64, a majority vote over 64 tries, which its chart didn't show for rivals); the last two are Grok 3 Beta without thinking, where xAI's table column "Gemini 2.0" carries Google's published Gemini 2.0 Pro scores. Both models were still in training.

xAI's numbers

AIME 2025 (math contest, Think mode): 77.3%. −9.2 under o3-mini (high) (86.5%), −1.7 under o1 (medium) (79%), +7.3 over DeepSeek R1 (70%), +23.8 over Gemini 2.0 Flash Thinking (53.5%).
AIME 2024 (math contest, Think mode): 83.9%. −3.4 under o3-mini (high) (87.3%), +9.5 over o1 (74.4%), +4.1 over DeepSeek R1 (79.8%).
GPQA Diamond (grad-level science, Think mode): 80.2%. +0.5 over o3-mini (high) (79.7%), +2.2 over o1 (78%), +8.7 over DeepSeek R1 (71.5%).
LiveCodeBench (coding problems, Think mode): 70.6%. −3.5 under o3-mini (high) (74.1%), −2.3 under o1 (high) (72.9%), +24.8 over Gemini 2.0 Flash Thinking (45.8%).
MMMU (image understanding, Think mode): 76%. −2.2 under o1 (78.2%), +0.6 over Gemini 2.0 Flash Thinking (75.4%).
MMLU-Pro (broad knowledge, no thinking): 79.9%. +7.3 over GPT-4o (72.6%), +4.0 over DeepSeek V3 (75.9%), +1.9 over Claude 3.5 Sonnet (78.0%), +0.8 over Gemini 2.0 Pro (79.1%).
SimpleQA (factual recall, no thinking): 43.6%. +5.4 over GPT-4o (38.2%), +18.7 over DeepSeek V3 (24.9%), −0.7 under Gemini 2.0 Pro (44.3%).

Independent numbers

Artificial Analysis Intelligence Index (current v4.3.2, AA estimate): 10.4. −4.8 under o1 (15.2), −0.6 under o3-mini (high) (11.0), −1.0 under DeepSeek R1 (11.4), +1.7 over Gemini 2.0 Pro (8.7).
LMArena text Elo (at debut, as "chocolate"): 1402. +17 over Gemini 2.0 Flash Thinking (1385), +25 over GPT-4o (1377).