Created bymario-alka.dePowered bygodcore.denoob2claw.detricoma.de
DEEN
Sign in Sign up
Beta
vL
Modellserver / Inferenz-Engine

vLLM

Alle Performance-Benchmarks, die mit diesem Modellserver gemessen wurden – Durchsatz, Prefill und Latenz.

739Benchmarks
50Models
Best generation
1.608,5 tok/s
gemma-4-E2B-it
∅ Generation
265,9 tok/s
Schnitt aus 739 Läufen
Best prefill
55.260 tok/s
Prompt processing
Best TTFT
91 ms
Time to First Token

Top-Modelle auf vLLM

Bester Durchsatz je Modell – umschaltbar nach Generation, Prefill oder Kombiwert.

gemma-4-E2B-it5B
1.608,5 tok/s
Nemotron-3-Nano-4B4B
1.485,1 tok/s
gpt-oss-20b20B
1.443,9 tok/s
Qwen3-Coder-30B-A3B-Instruct30B
1.070,6 tok/s
Ornith-1.0-9B9B
1.062,5 tok/s
Qwen3.5-35B-A3B35B
1.028,6 tok/s
Qwen3-VL-30B-A3B-Instruct30B
1.028,5 tok/s
Meta-Llama-3.1-8B-Instruct8B
1.026,4 tok/s
gemma-4-E4B-it8B
1.022,8 tok/s
Ministral-3-14B-Reasoning-251214B
1.015,7 tok/s

Auf welcher Hardware laeuft vLLM am schnellsten?

Bester Durchsatz je Grafikkarte mit diesem Modellserver.

NVIDIA GeForce RTX 5070 Ti
1.608,5 tok/s
NVIDIA RTX A6000
1.568,0 tok/s
NVIDIA RTX PRO 6000 Blackwell Workstation Edition
1.443,9 tok/s
NVIDIA GeForce RTX 5090
1.386,8 tok/s
NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
1.238,3 tok/s
NVIDIA GeForce RTX 3090 Ti
901,9 tok/s
Performance profile

Durchsatz mit vLLM

Jede Blase steht für eine CPU, GPU bzw. ein Modell – Position: Prompt-Verarbeitung (X) × Ausgabe (Y), Blasengröße: Anzahl Messläufe.

CPUnach Prozessor

1.9111.5831.2559276008.56817.39026.21235.034Prefill (tok/s)Generation (tok/s)AMD Ryzen Threadripper PRO 5975WX 32-Cores - 1.608,5 tok/s Generation, 24.598 tok/s Prefill, TTFT 1.432 ms (3 Laufe)AMD Ryzen Threadrippe...AMD Ryzen Threadripper PRO 7955WX 16-Cores - 1.568,0 tok/s Generation, 30.199 tok/s Prefill, TTFT 1.039 ms (46 Laufe)AMD Ryzen Threadrippe...AMD Ryzen 9 9950X 16-Core Processor - 1.443,9 tok/s Generation, 26.424 tok/s Prefill, TTFT 1.242 ms (15 Laufe)AMD Ryzen 9 9950X 16-...AMD Ryzen 7 5800X3D 8-Core Processor - 1.386,8 tok/s Generation, 24.621 tok/s Prefill, TTFT 1.144 ms (9 Laufe)AMD Ryzen 7 5800X3D 8...AMD Ryzen Threadripper PRO 9965WX 24-Cores - 1.238,3 tok/s Generation, 17.768 tok/s Prefill, TTFT 1.023 ms (2 Laufe)AMD Ryzen Threadrippe...AMD Ryzen 9 8945HX with Radeon Graphics - 901,9 tok/s Generation, 13.403 tok/s Prefill, TTFT 6.591 ms (5 Laufe)AMD Ryzen 9 8945HX wi...
AMD Ryzen Threadripper PRO 5975WX 32-Cores 1.608,5 tok/sAMD Ryzen Threadripper PRO 7955WX 16-Cores 1.568,0 tok/sAMD Ryzen 9 9950X 16-Core Processor 1.443,9 tok/sAMD Ryzen 7 5800X3D 8-Core Processor 1.386,8 tok/sAMD Ryzen Threadripper PRO 9965WX 24-Cores 1.238,3 tok/sAMD Ryzen 9 8945HX with Radeon Graphics 901,9 tok/s

GPUnach Grafikkarte

1.9111.5831.2559276008.56817.39026.21235.034Prefill (tok/s)Generation (tok/s)NVIDIA GeForce RTX 5070 Ti - 1.608,5 tok/s Generation, 24.598 tok/s Prefill, TTFT 1.432 ms (3 Laufe)NVIDIA GeForce RTX 50...NVIDIA RTX A6000 - 1.568,0 tok/s Generation, 30.199 tok/s Prefill, TTFT 1.039 ms (46 Laufe)NVIDIA RTX A6000NVIDIA RTX PRO 6000 Blackwell Workstation Edition - 1.443,9 tok/s Generation, 26.424 tok/s Prefill, TTFT 1.242 ms (15 Laufe)NVIDIA RTX PRO 6000 B...NVIDIA GeForce RTX 5090 - 1.386,8 tok/s Generation, 24.621 tok/s Prefill, TTFT 1.144 ms (9 Laufe)NVIDIA GeForce RTX 50...NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition - 1.238,3 tok/s Generation, 17.768 tok/s Prefill, TTFT 1.023 ms (2 Laufe)NVIDIA RTX PRO 6000 B...NVIDIA GeForce RTX 3090 Ti - 901,9 tok/s Generation, 13.403 tok/s Prefill, TTFT 6.591 ms (5 Laufe)NVIDIA GeForce RTX 30...
NVIDIA GeForce RTX 5070 Ti 1.608,5 tok/sNVIDIA RTX A6000 1.568,0 tok/sNVIDIA RTX PRO 6000 Blackwell Workstation Edition 1.443,9 tok/sNVIDIA GeForce RTX 5090 1.386,8 tok/sNVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition 1.238,3 tok/sNVIDIA GeForce RTX 3090 Ti 901,9 tok/s

MODnach Modell

1.9661.5421.1176932692.91317.05131.19045.328Prefill (tok/s)Generation (tok/s)gemma-4-E2B-it - 1.608,5 tok/s Generation, 38.036 tok/s Prefill, TTFT 613 ms (8 Laufe)gemma-4-E2B-itNemotron-3-Nano-4B - 1.485,1 tok/s Generation, 33.698 tok/s Prefill, TTFT 805 ms (3 Laufe)Nemotron-3-Nano-4Bgpt-oss-20b - 1.443,9 tok/s Generation, 20.510 tok/s Prefill, TTFT 968 ms (10 Laufe)gpt-oss-20bQwen3-Coder-30B-A3B-Instruct - 1.070,6 tok/s Generation, 29.190 tok/s Prefill, TTFT 1.565 ms (6 Laufe)Qwen3-Coder-30B-A3B-I...Ornith-1.0-9B - 1.062,5 tok/s Generation, 20.011 tok/s Prefill, TTFT 1.127 ms (4 Laufe)Ornith-1.0-9BQwen3.5-35B-A3B - 1.028,6 tok/s Generation, 28.347 tok/s Prefill, TTFT 1.103 ms (4 Laufe)Qwen3.5-35B-A3BQwen3-VL-30B-A3B-Instruct - 1.028,5 tok/s Generation, 32.704 tok/s Prefill, TTFT 883 ms (4 Laufe)Qwen3-VL-30B-A3B-Inst...Meta-Llama-3.1-8B-Instruct - 1.026,4 tok/s Generation, 24.931 tok/s Prefill, TTFT 1.159 ms (3 Laufe)Meta-Llama-3.1-8B-Ins...gemma-4-E4B-it - 1.022,8 tok/s Generation, 18.744 tok/s Prefill, TTFT 2.204 ms (1 Lauf)gemma-4-E4B-itMinistral-3-14B-Reasoning-2512 - 1.015,7 tok/s Generation, 17.270 tok/s Prefill, TTFT 1.815 ms (3 Laufe)Ministral-3-14B-Reaso...Qwen3-30B-A3B-Instruct-2507 - 961,5 tok/s Generation, 26.280 tok/s Prefill, TTFT 2.845 ms (7 Laufe)Qwen3-30B-A3B-Instruc...Qwen3.6-35B-A3B - 952,8 tok/s Generation, 26.938 tok/s Prefill, TTFT 1.075 ms (4 Laufe)Qwen3.6-35B-A3BQwen3-Omni-30B-A3B-Thinking - 910,7 tok/s Generation, 33.907 tok/s Prefill, TTFT 1.527 ms (1 Lauf)Qwen3-Omni-30B-A3B-Th...Qwen3-30B-A3B-Thinking-2507 - 906,0 tok/s Generation, 28.029 tok/s Prefill, TTFT 3.190 ms (6 Laufe)Qwen3-30B-A3B-Thinkin...Gemma-4-26B-A4B-it - 841,8 tok/s Generation, 28.153 tok/s Prefill, TTFT 1.058 ms (4 Laufe)Gemma-4-26B-A4B-itLaguna-XS-2.1 - 836,4 tok/s Generation, 34.791 tok/s Prefill, TTFT 1.159 ms (4 Laufe)Laguna-XS-2.1Qwen-AgentWorld-35B-A3B - 799,7 tok/s Generation, 30.088 tok/s Prefill, TTFT 894 ms (2 Laufe)Qwen-AgentWorld-35B-A...Magistral-Small-2509 - 726,9 tok/s Generation, 10.204 tok/s Prefill, TTFT 2.619 ms (1 Lauf)Magistral-Small-2509Devstral-Small-2-24B-Instruct-2512 - 725,6 tok/s Generation, 11.872 tok/s Prefill, TTFT 2.430 ms (1 Lauf)Devstral-Small-2-24B-...gemma-4-12B-it - 681,8 tok/s Generation, 16.224 tok/s Prefill, TTFT 1.899 ms (1 Lauf)gemma-4-12B-itNorth-Mini-Code-1.0 - 648,5 tok/s Generation, 29.491 tok/s Prefill, TTFT 1.310 ms (1 Lauf)North-Mini-Code-1.0Qwen3-Coder-Next - 629,8 tok/s Generation, 26.016 tok/s Prefill, TTFT 1.347 ms (1 Lauf)Qwen3-Coder-NextOrnith-1.5-35B-A3B - 626,4 tok/s Generation, 28.681 tok/s Prefill, TTFT 928 ms (1 Lauf)Ornith-1.5-35B-A3B
gemma-4-E2B-it 1.608,5 tok/sNemotron-3-Nano-4B 1.485,1 tok/sgpt-oss-20b 1.443,9 tok/sQwen3-Coder-30B-A3B-Instruct 1.070,6 tok/sOrnith-1.0-9B 1.062,5 tok/sQwen3.5-35B-A3B 1.028,6 tok/sQwen3-VL-30B-A3B-Instruct 1.028,5 tok/sMeta-Llama-3.1-8B-Instruct 1.026,4 tok/sgemma-4-E4B-it 1.022,8 tok/sMinistral-3-14B-Reasoning-2512 1.015,7 tok/sQwen3-30B-A3B-Instruct-2507 961,5 tok/sQwen3.6-35B-A3B 952,8 tok/sQwen3-Omni-30B-A3B-Thinking 910,7 tok/sQwen3-30B-A3B-Thinking-2507 906,0 tok/sGemma-4-26B-A4B-it 841,8 tok/sLaguna-XS-2.1 836,4 tok/sQwen-AgentWorld-35B-A3B 799,7 tok/sMagistral-Small-2509 726,9 tok/sDevstral-Small-2-24B-Instruct-2512 725,6 tok/sgemma-4-12B-it 681,8 tok/sNorth-Mini-Code-1.0 648,5 tok/sQwen3-Coder-Next 629,8 tok/sOrnith-1.5-35B-A3B 626,4 tok/s
All measurements

80 Benchmark-Läufe

Jeder veroeffentlichte Lauf mit vLLM.

← Alle Engines
Metric:
Size
#Model / MakerMetricsParallelGPU / CPU / RAMRuntime
1gemma-4-E2B-it5BGoogle Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.608,46 tok/s TG
Prefill 31.452 · TTFT 1.633 ms
10×NVIDIA GeForce RTX 5070 TiAMD Ryzen Threadripper PRO 5975WX 32-Cores · 247 GB RAMvLLMgodclawDetails →
2gemma-4-E2B-it5BGoogle Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.567,98 tok/s TG
Prefill 55.260 · TTFT 492 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
3Nemotron-3-Nano-4B4BNVIDIA Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.485,14 tok/s TG
Prefill 44.171 · TTFT 717 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
4gpt-oss-20b20BOpenAI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.443,88 tok/s TG
Prefill 35.348 · TTFT 560 ms
10×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawDetails →
5gpt-oss-20b20BOpenAI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.386,79 tok/s TG
Prefill 19.981 · TTFT 1.205 ms
10×NVIDIA GeForce RTX 5090AMD Ryzen 7 5800X3D 8-Core Processor · 126 GB RAMvLLMcodex_cliDetails →
6gemma-4-E2B-it5BGoogle Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.359,09 tok/s TG
Prefill 47.158 · TTFT 587 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
7gpt-oss-20b20BOpenAI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.295,30 tok/s TG
Prefill 25.462 · TTFT 808 ms
10×NVIDIA GeForce RTX 5090AMD Ryzen 7 5800X3D 8-Core Processor · 126 GB RAMvLLMcodex_cliAWQDetails →
8gpt-oss-20b20BOpenAI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.238,26 tok/s TG
Prefill 21.843 · TTFT 905 ms
10×3x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation EditionAMD Ryzen Threadripper PRO 9965WX 24-Cores · 125 GB RAMvLLMgodclawDetails →
9gemma-4-E2B-it5BGoogle Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.122,50 tok/s TG
Prefill 38.712 · TTFT 690 ms
10×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMopenclaw_cliFP8Details →
10Qwen3-Coder-30B-A3B-Instruct30BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.070,56 tok/s TG
Prefill 33.741 · TTFT 976 ms
10×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawAWQDetails →
11Ornith-1.0-9B9BDeepReinforce Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.062,52 tok/s TG
Prefill 21.965 · TTFT 1.226 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
12Qwen3.5-35B-A3B35BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.028,59 tok/s TG
Prefill 24.461 · TTFT 1.670 ms
10×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawAWQDetails →
13Qwen3-VL-30B-A3B-Instruct30BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.028,53 tok/s TG
Prefill 36.884 · TTFT 916 ms
10×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawAWQDetails →
14Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.026,44 tok/s TG
Prefill 26.805 · TTFT 1.371 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
15gemma-4-E4B-it8BGoogle Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.022,79 tok/s TG
Prefill 18.744 · TTFT 2.204 ms
10×NVIDIA GeForce RTX 5070 TiAMD Ryzen Threadripper PRO 5975WX 32-Cores · 247 GB RAMvLLMgodclawDetails →
16Ministral-3-14B-Reasoning-251214BMistral AI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
1.015,69 tok/s TG
Prefill 20.061 · TTFT 1.289 ms
10×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawAWQDetails →
17gemma-4-E2B-it5BGoogle Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
973,54 tok/s TG
Prefill 23.599 · TTFT 460 ms
5×NVIDIA GeForce RTX 5070 TiAMD Ryzen Threadripper PRO 5975WX 32-Cores · 247 GB RAMvLLMgodclawDetails →
18Qwen3-Coder-30B-A3B-Instruct30BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
966,21 tok/s TG
Prefill 33.768 · TTFT 1.793 ms
10×NVIDIA GeForce RTX 5090AMD Ryzen 7 5800X3D 8-Core Processor · 126 GB RAMvLLMcodex_cliAWQDetails →
19Qwen3-30B-A3B-Instruct-250730BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
961,53 tok/s TG
Prefill 32.305 · TTFT 1.633 ms
10×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawAWQDetails →
20Qwen3.6-35B-A3B35BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
952,78 tok/s TG
Prefill 20.106 · TTFT 1.542 ms
10×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawAWQDetails →
21Qwen3-Omni-30B-A3B-Thinking30BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
910,74 tok/s TG
Prefill 33.907 · TTFT 1.527 ms
10×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawAWQDetails →
22Qwen3-30B-A3B-Instruct-250730BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
909,25 tok/s TG
Prefill 37.116 · TTFT 1.783 ms
10×NVIDIA GeForce RTX 5090AMD Ryzen 7 5800X3D 8-Core Processor · 126 GB RAMvLLMcodex_cliAWQDetails →
23gemma-4-E2B-it5BGoogle Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
909,00 tok/s TG
Prefill 43.093 · TTFT 288 ms
5×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
24Qwen3-30B-A3B-Thinking-250730BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
906,04 tok/s TG
Prefill 39.029 · TTFT 1.193 ms
10×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawAWQDetails →
25Qwen3-30B-A3B-Instruct-250730BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
901,86 tok/s TG
Prefill 12.965 · TTFT 12.663 ms
10×NVIDIA GeForce RTX 3090 TiAMD Ryzen 9 8945HX with Radeon Graphics · 92 GB RAMvLLMwebsocketAWQDetails →
26Nemotron-3-Nano-4B4BNVIDIA Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
899,41 tok/s TG
Prefill 30.214 · TTFT 473 ms
5×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
27Qwen3-30B-A3B-Thinking-250730BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
883,71 tok/s TG
Prefill 12.981 · TTFT 12.917 ms
10×NVIDIA GeForce RTX 3090 TiAMD Ryzen 9 8945HX with Radeon Graphics · 92 GB RAMvLLMwebsocketAWQDetails →
28Ornith-1.0-9B9BDeepReinforce Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
881,75 tok/s TG
Prefill 22.565 · TTFT 1.184 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
29Qwen3-30B-A3B-Thinking-250730BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
873,72 tok/s TG
Prefill 33.023 · TTFT 1.835 ms
10×NVIDIA GeForce RTX 5090AMD Ryzen 7 5800X3D 8-Core Processor · 126 GB RAMvLLMcodex_cliAWQDetails →
30gpt-oss-20b20BOpenAI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
842,94 tok/s TG
Prefill 31.098 · TTFT 319 ms
5×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawDetails →
31Gemma-4-26B-A4B-it26BGoogle Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
841,75 tok/s TG
Prefill 32.027 · TTFT 923 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
32Laguna-XS-2.1Poolside Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
836,37 tok/s TG
Prefill 39.157 · TTFT 1.033 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
33Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
832,96 tok/s TG
Prefill 27.508 · TTFT 1.327 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
34gpt-oss-20b20BOpenAI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
803,96 tok/s TG
Prefill 16.236 · TTFT 616 ms
5×NVIDIA GeForce RTX 5090AMD Ryzen 7 5800X3D 8-Core Processor · 126 GB RAMvLLMcodex_cliDetails →
35gemma-4-E2B-it5BGoogle Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
803,18 tok/s TG
Prefill 35.757 · TTFT 347 ms
5×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
36Qwen3.6-35B-A3B35BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
800,39 tok/s TG
Prefill 32.313 · TTFT 827 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
37Qwen-AgentWorld-35B-A3B35BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
799,65 tok/s TG
Prefill 31.954 · TTFT 834 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
38gpt-oss-20b20BOpenAI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
797,18 tok/s TG
Prefill 11.820 · TTFT 1.789 ms
10×NVIDIA GeForce RTX 3090 TiAMD Ryzen 9 8945HX with Radeon Graphics · 92 GB RAMvLLMwebsocketDetails →
39gpt-oss-20b20BOpenAI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
796,82 tok/s TG
Prefill 11.892 · TTFT 1.781 ms
10×NVIDIA GeForce RTX 3090 TiAMD Ryzen 9 8945HX with Radeon Graphics · 92 GB RAMvLLMwebsocketDetails →
40Nemotron-3-Nano-4B4BNVIDIA Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
791,54 tok/s TG
Prefill 26.710 · TTFT 1.226 ms
10×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMopenclaw_cliFP8Details →
41gpt-oss-20b20BOpenAI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
790,09 tok/s TG
Prefill 13.693 · TTFT 1.142 ms
5×3x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation EditionAMD Ryzen Threadripper PRO 9965WX 24-Cores · 125 GB RAMvLLMgodclawDetails →
42Qwen3-Coder-30B-A3B-Instruct30BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
789,24 tok/s TG
Prefill 37.676 · TTFT 952 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
43Qwen3.5-35B-A3B35BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
787,25 tok/s TG
Prefill 32.522 · TTFT 826 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
44Ornith-1.0-9B9BDeepReinforce Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
786,19 tok/s TG
Prefill 19.249 · TTFT 1.376 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
45gpt-oss-20b20BOpenAI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
784,45 tok/s TG
Prefill 17.724 · TTFT 560 ms
5×NVIDIA GeForce RTX 5090AMD Ryzen 7 5800X3D 8-Core Processor · 126 GB RAMvLLMcodex_cliAWQDetails →
46Laguna-XS-2.1Poolside Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
771,80 tok/s TG
Prefill 34.194 · TTFT 1.184 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
47Qwen3-VL-30B-A3B-Instruct30BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
763,50 tok/s TG
Prefill 35.900 · TTFT 987 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
48Qwen3-30B-A3B-Instruct-250730BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
758,69 tok/s TG
Prefill 34.913 · TTFT 992 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
49Gemma-4-26B-A4B-it26BGoogle Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
754,69 tok/s TG
Prefill 26.306 · TTFT 1.120 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
50Qwen3-30B-A3B-Thinking-250730BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
751,67 tok/s TG
Prefill 32.340 · TTFT 1.271 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
51Ministral-3-14B-Reasoning-251214BMistral AI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
735,14 tok/s TG
Prefill 16.886 · TTFT 1.945 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
52Qwen-AgentWorld-35B-A3B35BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
732,27 tok/s TG
Prefill 28.221 · TTFT 955 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
53Qwen3.6-35B-A3B35BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
729,28 tok/s TG
Prefill 27.628 · TTFT 966 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
54Magistral-Small-250924BMistral AI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
726,94 tok/s TG
Prefill 10.204 · TTFT 2.619 ms
10×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawAWQDetails →
55Devstral-Small-2-24B-Instruct-251224BMistral AI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
725,58 tok/s TG
Prefill 11.872 · TTFT 2.430 ms
10×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawAWQDetails →
56Qwen3-Coder-30B-A3B-Instruct30BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
720,92 tok/s TG
Prefill 30.926 · TTFT 1.138 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
57Qwen3.5-35B-A3B35BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
718,47 tok/s TG
Prefill 28.493 · TTFT 951 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
58Laguna-XS-2.1Poolside Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
712,04 tok/s TG
Prefill 35.143 · TTFT 1.137 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
59Qwen3-VL-30B-A3B-Instruct30BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
697,04 tok/s TG
Prefill 30.227 · TTFT 1.164 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
60Qwen3-30B-A3B-Instruct-250730BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
691,32 tok/s TG
Prefill 30.805 · TTFT 1.147 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
61Gemma-4-26B-A4B-it26BGoogle Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
690,88 tok/s TG
Prefill 29.513 · TTFT 1.000 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
62gemma-4-12B-it12BGoogle Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
681,82 tok/s TG
Prefill 16.224 · TTFT 1.899 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
63Qwen3-Coder-30B-A3B-Instruct30BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
676,76 tok/s TG
Prefill 21.675 · TTFT 725 ms
5×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawAWQDetails →
64Meta-Llama-3.1-8B-Instruct8BMeta Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
675,29 tok/s TG
Prefill 20.479 · TTFT 781 ms
5×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
65Qwen3-30B-A3B-Thinking-250730BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
673,19 tok/s TG
Prefill 30.512 · TTFT 1.156 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
66Qwen3-Coder-30B-A3B-Instruct30BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
670,65 tok/s TG
Prefill 17.357 · TTFT 3.808 ms
10×NVIDIA GeForce RTX 3090 TiAMD Ryzen 9 8945HX with Radeon Graphics · 92 GB RAMvLLMwebsocketAWQDetails →
67Qwen3-VL-30B-A3B-Instruct30BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
666,52 tok/s TG
Prefill 27.804 · TTFT 465 ms
5×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawAWQDetails →
68Qwen3.6-35B-A3B35BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
664,59 tok/s TG
Prefill 27.704 · TTFT 965 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
69Qwen3.5-35B-A3B35BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
658,68 tok/s TG
Prefill 27.910 · TTFT 964 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
70Qwen3-30B-A3B-Instruct-250730BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
655,83 tok/s TG
Prefill 17.990 · TTFT 931 ms
5×NVIDIA GeForce RTX 5090AMD Ryzen 7 5800X3D 8-Core Processor · 126 GB RAMvLLMcodex_cliAWQDetails →
71Laguna-XS-2.1Poolside Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
655,42 tok/s TG
Prefill 30.671 · TTFT 1.283 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
72Ministral-3-14B-Reasoning-251214BMistral AI Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
655,08 tok/s TG
Prefill 14.863 · TTFT 2.211 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
73Qwen3-30B-A3B-Instruct-250730BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
649,04 tok/s TG
Prefill 17.868 · TTFT 764 ms
5×NVIDIA RTX PRO 6000 Blackwell Workstation EditionAMD Ryzen 9 9950X 16-Core Processor · 92 GB RAMvLLMgodclawAWQDetails →
74North-Mini-Code-1.0Cohere Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
648,52 tok/s TG
Prefill 29.491 · TTFT 1.310 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
75Ornith-1.0-9B9BDeepReinforce Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
647,82 tok/s TG
Prefill 16.265 · TTFT 725 ms
5×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
76Qwen3-30B-A3B-Thinking-250730BQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
633,41 tok/s TG
Prefill 20.292 · TTFT 767 ms
5×NVIDIA GeForce RTX 5090AMD Ryzen 7 5800X3D 8-Core Processor · 126 GB RAMvLLMcodex_cliAWQDetails →
77gemma-4-E2B-it5BGoogle Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
631,69 tok/s TG
Prefill 29.259 · TTFT 405 ms
5×NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMopenclaw_cliFP8Details →
78Gemma-4-26B-A4B-it26BGoogle Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
630,94 tok/s TG
Prefill 24.767 · TTFT 1.191 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →
79Qwen3-Coder-NextQwen (Alibaba) Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
629,83 tok/s TG
Prefill 26.016 · TTFT 1.347 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawAWQDetails →
80Ornith-1.5-35B-A3B35BDeepReinforce Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
626,41 tok/s TG
Prefill 28.681 · TTFT 928 ms
10×2x NVIDIA RTX A6000AMD Ryzen Threadripper PRO 7955WX 16-Cores · 184 GB RAMvLLMgodclawFP8Details →