Created bymario-alka.dePowered bygodcore.denoob2claw.detricoma.de
DEEN
Sign in Sign up
Beta
microsoft

Phi-3.5-mini-instruct3.8B

Benchmark profile and published results.

Dense3.8B
Position in the field

Best values compared

Best metrics of this model against the minimum, average and maximum of all published systems.

AMD Radeon RX 7900 XTXAll GPUs
Generation302,0 tok/s
Min 0,1Ø 159,1Max 717,6
Ueber dem Durchschnitt · 361 Systeme im Feld
Prefill19.406 tok/s
Min 39Ø 5.819Max 52.503
Ueber dem Durchschnitt · 361 Systeme im Feld
Time to First Token380 ms
Min 156Ø 18.353Max 354.928
Ueber dem Durchschnitt · 361 Systeme im Feld
Performance profile

Throughput by hardware & engine

Each bubble is a GPU, CPU or engine – position shows prompt processing (X) and output speed (Y), bubble size the number of runs.

GPUby graphics card

3323173022872728.0128.3538.6949.035Prefill (tok/s)Generation (tok/s)AMD Radeon RX 7900 XTX - 302,0 tok/s Generation, 8.523 tok/s Prefill, TTFT 9.068 ms (6 Laufe)AMD Radeon RX 7900 XTX
AMD Radeon RX 7900 XTX 302,0 tok/s

CPUby processor

3383132872622376635.90311.14416.384Prefill (tok/s)Generation (tok/s)AMD Ryzen Threadripper PRO 3955WX 16-Cores - 302,0 tok/s Generation, 13.699 tok/s Prefill, TTFT 1.150 ms (3 Laufe)AMD Ryzen Threadrippe...Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz - 272,9 tok/s Generation, 3.348 tok/s Prefill, TTFT 16.986 ms (3 Laufe)Intel(R) Xeon(R) CPU ...
AMD Ryzen Threadripper PRO 3955WX 16-Cores 302,0 tok/sIntel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz 272,9 tok/s

ENGby engine

3383132872622376635.90311.14416.384Prefill (tok/s)Generation (tok/s)vLLM - 302,0 tok/s Generation, 13.699 tok/s Prefill, TTFT 1.150 ms (3 Laufe)vLLMllama.cpp - 272,9 tok/s Generation, 3.348 tok/s Prefill, TTFT 16.986 ms (3 Laufe)llama.cpp
vLLM 302,0 tok/sllama.cpp 272,9 tok/s
Throughput & latency

Performance benchmark

Metric:
#Model / MakerMetricsParallelGPU / CPU / RAMRuntime
1Phi-3.5-mini-instruct3.8Bmicrosoft Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
301,98 tok/s TG
Prefill 19.406 · TTFT 1.945 ms
10×2x AMD Radeon RX 7900 XTXAMD Ryzen Threadripper PRO 3955WX 16-Cores · 63 GB RAMvLLMgodclawDetails →
2Phi-3.5-mini-instruct3.8Bmicrosoft Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
272,93 tok/s TG
Prefill 691 · TTFT 38.379 ms
10×2x AMD Radeon RX 7900 XTX i4x Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz · 504 GB RAMllama.cppgodclawQ4_K_MDetails →
3Phi-3.5-mini-instruct3.8Bmicrosoft Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
189,39 tok/s TG
Prefill 14.992 · TTFT 1.125 ms
5×2x AMD Radeon RX 7900 XTXAMD Ryzen Threadripper PRO 3955WX 16-Cores · 63 GB RAMvLLMgodclawDetails →
4Phi-3.5-mini-instruct3.8Bmicrosoft Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
172,34 tok/s TG
Prefill 5.226 · TTFT 11.908 ms
5×2x AMD Radeon RX 7900 XTX i4x Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz · 504 GB RAMllama.cppgodclawQ4_K_MDetails →
5Phi-3.5-mini-instruct3.8Bmicrosoft Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
99,48 tok/s TG
Prefill 4.127 · TTFT 671 ms
1×2x AMD Radeon RX 7900 XTX i4x Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz · 504 GB RAMllama.cppgodclawQ4_K_MDetails →
6Phi-3.5-mini-instruct3.8Bmicrosoft Performance benchmarkTimebench 3 - Kombi (Prefill + Generation)
39,32 tok/s TG
Prefill 6.700 · TTFT 380 ms
1×2x AMD Radeon RX 7900 XTXAMD Ryzen Threadripper PRO 3955WX 16-Cores · 63 GB RAMvLLMgodclawDetails →
Agent & chat rating

Harness benchmark

Noch keine Harnessbenchmarks fuer dieses Modell.