TensorSharp作者实测6大LLM推理引擎速度与质量对比

Most LLM inference benchmarks focus on tokens/sec, TTFT, or throughput.

But faster inference doesn't necessarily mean better inference, especially when quantization, speculative decoding, long contexts, and concurrency enter the picture.

So we ran a comprehensive benchmark of Qwen3.8-27B across six inference configurations, measuring not just speed, but also output quality.

Full disclosure: I'm the author of TensorSharp, one of the engines being benchmarked.

What makes this benchmark different?

1. Speed AND quality, measured together

We compared:

• Time to First Token (TTFT)
• Prefill throughput (tokens/sec)
• Decode throughput (tokens/sec)
• End-to-end task latency
• Concurrent request performance
• Answer quality against automated checks

A fast engine isn't particularly useful if its optimizations introduce incorrect outputs.

That's why we included quality validation alongside performance measurements, rather than assuming that every engine produces equivalent results.

2. Ultra-long contexts: 64K, 128K, and 256K tokens

Many benchmarks use relatively short prompts, but real-world agentic workloads can be very different.

Think about coding agents processing entire repositories, RAG pipelines handling large documents, or multi-agent systems maintaining extensive histories.

We tested long-context inference at 64K, 128K, and 256K, including concurrent workloads.

At these context lengths, KV-cache management, memory pressure, prefill efficiency, and scheduling become critical.

An engine that wins at 4K or 8K doesn't necessarily win at 256K.

3. GGUF vs NInfer model formats

Another interesting aspect is comparing inference configurations using GGUF and NInfer model representations.

The goal isn't just to compare engines, but also to understand how model format, quantization, and inference optimizations affect performance and quality.

This matters because developers often choose a model format based on compatibility or memory requirements without fully understanding its performance implications.

4. Real-world task latency, not just synthetic throughput

We also measured end-to-end latency across different task scenarios, including coding, reasoning, and agent-oriented workloads.

For production applications, the fastest tokens/sec number doesn't always translate into the fastest completed task.

Why this matters

I think inference benchmarks should answer three questions together:

• How fast is it?
• Does it still produce correct answers?
• Does it remain practical at long context lengths and under concurrency?

Optimizing only one of these can be misleading.

The attached poster includes the engine rankings, speed/quality comparisons, 64K–256K context results, task-level latency, and test configurations.

I'd love feedback from other LLM developers

• Do you consider quality validation mandatory when comparing inference engines?
• How important is 128K–256K context support for your production workloads?
• Have you observed meaningful differences between GGUF and other model formats?
• Which matters more in your applications: TTFT, decode speed, concurrent throughput, or end-to-end agent latency?

I'm particularly interested in what additional tests would make benchmarks like this more useful and reproducible for the community.

Project: https://github.com/zhongkaifu/TensorSharp

Benchmark details and results are in the attached poster.