High-Performance LLM Inference Server
Rust inference runtime built on continuous batching and explicit KV-cache management, with live throughput and tail-latency signals.
Use case
The problem space.
Serve concurrent LLM requests without the throughput collapse and p99 spikes that single-request inference hits under load.
What's implemented
Built with intention.
Continuous scheduler01
A Rust scheduler admits compatible requests at decode boundaries instead of waiting for a full batch to finish.
Memory-aware serving02
KV-cache slots are reserved, reused, and released predictably; model loading supports INT8 and FP16 variants.
Inspectable performance03
gRPC exposes generation controls, while React visualises token rate, batch fill, and memory pressure in real time.
Architecture
Systems in concert.
The primary request and data paths, presented as a compact operating model.
A conceptual architecture for communicating the system design and operational responsibilities.
Step by step
From zero to
running.
Representative local-development commands that show the implementation path and operating sequence.
- 01
Build the runtime
Compile the Rust server with the CUDA or CPU feature set required by the target machine.
cargo build --release --features cuda - 02
Prepare a model
Fetch a supported checkpoint and select the quantisation profile for the serving environment.
./target/release/inference pull --model ./models/model.gguf --quant int8 - 03
Start the gRPC service
Launch the scheduler with bounded batch and cache settings.
./target/release/inference serve --port 50051 --max-batch 32 - 04
Watch and load test
Open the dashboard, then drive concurrent streaming prompts through the gRPC benchmark client.
cd dashboard && npm run dev python bench.py --concurrency 64