Definition: Quantization stores a model's weights, and sometimes its activations, at lower numerical precision, such as 8-bit or 4-bit instead of 16-bit, so the model needs less memory and runs faster with a modest loss in quality.
Neural network weights are usually trained in 16-bit or 32-bit floating point. Quantization maps them to fewer bits. Post-training methods such as GPTQ and AWQ quantize a finished model, while quantization-aware training builds lower precision into training itself. Formats such as GGUF package quantized models for local runtimes.
Lower precision lets a model fit on fewer or cheaper GPUs, raises throughput and can make on-device or on-premises deployment practical. The trade-off is some accuracy loss, which tends to grow at very low bit widths and varies by task.
Teams that self-host open-weight models compare quantized variants on their own evals, measure latency and cost per request, and choose the lowest precision that still meets quality targets.
Ryz MLOps engineers optimize model serving in your cloud, working on your team and reporting to your leads.
Thanks — your message has been sent. We’ll get back to you soon.
Something went wrong while sending your message. Please try again or email info@ryzlabs.com.