Quantization
Compressing a model by representing its weights with fewer bits — typically 8-bit or 4-bit instead of 16-bit.
What is Quantization?
LLMs are stored as floating-point weights — typically 16-bit (fp16) or 32-bit (fp32). Quantization compresses these to 8-bit (int8) or even 4-bit (int4) integers. A 70B fp16 model is 140GB; quantized to int4 it shrinks to ~35GB and fits on consumer GPUs.
There is a quality cost. Aggressive 4-bit quantization typically loses 1–3% on benchmarks vs the full model — usually acceptable for production. Modern methods (GPTQ, AWQ, GGUF) preserve quality better than naive rounding.
For Indian engineers, quantization is what makes running open-weight LLMs on premise affordable. A quantized Llama 70B serves real production traffic on hardware that costs lakhs not crores.
Quantization is the difference between "we use Llama" being a slide-deck claim and an actual production deployment. Every senior Gen AI engineer needs it.
A Bengaluru health-tech runs Llama 3 70B quantized to 4-bit (via GGUF + llama.cpp) on a single RTX 4090. Inference latency 12 tokens/sec, hardware cost ₹2.5L, no API spend — for a workload that was costing them ₹4L/month on Claude.
Want to master this?
Learn Quantization in a structured cohort
3-month live program with mentors, real projects, and 50+ partner placement support.