LLMs are parameterized by a ton of weights, when we say something like 400B we mean it has 400 billion parameters. In modern LLMs those parameters are basically always 16 bit floating point numbers.
It turns out you can get nearly as good results by reducing the precision of those numbers, for instance by using 4 bits per parameter instead of 16, meaning each parameter can only take on one of 16 possible values instead of one of 65536.
Most claims of "nearly as good results" are massively overblown.
Even the so called "good" quants of huge models are extremely crippled.
Nothing is ever free, and even going from 16 to 8bit will massively reduce the quality of your model, no matter whatever their hacked benchmarks claim.
No, it doesn't help because of "free regularization" either. Dropout and batch norm were also placebo BS that didn't actually help to back in the day when they were still being used.
Interestingly enough, Llama3 suffers more performance loss than Llama2 did at identical quantizations. https://arxiv.org/abs/2404.14047
There's some speculation that a net trained for more epochs on more data learns to pack more information into the weights, and so does worse when weight data is degraded.
LLMs are parameterized by a ton of weights, when we say something like 400B we mean it has 400 billion parameters. In modern LLMs those parameters are basically always 16 bit floating point numbers.
It turns out you can get nearly as good results by reducing the precision of those numbers, for instance by using 4 bits per parameter instead of 16, meaning each parameter can only take on one of 16 possible values instead of one of 65536.