Choose quantization for the model and the machine
Lower-precision weights can save memory, but different methods make different demands on deployment.
Quantization evaluates a tradeoff between representation size and retained behavior, not just whether a model fits in memory. The Transformers quantization overview describes lower-precision weights as a way to reduce memory requirements while aiming to preserve accuracy. It is ongoing technical documentation, with applicability conditioned on the model, method and hardware.
The methods have different tradeoffs. Some require calibration, while others support quantization during loading. Model compatibility, supported operations and hardware all influence the choice. Selecting a method solely from the resulting file size leaves important parts of the integration unexamined.
Memory savings and speed also need separate treatment. A smaller representation does not guarantee faster execution on an arbitrary machine; the implementation and hardware must support the relevant path. Response quality should likewise be checked against the inputs the product actually receives, not assumed from the precision label.
A comparison on representative inputs could establish whether a compatible method meets the application's resource and response-quality constraints; no such measurements are reported here. The documentation supplies candidate methods, while file size alone leaves speed and task behavior unresolved. The relevant uncertainty is how the chosen representation executes on the intended hardware and what changes in its responses.