知Yizhi Daily
← Back to this edition

AI

Huawei Releases MACaQ Quantization Tool for Local Low-Precision Inference of Multimodal Models

Edition 2026-11-12 · 08:00 Beijing time

A Hacker News page shows that Huawei has released MACaQ, an internal quantizer that can generate low-precision weight and activation formats for text, image, video, and audio models and adapt to llama.cpp, vLLM, Diffusers, and PyTorch. The project says a 122000000000-parameter model can be quantized in 64 minutes using 32.5GB of GPU memory.

According to project documentation posted on Hacker News, MACaQ is positioned as a model-agnostic calibration quantization tool for local inference. It supports text, image, video, and audio models and can work with llama.cpp, vLLM, Diffusers, and PyTorch respectively for quantization and execution. The project provides high-quality GGUF and supports quantization workflows beyond the text modality.

The performance description says MACaQ can quantize a 122000000000-parameter model in 64 minutes with only 32.5GB of device memory. The project team says its perplexity can be lower than MR-GPTQ and DuQuant++; on Llama-3-8B, it can also reach the best perplexity level of those two with 11 times less quantization time.

For specific backends, the Diffusers path supports W4A4 for image, video, and audio generation; llama.cpp supports 2-, 3-, and 4-bit GGUF for text generation and can run on CPU; vLLM supports W4A4 or weight-only quantization for generating and deploying text models; and the PyTorch path covers W4A4 for speech generation. The project also provides MACaQ-R for optimizing weight codes while retaining the native GGUF format and block scaling factors, and the generated files can be run by standard llama.cpp.

The documentation also lists integration of Ascend CANN-related kernels and notes that full model execution requires combination with the Ascend inference runtime. The project uses the Apache 2.0 license, while included third-party components retain their respective licenses.

Original sources

Hacker News ↗
Back to this editionOpen interactive reader