AI
LookAtMyQuant Quantization Compiler Compresses Inkling-2-Mini to 5.38GB and Up
LookAtMyQuant is a quantization compiler that allocates encoding according to compression damage, using post-training quantization, Trellis codes, Fisher-information row-level bit allocation, and 4000-step distillation. It produces four Inkling-2-Mini MLX versions for Apple Silicon, compressing text weights from 18.89GB to between 5.38GB and 12.76GB.
The project says the compiler measures the damage to output caused by compressing each stored part of a model, then chooses encodings for each part to minimize overall loss within a specified size budget. The code uses the MIT license. The page also gives results for Qwen3.5-4B: 1.23GB, an average of 2.3 bits per weight, able to answer 330 of 540 exact-answer mathematics problems, with the complete recipe costing about $34 in GPU costs.
On Qwen3.8-27B, the same method compresses the 53.7GB model to 7.2GB at 2.1 bits, answering 401 of 540 mathematics problems, the same number as the original model; the same-sized Unsloth UD-IQ2_XXS GGUF answered 348. HumanEval, MMLU, and GSM8K scores were 73.8%, 78.8%, and 94.5%, respectively. The accompanying 4-bit DSpine drafter is 1.02GB and is said to roughly double mathematics and code decoding speed on Apple Silicon.
For Inkling-2-Mini, a 3x recurrent MoE model with 26.4B physical parameters and 5.4B active parameters, the compiler combines full-MoE proximal fine-tuning to turn 18.89GB of text weights into four Apple Silicon MLX versions of 5.38GB, 8.87GB, 11.18GB, and 12.76GB, corresponding to about 1.65, 2.72, 3.43, and 3.92 bits per weight.
Evaluation used untruncated output, greedy decoding, and reasoning_effort="none" settings, covering the first 200 GSM8K problems, 164 executable HumanEval problems, four visual VQA tests, and multilingual TextMix windows. The 12.76GB version reached 100% of the uncompressed BF16 teacher on 349 aggregated benchmarks, with text-weight memory reduced by 32.5%; the 5.38GB version scored 88.0% on GSM8K and had strict HumanEval pass rates of 105/164.