We propose a co-design framework that integrates hardware-aware operator optimization with a heuristic mixed-precision search. We systematically optimize key Linear and Attention operators via loop tiling, high-throughput kernel implementation, and operator fusion, specifically tuned for ARM architectures. Concurrently, a heuristic search method based on a genetic algorithm is proposed to identify an optimal mixed-precision quantization scheme that balances model accuracy, memory footprint, and inference speed. Experiments on ARM devices show that our approach substantially reduces resource usage while accelerating inference. Under comparable accuracy, the optimized model consumes only 8.89% of the original memory and achieves 5.11× higher throughput in the prefill stage and 11.86× in the decode stage. Compared to state-of-the-art (SOTA) methods, our framework attains higher accuracy with 4.03× greater memory reduction and a further 1.25× prefill throughput improvement.
Track ID:
15.13
Track Name:
Specialized Hardware for Embodied AI Application and Neuromorphic Computing
