The attention mechanism is the core operation in Transformer models but remains difficult to deploy at the edge due to high memory access and data movement requirements. This work presents BumbleBee, an 18nm FD-SOI CMOS co-processor implementing the Flash Attention mechanism for the first time. It achieves 3mW power consumption and reduces bus transfers by 5 through a fully dataflow-driven architecture. Beyond energy efficiency, this dataflow also reduces latency by minimizing memory stalls, enabling fast token-level processing. The architecture integrates a dedicated attention head combining an addressable MAC array together with systolic arrays, and leverages binarized representations to further reduce computational cost. It achieves 20 and 248mJ/inference for BERT-Mini and DETR workloads, respectively.
