Hardware-Aware Co-Design of Vision-Language-Action Models for Real-Time Autonomous Driving on Edge NPUs HACT-VLA: Query-Conditioned Concept Routing with Low-Rank Integer-Friendly Fusion
Vision-language-action (VLA) models are attractive for autonomous driving because language can expose intent, support human interaction, and organize scene reasoning, yet contemporary VLA pipelines are difficult to deploy under the deterministic latency, memory, and operator constraints of automotive edge neural processing units (NPUs). This paper presents HACT-VLA, a concrete hardware-aware co-design that replaces unrestricted cross-modal attention in the action head with a query-conditioned concept-token route, a low-rank multiplicative fusion operator, and an integerfriendly decision path. The key idea is to exploit the structure of driving decisions: a language action query such as "may I turn left?" does not require every semantic concept inferred from the scene. HACT-VLA predicts 21 interpretable driving concepts, scores them with a compact language-conditioned router, selects a fixed top-๐ subset, and evaluates the requested action using only dense linear projections, elementwise operations, and a small top-๐ primitive. This yields a fixed-shape computation graph suitable for ahead-of-time deployment while reducing the trainable action-head parameter count by 25.2% relative to a dense query baseline in our implementation. The study is organized around the public BDD-OIA task, which contains four driving actions and 21 action-inducing explanation concepts. We provide a verified real-data pipeline for the public 2048-dimensional Faster-RCNN-derived BDD-OIA embeddings and a fully measured BDD-OIAcompatible surrogate experiment for hardware/co-design analysis. On the executed surrogate, the dense query model reaches 0.9998 macro-F1 with 133.7k parameters; HACT-VLA uses 100.0k parameters and reaches 0.9964 macro-F1 at ๐ = 4. The same model remains above 0.9938 macro-F1 under additive feature noise with standard deviation 0.6. Sensitivity experiments reveal a non-monotonic routing effect: increasing ๐ from 4 to 12 reduces macro-F1 from 0.9964 to 0.9896, indicating that hardware-driven sparsity can also act as semantic regularization. Actual CPU timing further shows that naive dynamic INT8 is not automatically faster for a tiny batch-1 policy head because quantization dispatch overhead dominates; this finding motivates static NPU compilation rather than assuming low precision guarantees low latency. All reported timing values are measured on the provided runtime, and no CPU number is relabeled as NPU latency. The resulting method, analysis, and code form a reproducible basis for completing device-specific NPU deployment without changing the learned policy interface.