Steering Conceptual Bias via Transformer Latent-Subspace Activation
Authors:
Preprint
Abstract
This work examines whether activating latent subspaces in language models (LLMs) can steer scientific code generation toward a specific programming language. Five causal LLMs were first evaluated on scientific coding prompts to quantify their baseline bias among four programming languages. A static neuron-attribution method, perturbing the highest activated MLP weight for a C++ or CPP token, proved brittle and exhibited limited generalization across prompt styles and model scales. To address these limitations, a gradient-refined adaptive activation steering framework (G-ACT) was developed: per-prompt activation differences are clustered into a small set of steering directions, and lightweight per-layer probes are trained and refined online to select the appropriate steering vector. In LLaMA-3.2 3B, this approach reliably biases generation towards the CPP language by increasing the average probe classification accuracy by 15% and the early layers (0-6) improving the probe classification accuracy by 61.5% compared to the standard ACT framework. For LLaMA-3.3 70B, where attention-head signals become more diffuse, targeted injections at key layers still improve language selection. Although per-layer probing introduces a modest inference overhead, it remains practical by steering only a subset of layers and enables reproducible model behavior. These results demonstrate a scalable, interpretable and efficient mechanism for concept-level control for practical agentic systems.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categories2025
- Empowering biomedical discovery with AI agentsCell, 2024
- SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models2024
- Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agents2024
- A reliable knowledge processing framework for combustion science using foundation modelsEnergy and AI, 2024
- A survey on large language model based autonomous agentsFrontiers of Computer Science, 2024
- Unsupervised discovery of interpretable visual conceptsInformation Sciences, 2024
- Robust explanations for graph neural network with neuron explanation componentInformation Sciences, 2023
- Unlocking the black box of CNNs: Visualising the decision-making process with PRISMInformation Sciences, 2023