Analyzing Latent Concepts in Code Language Models
Preprint
Abstract
Interpreting the internal behavior of large language models trained on code remains a critical challenge, particularly for applications demanding trust, transparency, and semantic robustness. We propose Code Concept Analysis (CoCoA): a global post-hoc interpretability framework that uncovers emergent lexical, syntactic, and semantic structures in a code language model's representation space by clustering contextualized token embeddings into human-interpretable concept groups. We propose a hybrid annotation pipeline that combines static analysis tool-based syntactic alignment with prompt-engineered large language models (LLMs), enabling scalable labeling of latent concepts across abstraction levels. We analyse the distribution of concepts across layers and across three finetuning tasks. Emergent concept clusters can help identify unexpected latent interactions and be used to identify trends and biases within the model's learned representations. We further integrate LCA with local attribution methods to produce concept-grounded explanations, improving the coherence and interpretability of token-level saliency. Empirical evaluations across multiple models and tasks show that LCA discovers concepts that remain stable under semantic-preserving perturbations (average Cluster Sensitivity Index, CSI = 0.288) and evolve predictably with fine-tuning. In a user study on the programming-language classification task, concept-augmented explanations disambiguated token roles and improved human-centric explainability by 37 percentage points compared with token-level attributions using Integrated Gradients.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Redundancy and Concept Analysis for Code-trained Language Models2024
- DeciX: Explain Deep Learning Based Code Generation ApplicationsProceedings of the ACM on Software Engineering, 2024
- Latent Concept-based Explanation of NLP Models2024
- An Explanation Method for Models of CodeProceedings of the ACM on Programming Languages, 2023
- ConceptX: A Framework for Latent Concept AnalysisProceedings of the AAAI Conference on Artificial Intelligence, 2023
- Towards Concept-Aware Large Language Models2023
- Diet code is healthy: simplifying programs for pre-trained models of code2022
- AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language models | 37th IEEE/ACM International Conference on Automated Software Engineering2022
- Probing Classifiers: Promises, Shortcomings, and AdvancesComputational Linguistics, 2022
- CodeBERT: A Pre-Trained Model for Programming and Natural Languages2020