Knowledge Distillation in Compact Models: An Approach Applied to Text Processing for Public Security

Authors:
Open Access
Publisher website
Google Scholar
Abstract
This article aims to develop a model for summarizing police reports in the context of public security, with a focus on execution on limited hardware. The approach combined hybrid distillation (logits and intermediate representations) and supervised fine-tuning with LoRA, using a corpus of 19,286 police reports. The evaluation was conducted using automatic metrics (BERTScoreF1) and qualitative analysis by specialists. The results demonstrate that the distilled model generates clear, coherent, and semantically appropriate summaries, comparable to those of the larger model, with superior computational performance, including in hardware-constrained environments.