Haomin Qi

We present a governance-aware hybrid fine-tuning framework for multilingual, low-resource adaptation of large language models. The core algorithm combines gradient-aligned low-rank updates with structured orthogonal transformations through layer-wise mixing and introduces unitary constraints in selected sub-layers to stabilize deep optimization. In tandem with lightweight, label-free data governance steps, including language identification, near-duplicate removal, and quality filtering, the framework targets accuracy, calibration, and cross-language parity under tight compute budgets. Across XNLI and FLORES, the hybrid approach delivers consistent gains over strong PEFT baselines while maintaining directional balance and improving probability calibration, as shown in Tables II and III. It is more resilient to lightweight orthographic variants, as shown in Table IV, and benefits additively from simple governance steps, as shown in Table V. Training footprint measurements indicate modest overhead and a favorable cost-quality frontier, as shown in Table VI and Figure 2. Together, these results show that hybrid and unitary PEFT provide a stable and accessible path to resource-efficient multilingual adaptation when paired with practical data governance.

Fine-tuning large language models (LLMs) remains a computational bottleneck due to their scale and memory demands. This paper presents a comprehensive evaluation of parameter-efficient fine-tuning (PEFT) techniques, including LoRA, BOFT, LoRA-GA, and uRNN, and introduces a novel hybrid strategy that dynamically integrates BOFT’s orthogonal stability with LoRA-GA’s gradient-aligned rapid convergence. By computing per-layer adaptive updates guided by gradient norms, the hybrid method achieves superior convergence efficiency and generalization across diverse tasks. We also explore, for the first time, the adaptation of unitary RNN (uRNN) principles to Transformer-based LLMs, enhancing gradient stability through structured unitary constraints. Across GLUE, GSM8K, MT-Bench, and HumanEval with models from 7B to 405B, the hybrid approach yields consistent gains across three independent runs per task and model, approaching the quality of full fine-tuning while reducing training time by about 2.1 × and peak memory by nearly 50%, indicating practical significance under resource constraints. A compact multilingual and low-resource study on XNLI and FLORES with 32 examples per language shows consistent gains under the same budget with a small, stable footprint. These results indicate a practical and scalable path to accessible LLM fine-tuning under resource constraints.

We propose ABA-RAG, a retrieval-augmented generation (RAG) framework specifically tailored for applied behavior analysis (ABA) interventions, which integrates real-time emotional and behavioral data from Internet-of-Things (IoT) wearable devices. In ABA-RAG, we systematically retrieve semantically relevant prompts from a structured ABA task repository using embedding-based semantic search and dynamically construct context-aware prompts by incorporating IoT-derived predictions of learners’ emotional and behavioral states. To efficiently adapt large language model (LLM) to ABA contexts, we evaluate and compare several low-rank fine-tuning methods—including Low-Rank Adaptation (LoRA), Butterfly Orthogonal Fine-Tuning (BOFT), and LoRA with Gradient Approximation (LoRA-GA). Among these techniques, LoRA-GA demonstrates the best balance between computational efficiency and generation quality, making it particularly suitable for resource-constrained environments. Comprehensive experiments, validated through rigorous quantitative metrics and expert ABA practitioner evaluations, demonstrate that ABA-RAG significantly reduces manual workload while enhancing the precision, contextual relevance, and clinical utility of generated ABA tasks. Integrated into a practical web-based system, ABA-RAG provides ABA professionals with a scalable and real-time platform to generate individualized interventions.