Zhenyu Yan

Procedural tasks with multiple ordered steps are ubiquitous in daily life. Recent advances in multimodal large language models (MLLMs) have enabled personal assistants that support daily activities. However, existing systems primarily provide reactive guidance triggered by user queries, or limited proactive assistance for isolated short-term events rather than long-horizon procedural tasks. In this work, we introduce Pro ² Assist, a step-aware proactive assistant that continuously tracks fine-grained task progress and reasons over the user's evolving state to provide timely assistance throughout tasks. Pro ² Assist leverages multimodal data from augmented reality (AR) glasses to achieve motion-based perception. It then extracts step-oriented procedural context from multi-scale temporal dynamics and task-specific expert knowledge. Based on both sensory input and procedural context, Pro ² Assist performs continuous reasoning to infer user needs and display timely assistance on AR glasses. We evaluate Pro ² Assist using a dataset curated from public sources and a real-world dataset collected on our testbed with AR glasses. Extensive evaluations show that Pro ² Assist outperforms the best-performing baselines by over 21% in procedural action understanding accuracy, and it achieves up to 2.29X the proactive timing accuracy of baselines. A user study with 20 participants further shows that 90% find Pro ² Assist useful, indicating its effectiveness for real-world procedural assistance.

Biosignals collected from wearable devices are widely utilized in healthcare applications. Machine learning models used in these applications often rely on features extracted from biosignals due to their effectiveness, lower data dimensionality, and wide compatibility across various model architectures. However, existing feature extraction methods often lack task-specific contextual knowledge, struggle to identify optimal features in high-dimensional combinatorial feature space, and are prone to automated code generation and execution errors. In this paper, we propose DeepFeature, the first LLM-empowered, context-aware feature generation framework for wearable biosignals. DeepFeature introduces a multi-source feature generation mechanism that integrates the inherent ability of LLMs, expert knowledge and inter-feature interactions. It also employs an iterative feature refinement process that uses feature assessment-based feedback for feature re-selection. Additionally, DeepFeature utilizes a robust multi-layer filtering and verification approach for feature description-to-code translation to ensure that the feature extraction functions run without crashing. Experimental evaluation results show that DeepFeature achieves the highest average AUROC across eight tasks under both sample-level and subject-level settings, outperforming the best baselines by 4.60% and 4.61%, respectively. DeepFeature achieves the most pronounced gains on the PPG-BP tasks, while remaining competitive with the best-performing baselines on Epilepsy, WESAD, and our self-collected SEN dataset.

BACKGROUND Delayed optimization of guideline-directed medical therapy (GDMT) after hospitalization for acute decompensated heart failure with reduced ejection fraction (HFrEF) remains common. Mobile health and telemonitoring platforms may help convert post-discharge medication titration from an episodic clinic-based process into a structured remote-care workflow. OBJECTIVE To evaluate the effectiveness and safety of the Telemonitoring to Optimize Medication for Heart Failure (TOM-HF) mobile health platform for remote GDMT titration after hospitalization for new-onset acute decompensated HFrEF. METHODS Adults hospitalized for HFrEF were prospectively enrolled in the Telemonitoring to Optimize Medication for Heart Failure (TOM-HF) study and followed for 12 weeks. Remote titration was delivered through a mobile health platform with algorithm-supported and specialist-adjudicated dosing decision. A 1:2 cardinality-matched historical cohort from the same centre’s heart failure registry served as the control group. The primary endpoint was the modified Heart Failure Collaboratory (mHFC) score, which is a standardized measure of heart failure GDMT intensity. RESULTS Overall, 28 TOM-HF participants were matched to 56 registry controls. The mHFC score was significantly higher in TOM-HF versus controls at discharge (64.3 vs 54.7; P=0.035), 4 weeks (98.2 vs 64.7; P<0.001), and 12 weeks (97.8 vs 69.2; P<0.001). Median time to maximally tolerated GDMT was 12 days (IQR 9–18) versus 82 days (IQR 23–90; log-rank P<0.001), with 100.0% versus 62.5% achieving maximally tolerated GDMT by 12 weeks. At 12 weeks, TOM-HF patients demonstrated greater improvements in 6-minute walk distance (+32.2% vs +23.6%; P<0.001), left ventricular ejection fraction (+21.7% vs +13.0%; P<0.001), and NT-proBNP (-90.8% vs -26.4%; P<0.001). Composite adherence was 83.2% (IQR, 76.5–91.4), and mean System Usability Scale score was 75.2. Hypotension was more frequent in TOM-HF, whereas serious adverse events were not increased. CONCLUSIONS A mobile health-enabled remote titration strategy effectively and safely accelerated GDMT optimization after acute HFrEF hospitalization, warranting assessment in large-scale randomized trials. CLINICALTRIAL Trial registration number: NCT07016815

Multimodal human action recognition (HAR) utilizes complementary data for activity classification. Built on traditional HAR tasks, recent advances in Large Language Models (LLMs) enable detailed descriptions and causal reasoning of human actions, advancing new tasks of human action understanding (HAU) and human action reasoning (HARn). However, most LLMs, especially multimodal Large Vision-Language Models (LVLMs), struggle with modalities other than RGB images, like depth, IMU, ormmWave, due to a lack of large-scale datasets in these task domains. Existing HAR datasets provide only coarse-grained annotations, in-sufficient for depicting the detailed action dynamics required in HAU and HARn tasks. Simply combining annotations and generating captions with LLMs often lacks necessary logical and spatiotemporal consistency. In this paper, we introduce CUHK-X, a large-scale multi-modal dataset and benchmarks for HAR, HAU, and HARn. It includes 64,267 samples of 40 actions performed by 30 participants across two indoor environments, covering diverse daily scenarios. To address the challenge of spatiotemporal inconsistencies in captions, we propose a prompt-based scene creation method that leverages LLMs to generate logically connected activity sequences. CUHK-X also includes three benchmarks with six tasks to evaluate state-of-the-art models. Experimental results show average accuracies of 76.52% for HAR, 40.76% for HAU, and 70.25% for HARn. This large-scale multimodal dataset aims to empower the research community to apply, develop, and adapt data-intensive learning techniques for a wide range of human activity-related tasks.