Tharindu Kaluarachchi

'LightSense - Long Distance' explores remote interaction with architectural space. It is a virtual extension of the project 'LightSense,' which is currently presented at the exhibition 'Cyber Physical: Architecture in Real Time' at EPFL Pavilions in Switzerland. Using numerous VR headsets, the setup at the Art Gallery at SIGGRAPH Asia establishes a direct connection between both exhibition sites in Sydney and Lausanne. 'LightSense' at EPFL Pavilions is an immersive installation that allows the audience to engage in intimate interaction with a living architectural body. It consists of a 12-meter-long construction that combines a lightweight structure with projected 3D holographic animations. At its core sits a neural network, which has been trained on sixty thousand poems. This allows the structure to engage, lead, and sustain conversations with the visitor. Its responses are truly associative, unpredictable, meaningful, magical, and deeply emotional. Analysing the emotional tenor of the conversation, 'LightSense' can transform into a series of hybrid architectural volumes, immersing the visitors in Pavilions of Love, Anger, Curiosity, and Joy. 'LightSense's' physical construction is linked to a digital twin. Movement, holographic animations, sound, and text responses are controlled by the cloud-based AI system. This combination creates a location-independent cyber-physical system. As such, the 'Long Distance' version, which premiered at SIGGRAPH Asia, enables the visitors in Sydney to directly engage with the physical setup in Lausanne. Using VR headsets with a new 360-degree 4K live streaming system, the visitors find themselves teleported to face 'LightSense', able to engage in a direct conversation with the structure on-site. 'LightSense - Long Distance' leaves behind the notion of architecture being a place-bound and static environment. Instead, it points toward the next generation of responsive buildings that transcend space, are capable of dynamic behaviour, and able to accompany their visitors as creative partners.

Corneal Surface Reflections, or reflections on our eye-surface, have been shown as a valid and more socially acceptable source of information for passive lifelogging applications by prior work. However, automatic analysis of corneal surface reflections from a single RGB camera to support passive lifelogging is not extensively investigated in prior work. To address this, we developed a synthetic and self-supervised learning-based two-stage pipeline of deep learning models to detect objects in these reflections. Our prototype only consists a single RGB camera looking into the eye. We collected data from different users in uncontrolled environments using the prototype and trained our system to detect multiple classes of objects present in a typical office environment. We then evaluated our model in partially-controlled and in-the-wild scenarios. In addition, based on the findings from a follow up user study and prior work, we discuss strengths and weaknesses of our system and using corneal surface reflections for passive lifelogging. Finally, we opensource our source codes and trained checkpoints.

Extracting emotions from physiological signals has become popular over the past decade. Recent advancements in wearable smart devices have enabled capturing physiological signals continuously and unobtrusively. However, signal readings from different smart wearables are lossy due to user activities, making it difficult to develop robust models for emotion recognition. Also, the limited availability of data labels is an inherent challenge for developing machine learning techniques for emotion classification. This paper presents a novel self-supervised approach inspired by contrastive learning to address the above challenges. In particular, our proposed approach develops a method to learn representations of individual physiological signals, which can be used for downstream classification tasks. Our evaluation with four publicly available datasets shows that the proposed method surpasses the emotion recognition performance of state-of-the-art techniques for emotion classification. In addition, we show that our method is more robust to losses in the input signal.

After Deep Learning (DL) regained popularity recently, the Artificial Intelligence (AI) or Machine Learning (ML) field is undergoing rapid growth concerning research and real-world application development. Deep Learning has generated complexities in algorithms, and researchers and users have raised concerns regarding the usability and adoptability of Deep Learning systems. These concerns, coupled with the increasing human-AI interactions, have created the emerging field that is Human-Centered Machine Learning (HCML). We present this review paper as an overview and analysis of existing work in HCML related to DL. Firstly, we collaborated with field domain experts to develop a working definition for HCML. Secondly, through a systematic literature review, we analyze and classify 162 publications that fall within HCML. Our classification is based on aspects including contribution type, application area, and focused human categories. Finally, we analyze the topology of the HCML landscape by identifying research gaps, highlighting conflicting interpretations, addressing current challenges, and presenting future HCML research opportunities.

Emotion Recognition is a challenging research area given its complex nature, and humans express emotional cues across various modalities such as language, facial expressions, and speech. Representation and fusion of features are the most crucial tasks in multimodal emotion recognition research. Self Supervised Learning (SSL) has become a prominent and influential research direction in representation learning, where researchers have access to pre-trained SSL models that represent different data modalities. For the first time in the literature, we represent three input modalities of text, audio (speech), and vision with features extracted from independently pre-trained SSL models in this paper. Given the high dimensional nature of SSL features, we introduce a novel Transformers and Attention-based fusion mechanism that can combine multimodal SSL features and achieve state-of-the-art results for the task of multimodal emotion recognition. We benchmark and evaluate our work to show that our model is robust and outperforms the state-of-the-art models on four datasets.

In this paper, we describe waveSense, a power-efficient optical voxel tracking system for hand tracking in resource-limited devices. waveSense uses off-the-shelf, low-cost, non-focused infrared (IR) sensors and emitters. To achieve power efficiency, we introduce a novel selective volumetric illumination (SVI) technique, which illuminates different areas of a scene with variable power IR light and captures the reflected energy using non-focused IR sensors. Our technical evaluation suggests the feasibility of using waveSense for hand tracking performed within 45cm radius of the device. We believe with waveSense, gesture-based interactions would become more practical on resource-constrained wearables such as head-mounted displays (HMDs) and smartwatches.

We present waveSense, a low power hand gestures recogni- tion system suitable for mobile and wearable devices. A novel Selective Volumetric Illumination (SVI) approach using off-the-shelf infrared (IR) emitters and non-focused IR sensors were introduced to achieve the power efficiency. Our current implementation consumes 8.65mW while sensing hand gestures within 60cm radius from the sensors. In this demo, we introduce the concept and the theoretical background of waveSense, details of the prototype implementation, and application possibilities.