Edward Kim

3D shape retrieval seeks to identify objects that are similar to a query in terms of geometry and structure, not only by its semantic category. Supervising this task is challenging because category labels do not adequately capture fine-grained shape similarity, and direct annotation of geometric similarity is both costly and subjective. We present a mesh-based retrieval framework (ShapeMRGNN) that combines graph representation learning with a practical strategy for constructing similarity-aware supervision. Each 3D object is represented as a mesh graph and encoded into a compact embedding with a geometry-aware network designed to capture structural properties while remaining robust to pose variation. To move beyond category-level supervision, we derive retrieval targets from similarity relations among canonical 2D renderings, embedded with the Florence vision foundation model, and use them to train a graph neural network (GNN) model with a contrastive objective. 3D shape retrieval is then performed by nearest-neighbor search in the learned embedding space. Experiments on ModelNet40 show that the proposed approach yields retrievals that more closely align with perceived 3D shape similarity than the geometric and spectral baselines, while remaining robust to object orientation.

The rapid growth of generative artificial intelligence has created opportunities across critical domains, yet fundamental trust barriers still limit deployment in high-stakes applications. The field currently faces three interconnected challenges: safety risks from agents that can be exploited for harmful purposes, privacy vulnerabilities arising from centralized inference, and unreliable hallucinated outputs. This dissertation addresses these challenges through four independent research contributions aligned with the themes of Safety, Security, Grounding, and Verification. The first contribution examines application-level safety by analyzing the behavior of LLM-powered penetration testing agents, characterizing their capabilities, limitations, and the risks associated with autonomous offensive-security tools. The second contribution investigates Secure Multi-Party Computation (SMPC) for privacy-preserving inference, demonstrating how generative models can operate over decentralized servers while protecting both sensitive data and proprietary model parameters. The third contribution focuses on grounding by coupling language models with knowledge graphs and embedding-guided graph traversal so that generated outputs remain connected to structured, verifiable information rather than relying on unconstrained text generation. The fourth contribution develops conformal prediction-based methods that provide finite-sample statistical guarantees on model outputs. By decomposing predictions into atomic statements and attaching calibrated confidence measures, this work offers a principled mechanism to quantify uncertainty, validate outputs, and enforce safety constraints during generative inference. Together, these contributions unify statistical guarantees, structured grounding, and privacy-preserving computation to support interpretable and secure deployment of generative AI in high-stakes environments.

Recent advancements in Large Language Models (LLMs) have focused on scaling architectures and developing efficient fine-tuning strategies like Low-Rank Adaptation (LoRA). While traditionally applied to autoregressive models, these methods are increasingly being adapted for text-based diffusion models. Standard fine-tuning, however, applies a static set of weight modifications across all samples and generation steps. In this work, we propose a dynamic, per-step adaptation method. We introduce a hypernetwork, termed the feedbackward (FB) model, which generates unique LoRA weights for a base feedforward (FF) model at each step of the diffusion process. Inspired by feedback connections in the brain, the FB model processes a masked input to generate LoRA weights for the query, key, value, and output projections within each of the FF model's attention blocks. Notably, these weights are generated in reverse sequence, from the final block to the first, allowing high-level contextual adjustments to inform lower-level feature representations. This dynamic weight generation is hypothesized to better guide the sampling trajectory for each input, offering more nuanced control than a static set of weights. The FF model, augmented with these bespoke LoRA weights, then processes the same input to produce the output for that generation step.

Catastrophic forgetting, the tendency of learning systems to lose previously acquired knowledge when trained on new information, remains one of the most fundamental challenges in continual learning. This dissertation contributes to a deeper understanding of why catastrophic forgetting occurs and investigates the mechanisms that enable learning models to maintain stability while acquiring new knowledge over time. The core of this work centers on the Cobweb framework, a psychologically inspired concept formation model that incrementally builds a hierarchy of concepts. A novel extension of this framework, Cobweb/4V, is introduced and systematically evaluated for its capacity to learn visual concepts in a continual manner without suffering from catastrophic forgetting. Motivated by the observed robustness of Cobweb/4V under continual learning, this research examines three hypotheses regarding factors that may contribute to such stability: (1) the adaptive structure hypothesis, which suggests that the ability to dynamically restructure the concept hierarchy as new data arrive supports continual learning by flexibly allocating representational capacity; (2) the sparse update hypothesis, which suggests that restricting parameter updates to localized regions of the model, rather than applying global changes, helps preserve prior knowledge by reducing representational overlap; and (3) the information-theoretic hypothesis, which proposes that employing closed-form updates based on sufficiency statistics enables precise, incremental learning without revisiting past data, offering potential advantages over gradient-based methods when previous data are unavailable. These hypotheses are evaluated through comparative experiments on datasets of varying complexity. By combining cognitive inspiration with controlled empirical evaluations, this dissertation offers new insights into the dynamics of memory retention and interference. These contributions not only clarify the foundations of catastrophic forgetting, but also point toward promising directions for developing continual learning systems that are both stable and adaptable.

Machine learning models offer transformative benefits across disciplines such as medicine, chemistry, and physics. However, as these models grow in size and usage, their energy demands increase dramatically, raising sustainability concerns. Neuromorphic hardware, inspired by the energy efficiency of the human brain, seeks to address this challenge by offering low-power, fast-processing alternatives to conventional computing. A key difference between neuromorphic systems and traditional architectures is the absence of shared memory between neurons, which poses a challenge to implementing learning algorithms. Yet, the brain learns effectively under this constraint, indicating the potential for machine learning methods to be adapted to such hardware. This research presents the design and implementation of the first ever fully on-chip neuromorphic Loihi 2 reinforcement learning agent. This circuit consists of a fully embedded Q-learning algorithm and an on-chip simulation of the CartPole-v0 environment on Intel's Loihi 2 neuromorphic processor. The system successfully trained agents that solved the CartPole-v0 task 36% of the time. Among all agents trained on Loihi 2, the top 50% achieved an average episode reward of 193.01, which is very near the benchmark score of 195 required to solve the task. In comparison, a similar Q-learning algorithm implemented on a conventional Intel Core i7-10870H CPU solved the task 62% of the time, with the top 50% of agents achieving a perfect score of 200. Despite the lower success rate, Loihi 2 exhibited major advantages in efficiency. During training, its dynamic power draw was only 0.02 watts, compared to 12 watts on the CPU. Execution time per weight update was also faster: 58.42 microseconds on Loihi 2 versus 162.54 microseconds on the CPU. Consequently, the neuromorphic system can train the same number of successful agents in only 44% of the required time for the CPU and with 600 times less power. This translates to a 1,365-fold increase in energy efficiency for an equivalent level of training success, and a 4,149-fold increase during inference. These findings demonstrate the viability of reinforcement learning on neuromorphic hardware and highlight its promise for building energy-efficient, real-time, embedded AI systems, as well as bring neuromorphic computing closer to realizing its potential as the backbone of a new sustainable, brain-like generation of AI.

Recent advancements in artificial intelligence have yielded impressive performance across domains; however, the deep learning methods that underlie these systems suffer from challenges in energy efficiency and remain vulnerable to exploits that degrade performance. As machine learning models grow in both size and popularity, improvements in energy efficiency and robustness are vital. In contrast, the brain has evolved to inherently express these capabilities. The organizational and functional principles that govern biological neural processing result in complex interactions that produce advanced, adaptable intelligence within physical constraints. In this work, we explore how key features of the brain can be leveraged to address critical challenges in machine learning, bridging the gap between biological and machine intelligence. By mimicking features of sparsity, competition, feedback, spiking communication, and local learning we develop inherently robust and efficient machine learning systems. We begin by exploring how biologically inspired sparse coding methods can detect and mitigate adversarial attacks that can mislead standard deep learning models. We develop hierarchical sparse coding models of visual processing in the brain at various levels and explore how features of lateral competition and top-down feedback create naturally robust and generalizable vision systems. We further demonstrate how the biologically plausible Locally Competitive Algorithm (LCA) for sparse coding can be applied to detect images corrupted by adversarial attacks. We explore novel spiking neuron models to balance computational efficiency and biological plausibility in frameworks compatible with energy efficient neuromorphic hardware. We provide evidence for sparse coding and spiking frameworks as valid models of processing in the brain, modeling neural responses with LCA and spiking neural networks (SNNs). Last, we present a novel energy-based architecture that implements LCA and Hopfield dynamics, combining sparse coding with energy-based models and equilibrium propagation for robust image classification and dictionary learning. Taken in sum, this work advances our understanding of biological and machine intelligence, laying the groundwork for the development of efficient and reliable artificial systems more aligned with the capabilities of the brain.

Just a few years ago, most vision-language modeling techniques consisted of individual processing branches for each modality along with a fusion layer to combine them. While the individual branches frequently leveraged unimodal pre-trained models, the fusion component would need to be learned from scratch. This was especially problematic in limited data scenarios where there is not enough data to sufficiently develop a robust multimodal feature space. The introduction of large-scale language-image pre-training techniques changed this, where few-shot, or even zero-shot, settings are now feasible for a number of standard downstream vision-language tasks. However, many tasks are not easily compatible with these foundation models out-of-the-box, either due to domain shift or a fundamental limitation of the model. In such scenarios, either the model needs to be fine-tuned, potentially destroying the existing weights and ruining its generalizability, or new modules must be added to the model and trained from scratch, requiring a substantial number of training examples to learn this new set of parameters. In this work, we set out to overcome these issues, proposing an alternative paradigm for leveraging foundation models in limited data scenarios that they are ill-suited for out-of-the-box, effectively utilizing their extensive knowledge without destroying their weights or adding new parameters. This enables the impressive few-shot performance of these models to be applied to new tasks. We begin by exploring the development of multimodal learning techniques, from earlier fusion- based models to more recent foundation models, with a particular emphasis on their ability to leverage limited data. Then, we set aside multimodality and turn our attention to a unimodal task, ultrasound video classification. We present a framework for learning robust models, using just a few dozen training examples, by leveraging subject matter expert knowledge, demonstrating the power of more traditional customized learning approaches over naively applying large pre-trained models to the task. Lastly, we combine these two lines of work by treating foundation models as the "experts" to facilitate training object detection models on noisy data in limited data settings. This is a unique task that foundation models are not particularly well suited for out-of-the-box. Yet, we demonstrate that applying segmentation and vision-language foundation models using this unique paradigm, obtains strong results even under severe noise.

Deep learning on 3D meshes presents significant challenges due to their intricate and non-uniform structure. Traditional deep learning methods, which are optimized for structured data such as time series, images, or voxel grids, struggle to effectively handle the complexity of mesh data. Additionally, achieving consistent performance in mesh analysis tasks across various mesh orientations is both crucial and challenging with conventional approaches. Graph Neural Networks (GNNs) offer an effective solution to these challenges by naturally adapting to the irregular, graph-like structure of 3D meshes. This dissertation details three major contributions to the field of 3D shape analysis based on GNNs. Firstly, in mesh segmentation, we introduce a deterministic process for transforming meshes into hybrid graphs. This is complemented by our Node-Edge-Face (NEF) embedding layer within the HyNet architecture, which excels in capturing complex geometric interactions for detailed and context-rich segmentation, as evidenced by superior performance in benchmark tests. Secondly, for mesh classification our novel representation learning model, RIMeshGNN, utilizes E(3) equivariant layers, aggregation functions, and pooling techniques to ensure rotation invariance, achieving exceptional accuracy without the need for rotation-augmented training. Lastly, in the area of mesh retrieval, we extend our classification model to encode objects into a discriminative embedding space using a contrastive loss function. This facilitates effective 3D object retrieval based on shape similarities. Due to the absence of a suitable 3D object dataset with ranked similarities for query objects, we created our own ground truth using an image-based retrieval method from Azure AI Vision Studio. Our comprehensive experiments across various datasets demonstrate that our methods achieve state-of-the-art performance, significantly enhancing the practicality and reliability of 3D shape analysis.