Matthew J. Mender

Objective. Brain–machine interfaces (BMI) aim to restore function to persons living with spinal cord injuries by ‘decoding’ neural signals into behavior. Recently, nonlinear BMI decoders have outperformed previous state-of-the-art linear decoders, but few studies have investigated what specific improvements these nonlinear approaches provide. In this study, we compare how nonlinear and linear approaches predict individuated finger movements in open and closed-loop settings. Approach. Two adult male rhesus macaques were implanted with Utah arrays in the motor cortex and performed a 2D dexterous finger movement task for a juice reward. Multiple linear and nonlinear ‘decoders’ were used to map from recorded spiking band power into movement kinematics. Performance of these decoders was compared and analyzed to determine how nonlinear decoders perform in both open and closed-loop scenarios. Main Results. We show that nonlinear decoders enable control which more closely resembles true hand movements, producing distributions of velocities 80.7% closer to true hand control than linear decoders. Addressing concerns that neural networks may come to inconsistent solutions, we find that regularization techniques improve the consistency of temporally-convolved feedforward neural network convergence by up to 188.9%, along with improving average performance and training speed. Finally, we show that TCNs and long short-term memory can effectively leverage training data from multiple task variations to improve generalization. Significance. The results of this study support artificial neural networks of all kinds as the future of BMI decoding and show potential for generalizing over less constrained tasks.

Individuals with severe neurological injuries often rely on assistive technologies, but current methods have limitations in accurately decoding multi-degree-of-freedom (DoF) movements. Intracortical brain-machine interfaces (iBMIs) use neural signals to provide a more natural control method, but currently struggle with higher-DoF movements—something the brain handles effortlessly. It has been theorized that the brain simplifies high-DoF movement through muscle synergies, which link multiple muscles to function as a single unit. These synergies have been studied using dimensionality reduction techniques like principal component analysis (PCA), non-negative matrix factorization (NMF), and demixed PCA (dPCA) and successfully used to reduce noise and improve offline decoder stability in non-invasive applications. However, their effectiveness in improving decoding and generalizability for implanted recordings across varied tasks is unclear. Here, we evaluated if brain and muscle synergies can enhance iBMI performance in non-human primates performing a two-DoF finger task. Specifically, we tested if PCA, dPCA, and NMF could compress and denoise brain and muscle data and improve decoder generalization across tasks. Our results showed that while all methods effectively compressed data with minimal loss in decoding accuracy, none improved performance through denoising. Additionally, none of the methods enhanced generalization across tasks. These findings suggest that while dimensionality reduction can aid data compression, alone it may not reveal the “true” control space needed to improve decoder performance or generalizability. Further research is required to determine whether synergies are the optimal control framework or if alternative approaches are required to enhance decoder robustness in iBMI applications. Significance Statement Many researchers believe that brain and muscle synergies represent a fundamental control strategy and could enhance brain-machine interface (BMI) decoding performance. These synergies, extracted through dimensionality reduction techniques, are thought to simplify complex neural data, improving the efficiency and accuracy of BMI systems. In our study, we evaluated brain and muscle synergies in a dexterous finger task. We found that while these synergies effectively compressed high-dimensional data, they did not improve performance through denoising or generalize well across different contexts. Instead, the highest performance was achieved when using all available data, suggesting that synergies, although useful for data compression, may not provide the “true” control space needed to enhance decoder robustness or adaptability in implanted BMI systems.

Recent advances in brain-machine interfaces (BMIs) using neural network decoders and increased channel count have improved the restoration of speech and motor function, but at the cost of higher power consumption. For wireless, implantable BMIs to be clinically viable, power consumption must be limited to prevent thermal tissue damage and enable long use without frequent charging. Here, we show how neural network “pruning” creates sparse decoders that require fewer computations and active channels for reduced power consumption. Across multiple movement decoding tasks using brain and muscle signals, recurrent neural network decoders can be compressed by over 100x while maintaining strong performance, enabling decoding on the implant with <1% power increase compared to decoding externally. Pruning also allows for deactivating up to 89% of channels, reducing BMI power by up to 5x. Counterintuitively, our findings suggest that BMIs employing a subset of a larger number of channels may achieve lower power consumption than BMIs with fewer channels, for a given performance level. These results suggest a path toward power-efficient, implantable BMIs suitable for long-term clinical use.

People with brain or spinal cord-related paralysis often need to rely on others for basic tasks, limiting their independence. A potential solution is brain-machine interfaces (BMIs), which could allow them to voluntarily control external devices (e.g., robotic arm) by decoding brain activity to movement commands. In the past decade, deep-learning decoders have achieved state-of-the-art results in most BMI applications, ranging from speech production to finger control. However, the ‘black-box’ nature of deep-learning decoders could lead to unexpected behaviors, resulting in major safety concerns in real-world physical control scenarios. In these applications, explainable but lower-performing decoders, such as the Kalman filter (KF), remain the norm. In this study, we designed a BMI decoder based on KalmanNet, an extension of the KF that augments its operation with recurrent neural networks to compute the Kalman gain. This results in a varying “trust” that shifts between inputs and dynamics. We used this algorithm to predict finger movements from the brain activity of two monkeys. We compared KalmanNet results offline (pre-recorded data, n = 13 days) and online (real-time predictions, n = 5 days) with a simple KF and two recent deep-learning algorithms with state-of-the-art results: tcFNN and LSTM. KalmanNet achieved comparable or better results than other deep learning models in offline and online modes, relying on the dynamical model for stopping while depending more on neural inputs for initiating movements. We further validated this mechanism by implementing a heteroscedastic KF that used the same strategy, and it also approached state-of-the-art performance while remaining in the explainable domain of standard KFs. However, we also see two downsides to KalmanNet. KalmanNet shares the limited generalization ability of existing deep-learning decoders, and its usage of the KF as an inductive bias limits its performance in the presence of unseen noise distributions. Despite this trade-off, our analysis successfully integrates traditional controls and modern deep-learning approaches to motivate high-performing yet still explainable BMI designs.

Intracortical brain-computer interfaces (iBCIs) can restore movement and communication abilities to individuals with paralysis by decoding their intended behavior from neural activity recorded with an implanted device. While this activity yields high-performance decoding over short timescales, neural data are often nonstationary, which can lead to decoder failure if not accounted for. To maintain performance, users must frequently recalibrate decoders, which requires the arduous collection of new neural and behavioral data. Aiming to reduce this burden, several approaches have been developed that either limit recalibration data requirements (few-shot approaches) or eliminate explicit recalibration entirely (zero-shot approaches). However, progress is limited by a lack of standardized datasets and comparison metrics, causing methods to be compared in an ad hoc manner. Here we introduce the FALCON benchmark suite (Few-shot Algorithms for COnsistent Neural decoding) to standardize evaluation of iBCI robustness. FALCON curates five datasets of neural and behavioral data that span movement and communication tasks to focus on behaviors of interest to modern-day iBCIs. Each dataset includes calibration data, optional few-shot recalibration data, and private evaluation data. We implement a flexible evaluation platform which only requires user-submitted code to return behavioral predictions on unseen data. We also seed the benchmark by applying baseline methods spanning several classes of possible approaches. FALCON aims to provide rigorous selection criteria for robust iBCI decoders, easing their translation to real-world devices. https://snel-repo.github.io/falcon/

Brain-machine interface (BMI) controlled functional electrical stimulation (FES) is a promising treatment to restore hand movements to people with cervical spinal cord injury. Recent intracortical BMIs have shown unprecedented successes in decoding user intentions, however the hand movements restored by FES have largely been limited to predetermined grasps. Restoring dexterous hand movements will require continuous control of many biomechanically linked degrees-of-freedom in the hand, such as wrist and finger flexion, that would form the basis of those movements. Here we investigate the ability to restore simultaneous wrist and finger flexion, which would enable grasping with a controlled hand posture and assist in manipulating objects once grasped. We demonstrate that intramuscular FES can enable monkeys with temporarily paralyzed hands to move their fingers and wrist across a functional range of motion, spanning an average 88.6 degrees at the metacarpophalangeal joint flexion and 71.3 degrees of wrist flexion, and intramuscular FES can control both joints simultaneously in a real-time task. Additionally, we demonstrate a monkey using an intracortical BMI to control the wrist and finger flexion in a virtual hand, both before and after the hand is temporarily paralyzed, even achieving success rates and acquisition times equivalent to able-bodied control with BMI control after temporary paralysis in two sessions. Together, this outlines a method using an artificial brain-to-body interface that could restore continuous wrist and finger movements after spinal cord injury. One Sentence Summary We show in non-human primates that intramuscular functional electrical stimulation can flex the wrist and fingers through a functional range of movement, can be controlled precisely, and that a brain-machine interface can determine command signals for intended finger and wrist flexion.

Brain-machine interfaces (BMI) aim to restore function to persons living with spinal cord injuries by ‘decoding’ neural signals into behavior. Recently, nonlinear BMI decoders have outperformed previous state-of-the-art linear decoders, but few studies have investigated what specific improvements these nonlinear approaches provide. In this study, we compare how temporally convolved feedforward neural networks (tcFNNs) and linear approaches predict individuated finger movements in open and closed-loop settings. We show that nonlinear decoders generate more naturalistic movements, producing distributions of velocities 85.3% closer to true hand control than linear decoders. Addressing concerns that neural networks may come to inconsistent solutions, we find that regularization techniques improve the consistency of tcFNN convergence by 194.6%, along with improving average performance, and training speed. Finally, we show that tcFNN can leverage training data from multiple task variations to improve generalization. The results of this study show that nonlinear methods produce more naturalistic movements and show potential for generalizing over less constrained tasks. Teaser A neural network decoder produces consistent naturalistic movements and shows potential for real-world generalization through task variations.

Objective. While brain–machine interfaces (BMIs) are promising technologies that could provide direct pathways for controlling the external world and thus regaining motor capabilities, their effectiveness is hampered by decoding errors. Previous research has demonstrated the detection and correction of BMI outcome errors, which occur at the end of trials. Here we focus on continuous detection and correction of BMI execution errors, which occur during real-time movements. Approach. Two adult male rhesus macaques were implanted with Utah arrays in the motor cortex. The monkeys performed single or two-finger group BMI tasks where a Kalman filter decoded binned spiking-band power into intended finger kinematics. Neural activity was analyzed to determine how it depends not only on the kinematics of the fingers, but also on the distance of each finger-group to its target. We developed a method to detect erroneous movements, i.e. consistent movements away from the target, from the same neural activity used by the Kalman filter. Detected errors were corrected by a simple stopping strategy, and the effect on performance was evaluated. Main results. First we show that including distance to target explains significantly more variance of the recorded neural activity. Then, for the first time, we demonstrate that neural activity in motor cortex can be used to detect execution errors during BMI controlled movements. Keeping false positive rate below 5 % , it was possible to achieve mean true positive rate of 28.1 % online. Despite requiring 200 ms to detect and react to suspected errors, we were able to achieve a significant improvement in task performance via reduced orbiting time of one finger group. Significance. Neural activity recorded in motor cortex for BMI control can be used to detect and correct BMI errors and thus to improve performance. Further improvements may be obtained by enhancing classification and correction strategies.

A key factor in the clinical translation of brain-machine interfaces (BMIs) for restoring hand motor function will be their robustness to changes in a task. With functional electrical stimulation (FES) for example, the patient’s own hand will be used to produce a wide range of forces in otherwise similar movements. To investigate the impact of task changes on BMI performance, we trained two rhesus macaques to control a virtual hand with their physical hand while we added springs to each finger group (index or middle-ring-small) or altered their wrist posture. Using simultaneously recorded intracortical neural activity, finger positions, and electromyography, we found that decoders trained in one context did not generalize well to other contexts, leading to significant increases in prediction error, especially for muscle activations. However, with respect to online BMI control of the virtual hand, changing either the decoder training task context or the hand’s physical context during online control had little effect on online performance. We explain this dichotomy by showing that the structure of neural population activity remained similar in new contexts, which could allow for fast adjustment online. Additionally, we found that neural activity shifted trajectories proportional to the required muscle activation in new contexts. This shift in neural activity possibly explains biases to off-context kinematic predictions and suggests a feature that could help predict different magnitude muscle activations while producing similar kinematics.

Brain-machine interfaces (BMIs) can restore motor function to people with paralysis but are currently limited by the accuracy of real-time decoding algorithms. Recurrent neural networks (RNNs) using modern training techniques have shown promise in accurately predicting movements from neural signals but have yet to be rigorously evaluated against other decoding algorithms in a closed-loop setting. Here we compared RNNs to other neural network architectures in real-time, continuous decoding of finger movements using intracortical signals from nonhuman primates. Across one and two finger online tasks, LSTMs (a type of RNN) outperformed convolutional and transformer-based neural networks, averaging 18% higher throughput than the convolution network. On simplified tasks with a reduced movement set, RNN decoders were allowed to memorize movement patterns and matched able-bodied control. Performance gradually dropped as the number of distinct movements increased but did not go below fully continuous decoder performance. Finally, in a two-finger task where one degree-of-freedom had poor input signals, we recovered functional control using RNNs trained to act both like a movement classifier and continuous decoder. Our results suggest that RNNs can enable functional real-time BMI control by learning and generating accurate movement patterns.