Leslie Samuel Smith

Extraction of relevant lip features is of continuing interest in the visual speech domain. Using end-to-end feature extraction can produce good results, but at the cost of the results being difficult for humans to comprehend and relate to. We present a new, lightweight feature extraction approach, motivated by human-centric glimpse-based psychological research into facial barcodes, and demonstrate that these simple, easy to extract 3D geometric features (produced using Gabor-based image patches), can successfully be used for speech recognition with LSTM-based machine learning. This approach can successfully extract low dimensionality lip parameters with a minimum of processing. One key difference between using these Gabor-based features and using other features such as traditional DCT, or the current fashion for CNN features is that these are human-centric features that can be visualised and analysed by humans. This means that it is easier to explain and visualise the results. They can also be used for reliable speech recognition, as demonstrated using the Grid corpus. Results for overlapping speakers using our lightweight system gave a recognition rate of over 82%, which compares well to less explainable features in the literature.

Terry Sejnowski's 2020 paper [arXiv:2002.04806] is entitled "The unreasonable effectiveness of deep learning in artificial intelligence". However, the paper doesn't attempt to answer the implied question of why Deep Convolutional Neural Networks (DCNNs) can approximate so many of the mappings that they have been trained to model. While there are detailed mathematical analyses, this short paper attempts to look at the issue differently, considering the way that these networks are used, the subset of these functions that can be achieved by training (starting from some location in the original function space), as well as the functions to which these networks will actually be applied.

We fabricate a MEMS microphone that incorporates a graphene-based membrane that vibrates in response to acoustic forcing. We employ a novel fabrication process, where a graphene/PMMA bilayer membrane is transferred over a cavity on a separate chip before being affixed to the surface of another chip containing an electrode, resulting in the fabrication of a moveable capacitor with a membrane-to-electrode gap of 8 μm. The gap, which is less than half the size of other reported graphene membrane-based audio transducers, allows for the device to operate with low DC bias voltages of about 1 V and, when integrated with a custom-designed readout circuit, demonstrates a sensitivity to sound pressure between 0.1 mV/Pa and 10 mV/Pa across the range 100 Hz to 20 kHz. As well as a sensitivity that is comparable to previous work, the flat frequency response is stable when the sound pressure is varied between 70 dBSPL and 80 dBSPL, with the sensitivity value not varying by more than 0.2 mV/Pa.

It seems that seeing others in slow-motion by heroes does not belong only to movies. When Lionel Messi plays football, you can hardly see anything from him that other players cannot do. Then why he is not stoppable really? It seems the answer may be that opponents do not have enough time to do what they want; because in Messi's neural system, time passes slower. In differential equations that model a single neuron, this speed can be generated by multiplying an equal term in all equations. Or maybe interactions between neurons and the structure of neural networks play this role.

The CARMEN portal (developed under a UK EPSRC grant from 2006, and under a UK BBSRC grant until 2015) launched in 2008 (https://portal.carmen.org.uk). It allows researchers to store and share electrophysiology datasets and analysis tools, with a sophisticated security system, enabling researchers to collaborate and share datasets and tools. It was intended as a place where researchers could work collaboratively on data as though the analysis and modelling software, and the datasets were local to the researchers. Considerable effort was put into developing the metadata system [1] and the internal data format, Neurophysiology Data Translation Format (NDF) [2]. Metadata is critical for allowing re-use of datasets and tools. Nonetheless, users report that inputting metadata discourages them from uploading data. This problem has two aspects. Firstly, neurophysiology metadata is complex and multi-faceted, starting from the relatively simple structural metadata (about the detailed file format) and ranging through more complex issues (e.g. the type and precise geometrical arrangement of electrodes and the detailed description of the neural tissue being recorded from) to even more difficult descriptions of the stimuli being used while recordings are being made (if they are from a live animal). Secondly, neurophysiologists do record this information, but often in a physical lab book. Uploading it feels like duplication of effort, and this is made worse because of the constraints imposed by the (generally) form-based input system for metadata failing to match the text in the lab book. The obvious solution to this is the use of an electronic lab book, so that the data is only entered once, but here too, standardisation is critical, particularly for extracting metadata. Because most electrophysiology research is performed on equipment manufactured by a number of companies, and because they do not agree on standard formats, there are many different data formats for electrophysiology. Making analysis tools work across the plethora of formats for electrophysiology datasets entails an internal data format: this is critical, because the analysis and visualisation tools need to be able to interpret data without having to interrogate the format first. Services were provided to translate user data into this format. However, the format used [2] was designed at a time when overall formats were still changing. For example, HDF5 was not used, unlike the more recent formats proposed by Teeters et al [3] in their recent Neuroscience Without Borders project paper. Much CARMEN usage has been by groups collaborating geographically: the security system allows sharing datasets and analysis tools with a high degree of granularity. These users have made little use of workflow technology. However, this may be because there were problems in providing workflow capability. The capability now provided includes a graphical editor, but this does not permit loops in the workflow, seriously reducing its usefulness (for example, it does not currently support applying a workflow over a range of parameters). This is an example of ease of use taking precedence over usefulness: the intended users were thought not to be programmers, so a visual interface for building workflows was provided. However, it is very difficult to provide the range of capabilities in a visual programming system that can be provided by a textual system – and this can be seen in the rarity of visual programming languages for general-purpose programming. In particular, providing loops and if-branches that allow for alteration of parameters is much easier in a scripting language. The unintended effect was to make sophisticated use of the workflow facility essentially impossible. The recently released integrated capability to eyeball datasets has been useful, more so than the original tool used for data visualisation, which required the use of a separate program. This is an example of how developments in the technology alter the way in which one writes systems: in this case, the inclusion in HTML5 of an integrated (and cross-browser) capability for graph drawing changed the way in which graphics was best presented. Retinal neurophysiologists have been the most effective users of the CARMEN collaboratory. A cross-animal study of excised retinal development has re-used data from different animals, using workflows to perform identical analysis on the raw data from different experiments: all the data and tools are publicly available [4]. Aspects of CARMEN’s implementation are now out-of-date. As noted earlier, a new internal data format would use HDF5, and the more recent attempts at standardisation. Issues of storing metadata have not been entirely solved, although there is more recent work on metadata [5]. However, the issue of ensuring single entry metadata is not yet solved. On the more technical side, services currently run on single cores, and much of the client-side was coded as Java applets (for example, data uploading). A new implementation would change these early decisions. Very recently, a new collaboratory system has been unveiled by the Human Brain Project [6]. This has come from a large project, and covers much more than neurophysiology. It includes much of what CARMEN attempted in its SP5 Neuroinformatics Platform. The release to non-members of HBP is very recent, and as yet no comparisons have been made. It is, however, clear that the collaboratory concept remains very much alive, but that the technology to support it has moved on considerably since the CARMEN project was implemented.

Neuromorphic systems are used in variety of circumstances: as parts of sensory systems, for modelling parts of neural systems and for analog signal processing. In the sensory processing domain, neuromorphic systems can be considered in three parts: pre-transduction processing, transduction itself and post-transduction processing. Neuromorphic systems include transducers for light, odours and touch but so far neuromorphic applications in the sound domain have used standard microphones for transduction. We discuss why this is the case and describe what research has been done on neuromorphic approaches to transduction. We make a case for a change of direction towards systems where sound transduction itself has a neuromorphic component.