Etienne Barnard
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Etienne Barnard has a total of 103 co-authors. Here are the top co-authors by number of shared publications:
- Marelie Hattingh Davel
North-West University
12 shared publications Pieter Vermeulen
Electrical and Computer Engineering University of Pretoria, Pretoria, South Africa
4 shared publicationsYonghong Yan
4 shared publicationsAditi Sharma Grover
CSIR Meraka Institute, Pretoria, South Africa
3 shared publicationsCharl van Heerden
Human Language Technologies, Meraka Institute, CSIR, South Africa
3 shared publicationsD. Casasent
3 shared publications
Latest Publications
- Open Access
Hidden Markov Model Decoding for LDPC Codes
The paper proposes an iterative Hidden Markov Model (HMM) for decoding a Low Density Parity Check (LDPC) code. It is demonstrated that a first-order HMM provides a natural framework for the decoder. The HMM is time-homogeneous with a fixed transition matrix and is based on a random walk through the encoded frame bits. Each hidden state contains a pair of two encoded bits, and parity checks are naturally incorporated into the observation model. The paper shows that by implementing a forward-backward smoothing estimator for the hidden states, decoding is efficient and requires only a small number of iterations in most cases. The results show that the LDPC decoding threshold is significantly improved compared to belief propagation (BP) on a Tanner graph. Numerical results are presented showing that LDPC codes under the proposed decoder yield a frame error rate (FER) and decoding threshold comparable to that of a Polar code where Successive Cancellation List (SCL) - Cyclic Redundancy Check (CRC) decoding is deployed. This is shown to be achieved even if the frame length is short (on the order of $512$ bits or less) and a regular LDPC code is used. 1Open AccessHidden Markov Model Decoding for LDPC Codes
The paper proposes an iterative Hidden Markov Model (HMM) for decoding a Low Density Parity Check (LDPC) code. It is demonstrated that a first-order HMM provides a natural framework for the decoder. The HMM is time-homogeneous with a fixed transition matrix and is based on a random walk through the encoded frame bits. Each hidden state contains a pair of two encoded bits, and parity checks are naturally incorporated into the observation model. The paper shows that by implementing a forward-backward smoothing estimator for the hidden states, decoding is efficient and requires only a small number of iterations in most cases. The results show that the LDPC decoding threshold is significantly improved compared to belief propagation (BP) on a Tanner graph. Numerical results are presented showing that LDPC codes under the proposed decoder yield a frame error rate (FER) and decoding threshold comparable to that of a Polar code where Successive Cancellation List (SCL) - Cyclic Redundancy Check (CRC) decoding is deployed. This is shown to be achieved even if the frame length is short (on the order of $512$ bits or less) and a regular LDPC code is used. 1 - Open Access
Minimum Phase Linear Antenna Array Design
The paper considers the design of minimum phase discrete linear arrays. The paper introduces recent advances for the design of minimum phase Finite Impulse Response filters, as applied to the design of minimum phase linear arrays. The minimum phase linear array is demonstrated to require the least number of elements of all linear arrays that are able to achieve a given magnitude pattern specification. Three example designs are presented and compared to results from the literature.
Open AccessMinimum Phase Linear Antenna Array Design
The paper considers the design of minimum phase discrete linear arrays. The paper introduces recent advances for the design of minimum phase Finite Impulse Response filters, as applied to the design of minimum phase linear arrays. The minimum phase linear array is demonstrated to require the least number of elements of all linear arrays that are able to achieve a given magnitude pattern specification. Three example designs are presented and compared to results from the literature.
- Open Access
Manifold Characteristics That Predict Downstream Task Performance
Pretraining methods are typically compared by evaluating the accuracy of linear classifiers, transfer learning performance, or visually inspecting the representation manifold's (RM) lower-dimensional projections. We show that the differences between methods can be understood more clearly by investigating the RM directly, which allows for a more detailed comparison. To this end, we propose a framework and new metric to measure and compare different RMs. We also investigate and report on the RM characteristics for various pretraining methods. These characteristics are measured by applying sequentially larger local alterations to the input data, using white noise injections and Projected Gradient Descent (PGD) adversarial attacks, and then tracking each datapoint. We calculate the total distance moved for each datapoint and the relative change in distance between successive alterations. We show that self-supervised methods learn an RM where alterations lead to large but constant size changes, indicating a smoother RM than fully supervised methods. We then combine these measurements into one metric, the Representation Manifold Quality Metric (RMQM), where larger values indicate larger and less variable step sizes, and show that RMQM correlates positively with performance on downstream tasks.
Open AccessManifold Characteristics That Predict Downstream Task Performance
Pretraining methods are typically compared by evaluating the accuracy of linear classifiers, transfer learning performance, or visually inspecting the representation manifold's (RM) lower-dimensional projections. We show that the differences between methods can be understood more clearly by investigating the RM directly, which allows for a more detailed comparison. To this end, we propose a framework and new metric to measure and compare different RMs. We also investigate and report on the RM characteristics for various pretraining methods. These characteristics are measured by applying sequentially larger local alterations to the input data, using white noise injections and Projected Gradient Descent (PGD) adversarial attacks, and then tracking each datapoint. We calculate the total distance moved for each datapoint and the relative change in distance between successive alterations. We show that self-supervised methods learn an RM where alterations lead to large but constant size changes, indicating a smoother RM than fully supervised methods. We then combine these measurements into one metric, the Representation Manifold Quality Metric (RMQM), where larger values indicate larger and less variable step sizes, and show that RMQM correlates positively with performance on downstream tasks.
- Open Access
- 8 December 2020
- Published by South African Institute of Computer Scientists and Information Technologists in South African Computer Journal
- Vol. 32  (2)
Benign interpolation of noise in deep learning
The understanding of generalisation in machine learning is in a state of flux, in part due to the ability of deep learning models to interpolate noisy training data and still perform appropriately on out-of-sample data, thereby contradicting long-held intuitions about the bias-variance tradeoff in learning. We expand upon relevant existing work by discussing local attributes of neural network training within the context of a relatively simple framework. We describe how various types of noise can be compensated for within the proposed framework in order to allow the deep learning model to generalise in spite of interpolating spurious function descriptors. Empirically, we support our postulates with experiments involving overparameterised multilayer perceptrons and controlled training data noise. The main insights are that deep learning models are optimised for training data modularly, with different regions in the function space dedicated to fitting distinct types of sample information. Additionally, we show that models tend to fit uncorrupted samples first. Based on this finding, we propose a conjecture to explain an observed instance of the epoch-wise double-descent phenomenon. Our findings suggest that the notion of model capacity needs to be modified to consider the distributed way training data is fitted across sub-units.
- 8 December 2020
- Published by South African Institute of Computer Scientists and Information Technologists in South African Computer Journal
- Vol. 32  (2)
Open AccessBenign interpolation of noise in deep learning
The understanding of generalisation in machine learning is in a state of flux, in part due to the ability of deep learning models to interpolate noisy training data and still perform appropriately on out-of-sample data, thereby contradicting long-held intuitions about the bias-variance tradeoff in learning. We expand upon relevant existing work by discussing local attributes of neural network training within the context of a relatively simple framework. We describe how various types of noise can be compensated for within the proposed framework in order to allow the deep learning model to generalise in spite of interpolating spurious function descriptors. Empirically, we support our postulates with experiments involving overparameterised multilayer perceptrons and controlled training data noise. The main insights are that deep learning models are optimised for training data modularly, with different regions in the function space dedicated to fitting distinct types of sample information. Additionally, we show that models tend to fit uncorrupted samples first. Based on this finding, we propose a conjecture to explain an observed instance of the epoch-wise double-descent phenomenon. Our findings suggest that the notion of model capacity needs to be modified to consider the distributed way training data is fitted across sub-units.
- Open Access
- 3 April 2020
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 34  (04) , 3725-3732
DNNs as Layers of Cooperating Classifiers
A robust theoretical framework that can describe and predict the generalization ability of DNNs in general circumstances remains elusive. Classical attempts have produced complexity metrics that rely heavily on global measures of compactness and capacity with little investigation into the effects of sub-component collaboration. We demonstrate intriguing regularities in the activation patterns of the hidden nodes within fully-connected feedforward networks. By tracing the origin of these patterns, we show how such networks can be viewed as the combination of two information processing systems: one continuous and one discrete. We describe how these two systems arise naturally from the gradient-based optimization process, and demonstrate the classification ability of the two systems, individually and in collaboration. This perspective on DNN classification offers a novel way to think about generalization, in which different subsets of the training data are used to train distinct classifiers; those classifiers are then combined to perform the classification task, and their consistency is crucial for accurate classification.
- 3 April 2020
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 34  (04) , 3725-3732
Open AccessDNNs as Layers of Cooperating Classifiers
A robust theoretical framework that can describe and predict the generalization ability of DNNs in general circumstances remains elusive. Classical attempts have produced complexity metrics that rely heavily on global measures of compactness and capacity with little investigation into the effects of sub-component collaboration. We demonstrate intriguing regularities in the activation patterns of the hidden nodes within fully-connected feedforward networks. By tracing the origin of these patterns, we show how such networks can be viewed as the combination of two information processing systems: one continuous and one discrete. We describe how these two systems arise naturally from the gradient-based optimization process, and demonstrate the classification ability of the two systems, individually and in collaboration. This perspective on DNN classification offers a novel way to think about generalization, in which different subsets of the training data are used to train distinct classifiers; those classifiers are then combined to perform the classification task, and their consistency is crucial for accurate classification.
- Open Access
DNNs as Layers of Cooperating Classifiers
A robust theoretical framework that can describe and predict the generalization ability of deep neural networks (DNNs) in general circumstances remains elusive. Classical attempts have produced complexity metrics that rely heavily on global measures of compactness and capacity with little investigation into the effects of sub-component collaboration. We demonstrate intriguing regularities in the activation patterns of the hidden nodes within fully-connected feedforward networks. By tracing the origin of these patterns, we show how such networks can be viewed as the combination of two information processing systems: one continuous and one discrete. We describe how these two systems arise naturally from the gradient-based optimization process, and demonstrate the classification ability of the two systems, individually and in collaboration. This perspective on DNN classification offers a novel way to think about generalization, in which different subsets of the training data are used to train distinct classifiers; those classifiers are then combined to perform the classification task, and their consistency is crucial for accurate classification.
Open AccessDNNs as Layers of Cooperating Classifiers
A robust theoretical framework that can describe and predict the generalization ability of deep neural networks (DNNs) in general circumstances remains elusive. Classical attempts have produced complexity metrics that rely heavily on global measures of compactness and capacity with little investigation into the effects of sub-component collaboration. We demonstrate intriguing regularities in the activation patterns of the hidden nodes within fully-connected feedforward networks. By tracing the origin of these patterns, we show how such networks can be viewed as the combination of two information processing systems: one continuous and one discrete. We describe how these two systems arise naturally from the gradient-based optimization process, and demonstrate the classification ability of the two systems, individually and in collaboration. This perspective on DNN classification offers a novel way to think about generalization, in which different subsets of the training data are used to train distinct classifiers; those classifiers are then combined to perform the classification task, and their consistency is crucial for accurate classification.
- 14 October 2014
- Published by Springer Nature in Language Resources and Evaluation
- Vol. 49  (2) , 327-353
Efficient data selection for ASR
- 14 October 2014
- Published by Springer Nature in Language Resources and Evaluation
- Vol. 49  (2) , 327-353
Efficient data selection for ASR
- 1 September 2014
- Published by Elsevier in Computer Speech & Language
- Vol. 28  (5) , 1066-1082
Language independent search in MediaEval's Spoken Web Search task
- 1 September 2014
- Published by Elsevier in Computer Speech & Language
- Vol. 28  (5) , 1066-1082
Language independent search in MediaEval's Spoken Web Search task
- 1 January 2014
- Published by Elsevier in Speech Communication
- Vol. 56, 83-84
Introduction to the special issue on processing under-resourced languages
- 1 January 2014
- Published by Elsevier in Speech Communication
- Vol. 56, 83-84
Introduction to the special issue on processing under-resourced languages
- 1 January 2014
- Published by Elsevier in Speech Communication
- Vol. 56, 119-131
A smartphone-based ASR data collection tool for under-resourced languages
- 1 January 2014
- Published by Elsevier in Speech Communication
- Vol. 56, 119-131
A smartphone-based ASR data collection tool for under-resourced languages