Stamatis Mastromichalakis has a total of 2 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    EchoMuon: Cross-Timescale Gating in Muon, and the Learning-Rate Confound in Gated Optimizers

    by Stamatis Mastromichalakis

    Muon orthogonalizes the momentum matrix, granting every singular direction of the update equal trust. EchoMuon prices that trust by each direction's echo: its support in a second, slower momentum buffer. This paper reports what that buys, and what measuring such a gate honestly costs. The gate is a contraction, not a reallocation. Its multiplier never exceeds one in 5,760 logged layer-steps, and the arm carrying it sits higher on the learning-rate axis: over four cells it is penalized 3.7x less than ungated Muon for a step twice too large, and 1.6x more for one half too small, without exception in either direction. A shared learning-rate grid therefore does not compare the two fairly: when the grid sits above the joint optimum, the contracting arm is rescued and the other is not, by an amount comparable to the margins being measured. Our initial hypotheses about the gate were wrong on this point, so we re-ran the full suite under corrected selection. All eight CIFAR arm-cells had picked the bottom edge of a single shared grid; widening it and selecting on a held-out split cuts those margins by 42 to 80% and leaves none individually significant. Tiny ImageNet, whose optimum the original grid did contain, is unchanged at +1.65 pp (t = +7.4) pooled over its two cells, and on the clean cell a norm-matched scalar control reproduces only 30% of the gain, so what remains is directional rather than a step-size effect. Three further boundaries are measured. The FineWeb advantage falls from -0.031 nats at 0.3 tokens per parameter to -0.003 at 0.9, verified not to be a learning-rate artifact. Across ten vision cells the margin scales with the baseline's error rate and crosses zero near 90% accuracy. On byte-level text EchoMuon wins on clean enwik8 (t=-10.1) and loses only under input corruption, which replaces the tokenization boundary we first proposed. We release 1,815 runs, the re-run suite behind these numbers, and the reporting practice we recommend: publish the margin by which each learning rate beat its runner-up. Here a pick that won by 0.0026 in validation loss reversed under an equally valid split, with a 1.45 pp consequence on test accuracy.

  • Open Access

    OptiRoulette Optimizer: A New Stochastic Meta-Optimizer for up to 5.3x Faster Convergence

    by Stamatis Mastromichalakis

    This paper presents OptiRoulette, a stochastic meta-optimizer that selects update rules during training instead of fixing a single optimizer. The method combines warmup optimizer locking, random sampling from an active optimizer pool, compatibility-aware learning-rate scaling during optimizer transitions, and failure-aware pool replacement. OptiRoulette is implemented as a drop-in, "torch.optim.Optimizer-compatible" component and packaged for pip installation. We report completed 10-seed results on five image-classification suites: CIFAR-100, CIFAR-100-C, SVHN, Tiny ImageNet, and Caltech-256. Against a single-optimizer AdamW baseline, OptiRoulette improves mean test accuracy from 0.6734 to 0.7656 on CIFAR-100 (+9.22 percentage points), 0.2904 to 0.3355 on CIFAR-100-C (+4.52), 0.9667 to 0.9756 on SVHN (+0.89), 0.5669 to 0.6642 on Tiny ImageNet (+9.73), and 0.5946 to 0.6920 on Caltech-256 (+9.74). Its main advantage is convergence reliability at higher targets: it reaches CIFAR-100/CIFAR-100-C 0.75, SVHN 0.96, Tiny ImageNet 0.65, and Caltech-256 0.62 validation accuracy in 10/10 runs, while the AdamW baseline reaches none of these targets within budget. On shared targets, OptiRoulette also reduces time-to-target (e.g., Caltech-256 at 0.59: 25.7 vs 77.0 epochs). Paired-seed deltas are positive on all datasets; CIFAR-100-C test ROC-AUC is the only metric not statistically significant in the current 10-seed study.

  • Open Access

    Parametric Leaky Tanh: A New Hybrid Activation Function for Deep Learning

    by Stamatis Mastromichalakis

    Activation functions (AFs) are crucial components of deep neural networks (DNNs), having a significant impact on their performance. An activation function in a DNN is typically a smooth, nonlinear function that transforms an input signal into an output signal for the subsequent layer. In this paper, we propose the Parametric Leaky Tanh (PLTanh), a novel hybrid activation function designed to combine the strengths of both the Tanh and Leaky ReLU (LReLU) activation functions. PLTanh is differentiable at all points and addresses the 'dying ReLU' problem by ensuring a non-zero gradient for negative inputs, consistent with the behavior of LReLU. By integrating the unique advantages of these two diverse activation functions, PLTanh facilitates the learning of more intricate nonlinear relationships within the network. This paper presents an empirical evaluation of PLTanh against established activation functions, namely ReLU, LReLU, and ALReLU utilizing five diverse datasets.


  • A Moment Based Fractional Fourier Transform Scheme for MR Image Classification

    by Stamatis Mastromichalakis, Spiros Chountasis

    The classification of tomography images into unhealthy or healthy is a key preclinical process within medical image processing. The training and feature ex


  • Open Access

    SigmoReLU: An Improvement Activation Function by Combining Sigmoid and ReLU

    by Stamatis Mastromichalakis

    Two of the most common activation functions (AF) in deep neural networks (DNN) training are Sigmoid and ReLU. Sigmoid was tend to be more popular the previous decades, but it was suffering with the common vanishing gradient problems. ReLU has resolved these problems by using zero gradient and not tiny values for negative weights and the value “1” for all positives. Although it significant resolves the vanishing of the gradients, it poses new issues with dying neurons of the zero values. Recent approaches for improvements are in a similar direction by just proposing variations of the AF, such as Leaky ReLU (LReLU), while maintaining the solution within the same unresolved gradient problems. In this paper, the combining of the Sigmoid and ReLU in one single function is proposed, as a way to take the advantages of the two. The experimental results demonstrate that by using the ReLU’s gradient solution on positive weights, and Sigmoid’s gradient solution on negatives, has a significant improvement on performance of training Neural Networks on image classification of diseases such as COVID-19, text and tabular data classification tasks on five different datasets.


  • Open Access

    ALReLU: A different approach on Leaky ReLU activation function to improve Neural Networks Performance

    by Stamatis Mastromichalakis

    Despite the unresolved 'dying ReLU problem', the classical ReLU activation function (AF) has been extensively applied in Deep Neural Networks (DNN), in particular Convolutional Neural Networks (CNN), for image classification. The common gradient issues of ReLU pose challenges in applications on academy and industry sectors. Recent approaches for improvements are in a similar direction by just proposing variations of the AF, such as Leaky ReLU (LReLU), while maintaining the solution within the same unresolved gradient problems. In this paper, the Absolute Leaky ReLU (ALReLU) AF, a variation of LReLU, is proposed, as an alternative method to resolve the common 'dying ReLU problem' on NN-based algorithms for supervised learning. The experimental results demonstrate that by using the absolute values of LReLU's small negative gradient, has a significant improvement in comparison with LReLU and ReLU, on image classification of diseases such as COVID-19, text and tabular data classification tasks on five different datasets.


  • An MR image classification scheme based on fourier moment analysis and linear support vector machine

    by S. Mastromichalakis, S. Chountasis

    The classification of tomography images into non-healthy or healthy is a key pre-clinical state for patients. The training and feature extraction comprise the most time and memory consuming processes in a classifier. The aim of this paper is to provide a moment based scheme to improve the operation of classifying a given brain image as normal or abnormal. The moment features are used as input vectors to the SVM algorithm. The image is analyzed in orthogonal Fourier basis, prior to the feature extraction method. Image representation by a subset of coefficients facilitates image component selectivity, leading to extra robustness and accuracy. The proposed method has been applied on both training and classification procedures. To demonstrate the potential of the technique presented in this paper in comparison with other state-of-the-art methods, numerical simulations were performed.


  • Image representation of a signal for a fractional Fourier transform watermarking scheme

    by Stamatis Mastromichalakis, Spiros Chountasis, Michalis A. Savelonas

    This work introduces a signal watermarking scheme employing the fractional Fourier transform in the time-frequency domain. The proposed scheme employs an image representation of a signal, the so-called time-frequency representation, in order to embed the watermark, whereas the detector is applied on the fractional time-frequency distribution. Thus, it benefits from the two additional degrees of freedom, associated with the two angles of the 2D fractional Fourier transform, which allow hiding more watermarks than the ones allowed in 1D time or frequency domains. The proposed scheme is experimentally tested on mono-component and multi-component frequency modulated signals. It is shown that watermarking by means of 2D fractional Fourier transform achieves almost perfect detectability for noiseless signals and results in smaller distortions of the original 1D signal, than the ones induced by watermarking based on 1D fractional Fourier. In addition, the proposed watermarking scheme outperforms watermarking on 1D fractional Fourier domain in terms of robustness against noise.


    • 1 October 2017

    A Web Based Scheme for Image Tomography Applications

    by Stamatis Mastromichalakis