Lightweight convolutional neural networks are commonly selected for image classification on edge devices based on their ImageNet rankings. However, no prior studies have systematically examined whether these rankings remain valid for small-scale, domain-specific tasks. We address this gap and demonstrate that rankings can completely reverse: ShuffleNetV2, the weakest model among four representative baselines on ImageNet, becomes the strongest on our benchmarks, highlighting feature diversity rather than model capacity as the critical factor when training data is limited. Our analysis further identifies two architectural limitations that become pronounced in small-data regimes: (i) Ghost-based architectures extract features at a single spatial scale per block, lacking the multi-granularity context necessary for fine-grained discrimination, and (ii) squeeze-and-excitation (SE) attention introduces 2𝐶2∕𝑟 parameters per block, imposing a disproportionate overhead when the total model budget is only a few million parameters. To jointly address these limitations, we propose the efficient channel attention multi-scale ghost network (ECA-MGNet). This 2.50 million-parameter architecture augments a pretrained GhostNet backbone with two novel modules: a multi-scale refinement block, which injects multi-granularity context into the ghost framework via parallel 1 × 1, 3 × 3, 5 × 5, and global pooling branches, and a dual attention module, which replaces SE with ECA paired with spatial attention, reducing attention parameters by approximately 92,000× to fewer than 200. On a 10-class Flowers102 subset and EuroSAT (European satellite image dataset), ECA-MGNet matches or exceeds the accuracy of EfficientNetV2-Small and ConvNeXt-Tiny, while outperforming all lightweight baselines by 7.55 percentage points (𝑝 < 0.01, McNemar’s test).