$\ell$-adic images of Galois for elliptic curves over $\mathbb{Q}$

We discuss the $\ell$-adic case of Mazur's "Program B" over $\mathbb{Q}$, the problem of classifying the possible images of $\ell$-adic Galois representations attached to elliptic curves $E$ over $\mathbb{Q}$, equivalently, classifying the rational points on the corresponding modular curves. The primes $\ell=2$ and $\ell\ge 13$ are addressed by prior work, so we focus on the remaining primes $\ell = 3, 5, 7, 11$. For each of these $\ell$, we compute the directed graph of arithmetically maximal $\ell$-power level modular curves $X_H$, compute explicit equations for all but three of them, and classify the rational points on all of them except $X_{{\rm ns}}^{+}(N)$, for $N = 27, 25, 49, 121$, and two level $49$ curves of genus $9$ whose Jacobians have analytic rank $9$. Aside from the $\ell$-adic images that are known to arise for infinitely many $\overline{\mathbb{Q}}$-isomorphism classes of elliptic curves $E/\mathbb{Q}$, we find only 22 exceptional images that arise for any prime $\ell$ and any $E/\mathbb{Q}$ without complex multiplication; these exceptional images are realized by 20 non-CM rational $j$-invariants. We conjecture that this list of 22 exceptional images is complete and show that any counterexamples must arise from unexpected rational points on $X_{\rm ns}^+(\ell)$ with $\ell\ge 19$, or one of the six modular curves noted above. This yields a very efficient algorithm to compute the $\ell$-adic images of Galois for any elliptic curve over $\mathbb{Q}$. In an appendix with John Voight we generalize Ribet's observation that simple abelian varieties attached to newforms on $\Gamma_1(N)$ are of ${\rm GL}_2$-type; this extends Kolyvagin's theorem that analytic rank zero implies algebraic rank zero to isogeny factors of the Jacobian of $X_H$.