On the level of modular curves that give rise to sporadic $j$-invariants

We say a closed point $x$ on a curve $C$ is sporadic if $C$ has only finitely many closed points of degree at most $\operatorname{deg}(x)$. Motivated by well-known classification problems concerning rational torsion of elliptic curves, we study sporadic points on the modular curves $X_1(N)$. In particular, we show that any non-cuspidal non-CM sporadic point $x \in X_1(N)$ maps down to a sporadic point on a modular curve $X_1(d)$, where $d$ is bounded by a constant depending only on $j(x)$. Conditionally, we show that $d$ is bounded by a constant depending only on the degree of $\mathbb{Q}(j(x))$, so in particular there are only finitely many $j$-invariants of bounded degree that give rise to sporadic points.