It is well known that the minimization of a smooth function f(x) is equivalent to minimizing its gradient norm ‖g(x)‖2 in some sense. In this paper, we propose a modified steepest descent method, whose stepsizes alternately minimize the function value and the gradient norm along the line of steepest descent. Hence the name ‘alternate minimization (AM) gradient method’. For strictly convex quadratics, the AM method is proved to be Q‐superlinearly convergent in two dimensions, and Q‐linearly convergent in any dimension. Numerical experiments are presented for symmetric and positive definite linear systems. They suggest that the AM method is much better than the classical steepest descent (SD) method and comparable with some existing gradient methods. They also show that the AM method is an efficient alternative if a solution with a low precision is required. Two variants of the AM method, named shortened SD step gradient methods, are also presented and analysed in this paper. By designing a new kind of line search, the two variants are extended to the field of unconstrained optimization.