Particle based methods, such as smoothed particle hydrodynamics (SPH), are well suited for simulating large distortions of solids. However, SPH methods are computationally expensive compared to state-of-the-art finite element and spectral approaches, which limits their applicability to large and complex problems. We propose a novel, computationally efficient implicit SPH method for large distortions of polycrystals. The most expensive SPH interpolation equation is interpreted as a convolution and efficiently computed as a sequence of 1D convolutions, assuming regular spatial placement of particles. Since most present SPH methods for solid mechanics already use regular particle placement over most of the domain, they stand to gain a significant computational improvement from the method proposed here, with no change to their formulation other than the kernel. Furthermore, a purely viscous hourglass control and a contact formulation using a single ghost particle per contact particle are proposed. The hourglass control is shown to suppress hourglass modes during large distortions, while the contact formulation efficiently removes discretization error in rigid contact problems. A crystal plasticity (CP) constitutive model is implemented within the SPH solver and the predictions are verified against the finite element method and a spectral solver, with computational time per iteration similar to the periodic spectral solver. The model is further validated against single crystalline Taylor impact experiments. Overall, the developed SPH model represents a practical tool for simulating large distortions of problems involving crystalline materials under complex boundary conditions.