Real-time time-dependent density functional theory (RT-TDDFT) is known to be hindered by the very small time step (attosecond or smaller) needed in the numerical simulation, because of the fast oscillation of electron wave functions, which significantly limits its range of applicability for the study of ultrafast dynamics. In this paper, we demonstrate that such oscillation can be considerably reduced by optimizing the gauge choice using the parallel transport formalism. RT-TDDFT calculations can thus be significantly accelerated using a combination of the parallel transport gauge and implicit integrators, and the resulting scheme can be used to accelerate any electronic structure software that uses a Schrödinger representation. Using absorption spectrum, ultrashort laser pulse, and Ehrenfest dynamics calculations for example, we show that the new method can utilize a time step that is on the order of 10-100 attoseconds using a planewave basis set. Thanks to the significant increase of the size of the time step, we also demonstrate that the new method is more than 10 times faster, in terms of the wall clock time, when compared to the standard explicit fourth-order Runge-Kutta time integrator for silicon systems ranging from 32 to 1024 atoms.