An accurate dynamic 3D deformation measurement method realized by the combination of phase-shifting speckle interferometry and speckle correlation is proposed. By converting the speckle field and the reference field into a circular polarized and linear polarized state, the three-step phase-shifting speckle interferograms and one specklegram were recorded directly and simultaneously within a single image by using a polarization camera. Then, the out-of-plane deformation was demodulated from the synchronous phase-shifting fringe patterns, and the in-plane deformation was measured by performing correlation calculations by using specklegrams with the effect of the reference field ignored. Thus, the full-field 3D deformation was obtained precisely. Experimental results demonstrated the accuracy and dynamic measurement ability of the proposed method, which is compact and feasible for actual dynamic scenes.