Altered tissue mechanics is an important signature of invasive solid tumors. While the phenomena have been extensively studied by measuring the bulk rheology of the extracellular matrix (ECM) surrounding tumors, micromechanical remodeling at the cellular scale remains poorly understood. By combining holographic optical tweezers and confocal microscopy on in vitro tumor models, we show that the micromechanics of collagen ECM surrounding an invading tumor demonstrate directional anisotropy, spatial heterogeneity and significant variations in time as tumors invade. To test the cellular mechanisms of ECM micromechanical remodeling, we construct a simple computational model and verify its predictions with experiments. We find that collective force generation of a tumor stiffens the ECM and leads to anisotropic local mechanics such that the extension direction is more rigid than the compression direction. ECM degradation by cell-secreted matrix metalloproteinase softens the ECM, and active traction forces from individual disseminated cells re-stiffen the matrix. Together, these results identify plausible biophysical mechanisms responsible for the remodeled ECM micromechanics surrounding an invading tumor.