The inherent quasi-brittleness of cement-based materials, due to the disorder of their hydration products and pore structures, present significant challenges for directional matrix toughening. In this work, a rigid layered skeleton of cement slurry was prepared using a simplified ice-template method, and subsequently flexible polyvinyl alcohol hydrogel was introduced into the unidirectional pores between neighboring cement platelets, resulting in the formation of a multi-layered cement-based composite. A toughness improvement of over 175 times is achieved by the implantation of such hard-soft alternatively layered microstructure. The toughening mechanism is the stretching of hydrogels at the nano-scale and deflections of micro-cracks at the interfaces, which avoid stress concentration and dissipate huge energy. Furthermore, this cement-hydrogel composite also exhibits a low thermal conductivity (around 1/10 of normal cement) and density, high specific strength and self-healing properties, which can be used in thermal insulation, seismic high-rise buildings and long-span bridges.