Particle size plays a key role in the performance of metal nanoparticles (MNPs). However, the size-controlled synthesis of MNPs still represents a challenging task. In this work, we revealed a strong solvent effect on the growth of palladium nanoparticles (PdNPs), which was directed by a porous [2 + 3] organic molecular cage (OMC, Phos-cage) containing triphenylphosphine moieties. PdNPs with different average diameters of 0.8, 1.2, and 3.3 nm supported by Phos-cage were obtained by simply varying the reaction media. The catalytic performance of such ultrafine PdNPs in the reduction of p-nitrophenol and a Suzuki-Miyaura coupling reaction has been studied, which clearly shows size-dependent catalytic activity and stability. The knowledge gained in this study, controlling the size of PdNPs supported by the OMC template in different solvents, will open new possibilities for size-controlled synthesis of ultrafine MNPs with high catalytic activity and stability.