Rice is an important food crop in China, fertilization measures significantly affect soil properties and ultimately change rice yield. Thus, examining the effects of long-term green manure cultivation on the rice yield and the driving factors on rice yield, plays a crucial role in maintaining food security. Based on the long-term green manure cultivation, the treatments included no fertilizer (CK), chemical fertilizer (NPK), chemical fertilizer + Chinese milk vetch (NPK + GM), chemical fertilizer + Chinese milk vetch + rice straws (NPK + GM + S), and chemical fertilizer + Chinese milk vetch + pig manure (NPK + GM + M) treatments. One-way repeated ANOVA was used to determine the effects of diverse fertilizer modes on temporal variations in rice yields. The redundancy analysis (RDA) was used to calculate the magnitudes of the effects of soil properties on rice yield. Compared with the CK treatment, four fertilizer treatments led to significantly increased double-season rice yields (116.40-124.49%), with no significant difference between four fertilizer treatments ( p > 0.05). There were five soil properties accounting for 66.3% variation in rice yield ( p < 0.05), with available potassium (AK) being the most influential factor (32.2% variation), whereas potential of hydrogen (pH), total nitrogen (TN), total phosphorus (TP), and soil organic carbon (SOC) accounted for 15.3%, 10.5%, 5.1%, and 3.2% variation in rice yield ( p < 0.05), respectively. Thus, SOC, TN, TP, AK, and pH were major factors affecting the double-season rice yield of red paddy soil under long-term green manure cultivation. However, the results suggested that the effect of green manure on soil fertility is limited by the relatively large amount of chemical fertilizer. The results reported herein can not only increase soil fertility and improve the soil ecological environment, but also enhance and stabilize the yields of double-season rice grown in the red paddy soil of southern China.