Deep learning-based molecular generative models have garnered considerable interest in the field of de novo drug design. However, most extant models focus on either ligand-based or structure-based strategies, thereby failing to effectively harness the combined knowledge derived from both ligands and the structure of the binding target. In this article, we introduce LS-MolGen, a novel ligand and structure-integrated molecular generative model. This model synergistically combines representation learning, transfer learning, and reinforcement learning. The targeted knowledge assimilation from transfer learning, coupled with an advanced exploration strategy in reinforcement learning, empowers LS-MolGen to efficiently generate novel and high-affinity molecules efficiently. The comparable performance of our model is affirmed through multiple evaluations, including EGFR, DRD3, CDK2, AA2AR, ADRB2, and a dedicated case study of inhibitor design for SARS-CoV-2 Mpro. The results indicate that LS-MolGen performs better than other ligand-based or structure-based generative models in de novo designing promising compounds with novel scaffolds and high binding affinity. This proof-of-concept study signifies the potential of our ligand- and structure-based generative model, LS-MolGen, as a promising new tool for target-specific molecular generation and drug design.