Enzyme engineering involves the customization of enzymes by introducing mutations to expand the application scope of natural enzymes. One limitation of that is the complex interaction between two key properties, activity and stability, where the enhancement of one often leads to the reduction of the other, also called the trade-off mechanism. Although dozens of methods that predict the change of protein stability upon mutations have been developed, the prediction of the effect on activity is still in its early stage. Therefore, developing a fast and accurate method to predict the impact of the mutations on enzyme activity is helpful for enzyme design and understanding of the trade-off mechanism. Here, we introduce a novel approach, EnzyACT, a deep learning method that fuses graph technique and protein embedding to predict activity changes upon single or multiple mutations. Our model combines graph-based techniques and language models to predict the activity changes. Moreover, EnzyACT is trained on a new curated data set including both single- and multiple-point mutations. When benchmarked on multiple independent data sets, it shows uniform performance on problems affected by mutations. This work also provides insights into the impact of distant mutations within activity design, which could also be useful for predicting catalytic residues and developing improved enzyme-engineering strategies.