In any outbreak of infectious disease, the timely quarantine of infected individuals along with preventive measures strategy are the crucial methods to control new infections in the population. Therefore, this study aims to provide a novel fractional Caputo derivative-based susceptible-infected-quarantined-recovered-susceptible epidemic mathematical model along with a nonmonotonic incidence rate of infection. A new quarantined individual compartment is incorporated into the susceptible-infected-recovered-susceptible compartmental model by dividing the total population into four subpopulations. The nonmonotonic incidence rate of infection is considered as Monod-Haldane functional type to understand the psychological effects in the population. Qualitative analysis of the study shows that the model solutions are well-posed i.e., they are nonnegative and bounded in an attractive region. It is revealed that the model has two equilibria, namely, disease-free (DFE) and endemic (EE). The stability analysis of equilibria is investigated for local as well as global behaviors. Mathematical analysis of the model reveals that DFE is locally asymptotically stable when the basic reproduction number ( R 0 ) is lower than one. The basic reproduction number R 0 is computed using the next-generation matrix method. The existence of EE is shown and it is investigated that EE is locally asymptotically stable when R 0 > 1 under some appropriate conditions. Moreover, the global stability behaviors of DFE and EE are analyzed under some conditions using R 0 . Finally, some numerical simulations are performed to interpret the theoretical findings.