The current fatigue model of high-modulus asphalt mixture (HMAM) is usually built without considering the effect of loading frequency, which leads to the imprecision in the selection of fatigue parameters for the design of high-modulus asphalt pavement structures. The purposes of this study are to solve the problem of uncertainty in the characterization of fatigue performance of HMAM at various loading frequencies and to optimize the structural resistance design of high-modulus asphalt pavements. In this study, the four-point bending beam was first used to carry out the strength test on the HMAM at ten loading rates. The effects of different loading frequencies (1, 5, 10, 20, and 40 Hz) and stress levels on the four-point bending fatigue life of HMAM were then explored. The test results show that the fatigue curve derived from the nominal stress ratio deviates significantly from the theoretical strength failure point (1,1) for Nf = 1 by extending toward both ends; however, derived from the rate-dependent stress ratio (considering the effect of loading rate), it passes through the point (1,1), which reveals the correlation between the strength failure and the fatigue damage. In addition, a new method for calculating structural strength coefficients is proposed by developing a unified fatigue model for HMAM under different loading frequencies. The results of this study could predict the fatigue life of HMAM at different loading frequencies more precisely.