Adhesion between asphalt-aggregate and cohesion within asphalt mastic has a significant effect on the performance of asphalt mixtures. Conventional testing methods and studies normally only focus on one of the damage modes (adhesion or cohesion), although in real asphalt mixtures (asphalt-aggregate system), these two failure modes can happen together depending on the material and loading conditions. The objective of this paper was to characterize the interface bonding characteristics of the asphalt-aggregate system using the thin-film interface test and the fracture property parameter, critical state energy density (CSED), and identify the key factors that could affect the interface bonding characteristics. The thin-film interface test simulated a more realistic bonding condition and monitored the damage between thin-film asphalt and aggregate. Experimental results showed that the effects of temperature and loading rate on the interface bonding characteristics were significant. The time-temperature superposition principle is found to work in both the linear viscoelastic range (dynamic modulus test) and the damage domain (interface bonding fracture test). Binder type, degree of aging, and types of aggregates all played an important role in the bonding performance for an asphalt-aggregate system.