Aggregate is the key factor to ensure the performance of asphalt mixture. In order to analyze the impact of limestone, basalt, diabase, and steel slag on the road performance of asphalt mortar, with the ratio of mineral powder to asphalt in asphalt mortar being the same as the asphalt concrete (AC)-13 Styrene Butadiene Styrene (SBS) polymer–modified asphalt mixture, the four types of asphalt mortar specimens with different aggregates were designed and molded. The high-temperature performance, low-temperature performance, water stability, and fatigue performance tests were carried out to obtain the performance parameters of asphalt mortar. The multi-index weighted decision model was used to quantitatively evaluate the influence of the four types of aggregate on the comprehensive pavement performance of asphalt mortar. Results indicated that basalt asphalt mortar had the best high-temperature performance, steel slag asphalt mortar had the best low-temperature crack resistance and fatigue performance, and limestone asphalt mortar had the best water stability. Moreover, basalt asphalt mortar had the best comprehensive pavement performance. Finally, the optimized basalt asphalt mortar and steel slag asphalt mortar were applied to the engineering field, and both met the requirement.