This paper presents the experimental results to investigate the potential effect of fiber diameters on the performance of hot mix asphalt reinforced by basalt fibers (BFs). BFs with different diameters—7, 13, and 25 μm—were mixed into stone mastic asphalt in a dry process to prepare specimens of BF-reinforced hot mix asphalt. Several tests, such as the uniaxial penetration test, low-temperature beam bending test, immersed Marshall test, and semicircular bending (SCB) test were conducted to evaluate the performances of the specimens. Simultaneously with the SCB test, the digital image correlation (DIC) method was adopted to capture the images and further analyze the crack propagation process of the SCB specimens. According to the findings, the optimal asphalt contents became high with smaller fiber diameters. The 7-μm BFs enhanced the performances of BF-reinforced hot mix asphalt well. The crack propagation rate (v) obtained from the DIC method proved that BFs could well defer the propagation of the cracks. The slippage theory further explicated the strengthening mechanism of the BF-reinforced asphalt mixture.