This paper presents the results of investigations on the rheological properties of modified asphalt binders and their influence on the performance of asphalt mixes. Asphalt mixes with modified binders such as styrene butadiene styrene polymer, crumb rubber, natural rubber, and waste plastics were evaluated for their rheological properties and compared to the properties of asphalt mixes with unmodified VG30 (viscosity grade) asphalt binder. The dynamic modulus values and rutting characteristics of the asphalt mixes were studied with due consideration to different levels of aging and temperature variations. Studies on the rheological properties showed that the energy dissipated by unmodified asphalt (VG30) binder is higher than that of modified asphalt binders. Long-term aged natural rubber and waste plastic modified asphalt binders showed significant increase in the properties compared to unmodified asphalt binder (VG30). Reduced temperature susceptibility of polymer modified asphalt binder showed that only polymer modification can enhance both high temperature rutting resistance and low temperature thermal cracking resistance of asphalt mixes. The transient nature of polymer modified asphalt mix from viscoelastic solid-like to viscoelastic fluid-like condition is significantly shifted to higher temperature compared to that of the mix with unmodified asphalt binder. Aging and rutting indices showed that rubber modified asphalt mixes are highly susceptible to aging. Statistical analysis of test results showed that the process of modification of asphalt binder, aging, and temperature during the test influence the rheological and rutting characteristics of asphalt mixes significantly. Correlation between the asphalt binder properties and its influence on the rutting resistance are found to be statistically significant. The analysis using least significant difference showed that polymer modified asphalt binder significantly improves the aging and rutting resistance of asphalt mixes compared to unmodified asphalt binder.