The microhardness change of tantalum irradiated with heavy ions has been evaluated using a micro-indentation technique. The increase in hardness by irradiation was significant below ∼900K and negligible above ∼1100K. The hardness decreased steeply with temperature at ∼1000K. Correlation of the results with microstructure data showed that the strength of voids and line dislocations as barriers against deformation should be very weak relative to that of loops and TEM-invisible defects. The temperature and fluence dependence of the hardening by TEM-invisible defects, derived by the correlation, implied an accumulation process of cascade vacancy clusters during irradiation.