In a previous electron irradiation study it was shown that additions of elements from the IVa and Va columns of the periodic table to 316L stainless steel could have a pronounced effect on its swelling resistance. The results suggested that vacancy binding was the main operative mechanism, increasing in strength with increasing size factor. In the current experiment these same alloys were irradiated with fast neutrons, but the influence of the size factor does not appear to be the only important mechanism. Whereas essentially precipitation was limited during electron irradiation, substantial precipitation occurred during neutron irradiation at lower displacement rates, overshadowing the effect of size factor on swelling. The addition of Hf and especially Zr, however, was particularly effective in suppressing swelling in both the electron and neutron cases, although the suppression mechanism in the neutron case appears to be a combination of both size effect and precipitation to alter the sink structure.