In order to clarify the deterioration process and damage mechanism of lightweight aggregate concrete when subjected to freezing and thawing, it is necessary to investigate the effects of lightweight aggregate on the concrete performance and analyze the relationship of freeze-thaw resistance between them. In this study, rapid freezing and thawing tests were conducted on lightweight concrete and lightweight coarse aggregate, respectively. The results of the concrete tests showed that the freeze-thaw resistance of lightweight concrete was mainly controlled by the water content of the lightweight aggregate. The tests of lightweight aggregate produced similar results: The higher the water content of the aggregate, the higher the weight loss and grain ratio passing 10 mm sieve; the lower the crushing strength, the larger the pore volume after freezing and thawing. When the lightweight aggregate with high water content was subjected to freezing and thawing, the pore volume grew larger since the aggregate structure suffered damage from frost expansion inside the structure. Micro-cracks began to form in the lightweight aggregate, then enlarged and spread to the adjacent mortar, and finally caused expansion and damage of lightweight concrete.