Unfortunately, the sluggish pace of visitor growth throughout 1892 was a sign of the bad times to come for the sanitarium. When financial difficulties hit during the Panic of 1893, the Four Seasons Hotel and its Sanitarium, a 20 million dollar investment, collapsed. The Four Seasons Hotel was demolished and dismantled for scrap, leaving only the sanitarium and grounds to be sold by the Columbia Finance and Trust Company of Louisville. Eventually, Lincoln Memorial University rose up from the remains.
Grant-Lee Hall took its name from university founders who hoped that LMU would become an expression of renewed good will and fraternal feeling between what was once a divided nation. Board members had served in both the Union and Confederate armies. As such, some wanted to name the former sanitarium for one of the prominent generals from that war. Howard suggested that the two names come together, joining hands.
Now Grant-Lee Hall, the former sanitarium acted as housing for faculty. In 1904, however, the building burned to the ground after a massive fire swept through it. Workers had to sort through the rubble, with largely only the stone arches and base of building remaining. The building burned down again in 1950, giving birth to the story of the Lady in Red. According to campus legend, a woman dressed in red signaled to the students that a fire was about to occur just before the flames began.