The house was built in 1888 and completed in 1895, and was home to Virginia Attorney General Rufus A. Ayers. In 1885, Ayers joined John Imboden, Charles Sears, George Carter, and John Taggart in an effort to turn Big Stone Gap into the “Pittsburgh of the South. Owing to the area's iron ore and coal deposits, these men believed that they could develop the sparsely-populated region, but the costs of development and a severe recession limited their ability to attract other investors and support from leading financiers.
In 1929, this historic house was purchased by Lee County native, C. Bascom Slemp, who served many years in Congress. Slemp and his sister, Janie Slemp Newman collected historical artifacts and opened their collection to the public in what was once known as the Janie Slemp Newman Museum. C. Bascom Slemp died in 1943, and three years later, the home was acquired by the state to serve as a museum. Today, the museum is managed by the Department of Conservation and Recreation.