Claytor would only own the property for twelve years before selling it to Andrew Elliot in 1837. Elliot did not last much longer, selling it to Ambrose Page four years later. Page's family would keep hold of the house until 1921. Following the threat of demolition, the house was purchased and moved by the Lynchburg Historical Society in 1936.
The Miller-Claytor House was the fourth structure built in Lynchburg and it is the oldest building currently standing. Today, the house sees use by local events and was featured in The Mouse at Miller-Claytor House. It was added to the National Register of Historic Places in 1976.