John Henry, one of their sons, would inherit half of the property as well as the house from his mother. The original frame of the house would be incorporated into a larger structure during his lifetime. The property would continue to be passed down, first to John's son William Wirt Henry, and then split between William's children, excluding his daughter Lucy Gray Harrison. Lucy respected her father's wishes until several years later, she decided to buy-out her other siblings, move to Red Hill, and convert the home into an 18 room mansion. While the mansion was restored by 1912, it would be destroyed in a fire in 1919. Harrison continued to live on the property in the guest house and passed away in 1944.
Congress passed a movement to make the property into a memorial in 1935, though it was conditional on it being acquired by the Secretary of the Interior, which was never completed. After Lucy Gray Harrison's death in 1944, the Patrick Henry Memorial Foundation worked to raise the funds necessary to purchase the property. By 1954, they were successful and eventually reconstructed the mansion using blueprints from 1910-1912 renovation efforts. It was added to the National Register of Historic Places in 1978 and officially made into a National monument in 1986.