In Charles' final months, he ceded control of his lands to his son, Samuel. Following Charles' death in 1799, Happy Retreat, as well as the one hundred acres of land around it, was sold to his son-in-law, Thomas Hammond. It would stay in the Hammond family until 1837, where it was sold to Judge Isaac Douglass. Of the estate's owners, Douglass was the one that did the most renovations to it. Under Charles and his immediate family, the mansion consisted of the first story of the east and west wings, plus the kitchen. Douglass added on a second story to each wing, as well as a 2 1/2 story central building to connect them. He also renamed the mansion Mordington, after his ancestral estate in Scotland.
After Douglass, the home passed through multiple hands, until it was purchased in the 1960s by Mr. and Mrs. William Gavin. As of 2014, the Gavins were looking to sell the estate. The Friends of Happy Retreat (FOHR), a non-profit historical conservation society, and the city of Charles Town are raising the money to purchase it.