Following his death in 1936, Gwaltney's wife Estelle lived in the house until her own death in 1954. One of their sons, Julius, inherited the house and moved his family into the property. Not only that, but control of the meat company was split between Julius and his brother Howard.
The house, now known as the P. D. Gwaltney Jr. House, has never left the possession of the Gwaltney family. As of the house's inclusion into the National Register of Historic Places in 1999, Julius' wife Henrietta still resided on the property.