Fires destroyed the houses on the plantation on two occasions. The first occurred in 1862 and another in 1865 (the latter one is believed to have been caused by General Sherman's troops but there is speculation that one of the slaves on the plantation set the fire). The house was partially rebuilt after the war and Simms lived in it until his death in 1870. A second story was added in 1893 but a hurricane blew it off. It wasn't until 1925 that another second story was added (this version of the house remains today). Descendants of the Simms family lived at the house at various times and descendants of slaves worked and lived in some of the tenant houses on the property. Many descendants still live in Bamberg.