The plantation house had one of the most interesting architectural designs of its time. The architect used a combination of Greek Revival and Italianate designs for the plantation house. The house had a large Gothic Revival tower in the back used to supposedly keep an eye on the slaves out in the vast field. It was also said the tower was used as slave quarters. The house served as a hospital during the Civil War and several fallen soldiers were buried at the family cemetery.
After James Saunder's death, Rocky Hill Castle was passed down to his grandson, Dr. Dudley Saunders. According to the locals, the house was abandoned in the 1920s due to unexplained ghostly activity that happened daily. Once Dr. Dudley's family left the plantation, the property fell into disrepair. The house had another purchaser, H.D. Bynum, in the 1920s but they did not reside in the house. H.D. Bynum used the property for its farmland, and it was finally demolished in 1961.