The mansion remained in the family until 1910. Subsequent owners used the property for a variety of purposes. One owner, Douglas Patterson, built a skating rink and invited everyone in Newfane to come skate. Another owner, Mary Wagner, converted the mansion into an upscale restaurant called Green Acres in 1949. Then in 1964, it was converted into a boarding house. Unfortunately, this only lasted a few years and the building fell into disrepair and was vandalized. As the old building decayed, claims that it was haunted by the ghost of Malinda Van Horn, the wife of James Jr.— who it is believed died in childbirth—began to emerge. Fortunately, the Noury Chemical Company bought the mansion and donated it to the Society in 1987, which began the process of restoring the old building.