Mrs. Millord died a few years after moving into the mansion from complications after a leg amputation. Upon Mrs. Millord's death, Villa Paula was no longer used for Cuban governmental business. Muriel Reardon purchased the home upon Millord's death and lived in the mansion until her death in 1960. The property changed hands several times throughout the 1960s and early 1970s until it was purchased by Cliff Ensor in 1974. Ensor refurbished the house and reversed years of neglect. At the time, vandals had destroyed the beautiful stained glass windows and sprayed graffiti over the walls. Ensor also employed gardners and landscapers to restore the mansion's gardens which had not been tended to in many years.
Ensor recorded experiencing many unusual occurrences that he could not explain, such as loud thuds that sounded like footsteps, apparitions of a one legged lady, piano music, and the falling of a chandelier. Believing that the property might be cursed or haunted, Ensor hired a self-described psychic medium who reported that the spirit of Paula inhabited the building and loved to play the piano. The psychic also explained the death of Ensor's three cats by claiming that Paula Millord hated cats.
The city of Miami included the structure in their list of designated historic landmarks in 1983. Ensor sold the house at auction in 1985, but the winning bidder immediately backed out of the deal after he learned of its alleged haunted nature. The property sat empty for many years until it was restored and put to use as an art museum and gallery.