The couple bought the estate in 1928, largely because they were enchanted by the sunken gardens. Their original plan was to demolish the entire estate, keeping only the gardens, but the Depression forced them to change their plans. Instead of tearing down the estate, the remodeled it over the next several years, finally completing the process in 1939.
The following year, the Rosens began hosting musical performances at the estate for their friends. In 1946, the couple founded the Center and began hosting events which were open to the public. The first performance was held on February 16, 1946 with about 60 people in attendance.
Over the years, Caramoor has undergone several additions and renovations. In 2016, 200 new velvet chairs were ordered to replace older ones, some of which dated to the original 1946 performance. After the death of Lucie Rosen, a wing was added to house rooms from the Rosens' New York City apartment.
Caramoor hosts numerous performances throughout the year. Most are symphonic, opera, chamber, and jazz performances. The estate is listed on the National Register of Historic Places.