Sedgwick's wife, Mabel Cabot Sedgwick, was an accomplished horticulturalist and wrote The Garden Month by Month. The couple saw the hillside property in 1916 and fell in love with the views over the North Shore. They purchased the estate with the intention of using it as a summer retreat, but Mabel Sedgwick had grander plans and set about transforming the hillside into one of New England's most acclaimed gardens. She transformed the hillside but kept something of wild quality, developing her gardens right into the forest.
Following Mabel Sedgwick's death in 1937, Ellery Sedgwick married a second time, to the former Marjorie Russell, who was also a horticulturalist and a rare plants specialist. She continued and expanded the work of Mabel, designing the garden into a series of rooms. Located largely on the south-facing slope, the rooms essentially form micro-climates, which allow plants that normally would not thrive in New England, like southern magnolias, to flourish.
The home was built in 1921 and reflected the Sedgwicks' love of nature, with garden murals and paintings. The Sedgwicks designed the home to be a replica of the Isaac Ball House in Charleston, and when the South Carolina home was abandoned, Ellery Sedgwick had carvings and mantelpieces brought from Charleston and installed in the new house.
Long Hill remained in the Sedgwick family until the death of Marjorie in 1979, when the Sedgwick children donated it to the Trustees of the Reservations. It was opened to the public the following year.The gardens are meticulously maintained in keeping with the plans of both Mable and Marjorie Sedgwick.