The land on which the house is located was the site of a barracks built there in 1839 in response to increased tensions between Britain and the United States. The site was abandoned in 1845 and only one of these buildings, a two-story officers barracks, survived. It was bought by Albert P. Landing who expanded it. In 1883, Dexter Ramsay, a prominent local manufacturer, bought the home for his daughter, Mary, the wife of Ansley Wilcox. They expanded the house to essentially how it appears today. They passed away in the 1930s and the house became a restaurant until 1961. Renovations began in the 1970s.