Roseland Cottage was built to provide the family with an escape from the crowds and chaos of New York. Architect Joseph C. Wells designed the house, along with a bowling alley, woodshed, and carriage house. In 1850, the Bowens added an impressive parterre garden: 3000 square feet containing boxwood hedges and flower beds with dozens of plant species.
The Bowens stayed at Roseland every summer to engage in summer activities, sports for the boys, and more ladylike activities such as sewing and reading for the girls. Lucy Bowen died from complications giving birth to her tenth child in 1863. Henry remarried two years later. The family continued using Roseland as a summer retreat, and in the 1880s updated the house's decor.
In 1870, Henry Bowen began throwing large 4th of July parties, hosting hundreds of revelers at Roseland. These guests included Ulysses S. Grant, Benjamin Harrison, Rutherford B. Hayes, and William McKinley. Bowen became increasingly involved in politics over time. An abolitionist, he founded The Independent, an anti-slavery newspaper.