The house was mortgaged to Walter Dongan and, after Dongan's death, was purchased by Joseph Holmes Sr. The building may have been used as a tavern by Holmes, but there is no absolute evidence to indicate this. After Holmes' death, his wife gave the property to her daughter's husband, Edward Perine. The Perine family were responsible for the addition of three sections to the house, built ca. 1760, 1790, and 1830. The grounds were occupied by British soldiers during the Revolutionary War when Ann Perine lived in the house with her seven children. A Loyalist, Ann Perine later applied for reparations from the British government for the number of trees cut from her land. Ann Perine's will from 1806 mentions three enslaved persons in her estate.
The center and oldest two portions of the house are built in stone and have separate original entrances; the later additions are frame. The house is also notable for the differing heights of sections and slopes of roof. The house features a secret room. An unverified story states that the room was used to hide from British soldiers during the Revolutionary War. Since Ann was a Loyalist, this story seems to be false.
The house was later purchased by the Staten Island Antiquarian Society in 1915. The Box Tree Tea Room operated from the house in the late 1910's and 1920's. It was designated as an official New York City landmark in 1967 and listed in the National Register of Historic Places in 1976. Historic Richmond Town raised funds for a new roof in 2013 and has conducted more research on the house in recent years.