John Hunter, who was born in 1778, graduated from Columbia College in 1799. After college, he married Elizabeth Desbrosses, who came from a family of wealth. This provided Hunter with the opportunity to purchase the island, which he did in 1804. The land was bought for $40,000. While living in New York City, Hunter became a wealthy businessman and politician. In fact, he served in the New York State Senate for eight years. Thanks in part to a successful career, Hunter was able to build a mansion on the island, which he lived in until his death in 1852. In the mansion, Hunter often hosted important guests from around the world such as Joseph of Spain, the brother of Napolean Bonaparte. To the delight of his guest, the mansion was filled with fine wines and an expensive art college that included works by many of the old masters.
Hunter's Island was passed down to John Hunter's son, Elias, at the time of his death. Elias held onto the property until his death in 1865, at which time his son, John III, decided to not live on the property, so it was sold to New York City Mayor Ambrose Kingsland. The island was sold numerous times after that until 1889 when New York City bought the property for $324,000.
The City immediately put the mansion to use as it was used by the Society of Little Mothers as a children's welfare house. However, the mansion was later destroyed when Parks Commissioner Robert Moses initiated a project in 1937 that connected Hunter's Island to Rodman's neck, creating Orchard Beach. This project made Hunter's Island part of Pelham Bay Park. Later, in 1967, Hunter's Island became Hunter Island Marine Zoology and Geology Sanctuary, as declared by local law makers.