The Island was originally named Menauhant when it was bought along with other land surrounding Waquoit Bay by Richard Bourne. Menauhant was then bought by a former sea captain named Henry Bryant who built a large house on the island overlooking Vineyard Sound. After Bryant, Henry and Florence Washburn bought a large piece of the Island and moved into the home already there. During their time living there the island was renamed Washburn's Island. On October 26, 1926 the house caught fire and was destroyed, this would be the last time any single family would live on the island. For the next 15 years or so the island was used as a place for people to go to hangout and enjoy the water, beach parties were held and the island became vandalized over time.
During WWII the Army and Navy were looking for places to use as base for training and soldier boarding. Washburn and the rest of the Waquoit Bay area became the largest military camp on Cape Cod. The base specialized in training for amphibious warfare, mainly beach landings where boats would travel over Vineyard Sound to unload soldiers, trucks, tanks and equipment onto the shores of Washburn. Soldiers stationed here were called Cape Cod Commandos and the island itself was named by locals as the Cradle of Invasions for the amount of amphibious invasion training going on there. After being used for training, Washburn was then renovated to become a recovery center for soldiers at the end of the war. About 800 soldiers lived on the Island during the summer of 1945.
After the war the island was abandoned by the military and all structures left were demolished. The island was given back to its original owners who allowed visitors to use the island as they pleased. In 1976 the island was finally put up for sale, Massachusetts ended up being the buyer and the Island was turned into a state park which it remains today.