Gay was also a soldier. He was an ensign in the local militia before the Revolutionary War and in 1783, was appointed lieutenant colonel of the 14th regiment of the local—now patriot—militia. He participated in the Battle of Ridgefield in 1777 and provided financial assistance to the Continental Army, which apparently left him in financial ruin. He lost the house as a result.
A series of private owners lived here ever since, except for a short period when it became a boarding house. It was sold to the Society in 1951.