The earliest portion of the building was built in 1774 as the first permanent school of the Woodbury Friends.