During the 1840s and 1850s some of those same functions were shared by the Stone Academy on Chapel Avenue. When that structure burned during the winter of 1864, a much larger main Academy was planned at the intersection of School and Main Streets. As soon as that new building was underway in 1865, trustees voted to convert Bulfinch Hall to a gymnasium for seminarians and academy boys. With facilities for gymnastics, bowling, and boxing, the new gym opened on February 14, 1867 and was heavily used and poorly maintained for nearly thirty years. In 1896 a fire gutted the building completely.
In 1902 architect Guy Lowell renovated the brick shell for the school's main dining hall, which was on the first floor with a banquet room on the second and the kitchen and service areas in a new wing on the east. In 1936 Bulfinch Hall was rehabilitated as the English department. Perry, Shaw, & Hepburn, architects of Colonial Williamsburg, created an opulent colonial revival interior for classes, debates, and offices. The building was christened Bulfinch Hall at that time. In 2012 the building was renovated again to make it handicapped accessible, replace all systems, add new technology, and expand the structure modestly in the form of a single-story southeast wing attached to the 1902 east wing. Bulfinch, now served by an elevator, contains fourteen seminar rooms and one large multimedia room, four faculty-student conference rooms, and a large faculty room (formerly the debate room). Ann Beha Architects designed the 2012 renovation, and Consigli Construction was the general contractor. The project budget was $6.25 million. By contrast, the original building cost $13,252.73 in 1819 dollars. Of Asher Benjamin's original design, only the walls remain intact.