This home was constructed by free and enslaved laborers for James Brice starting in 1767. Brice became the mayor of Annapolis in the 1780s and later served on Maryland's Executive Council which led to him serving as the acting Governor of Maryland in 1792. Brice's father, a colonial-era judge, designed the home. The home stayed in the family for over a century and then passed to organizations such as St. John’s College. The state acquired the structure in 2014 and it is currently operated as the headquarters of Historic Annapolis.
The house was the headquarters of the International Masonry Institute before the state acquired the property in 2014 for the sum of $2.5 million.