Construction on the house began in, or around, April 1774 and a majority of the house was probably completed before the death of the architect in November or December of the same year. Patron Matthias Hammond probably never occupied his elegant house because he abruptly left Annapolis for his family's country estate in 1776. He died in 1786 after renting the house for many years. The house passed to his nephews John and then Philip Hammond who eventually sold the house to Ninian Pinkney in 1810. Pinkney, however quickly sold the house to Judge Jeremiah Townley Chase in 1811. Judge Chase bought the house as a home for the family of his daughter Frances Townley Chase Loockerman. He was well acquainted with the house because he rented the northeast wing beginning in the late 1770s. Judge Chase's descendants lived in the house until the death of his great-granddaughter Hester Ann Harwood in 1924. Judge Chase's granddaughter married William Harwood, the great-grandson of William Buckland, the architect of the house.


Hester Ann Harwood died intestate and the house was sold in 1926 to St. John's College. The College used the house as a teaching tool for one of America's first courses taught on the decorative arts until financial necessity forced the college to sell to the Hammond-Harwood House Association in 1940. This non-profit corporation continues to own the house and uses it as a museum that is open to the public.