Originally, the house was a two-rooms-up and two-rooms-down structure. From 1833 to 1866, Reverend Hoyt added four additional rooms. Reverend Hoyt served at First Presbyterian Church after Reverend Moses Waddel. According to Clarke county archives, Reverend Hoyt moved from Washington, GA to Athens to serve as pastorate of the church. He became a pastor in 1830 and served there for 36 years until his death in 1866. In the Annals of Athens, Reverend Hoyt was described as a sound theologian and fearless speaker. Two of his sons, Henry and Thomas, also joined the ministry.
Although it is unclear who occupied the house after the Hoyt family, it remained unaffected until the urban renewal movement in the 1960s resulted in its relocation from State and Hoyt Streets into the History Village Complex located where the Graduate Athens is located today. Despite this being its second move, the house remains in good condition today.
In 2001, the Hoyt House became The Hoyt House Restaurant at the Foundry Park Inn. In a Red and Black article from 2003, it was given a great review that said, the Hoyt House offers a unique taste with their American fusion cuisine. Despite its popularity, the restaurant was replaced in 2014 when the Foundry Park Inn became the Graduate Athens. The owners of the renovated hotel decided that the Hoyt House would be of better use as bridal suites. Guests can stay in the Hoyt Grand Suite on the second floor or the Hoyt Classic. While the interior has been remodeled to fit the needs of guests, the exterior is relatively similar to its original construction.