Brockenbrough's Hill became Carr's Hill in 1854 when the property was sold to boarding house owner Dabney S. Carr. In 1867, the University of Virginia purchased Carr's Hill for 10,000 from Addison Maupin, the manager of the University's dining halls and brother to Socrates Martin, the Chairman of Faculty. Socrates Martin's administrative duties closely resembled those the later U.Va presidents.
Now part of the University's holdings, Carr's Hill became the site of a rustic, two-story frame-built structure called Blue Cottage in 1837. The cottage was destroyed the same year in a fire, prompting the University to build a set of brick dormitories with 32 rooms. The area became a popular recreational spot where students played sports, such as basketball and baseball. The Fayerweather Gymnasium was built on Carr's Hill in 1893. Hot Feet, the social club, performed plays on the site. Future soldiers used the land to practice military drills.
Prior to the American Civil War, a prewar raising of the Rebel flag took place on Carr's Hill. Classes were canceled when a group of students hoisted a Confederate flag atop the Rotunda, which was subsequently moved to a dormitory on the hill. During the war, enrollment dropped off substantially. Only 46 students, a few professors, and doctors remained on campus. When General Philip Henry Sheridan's forces arrived in Charlottesville, Professor of Law John B. Minor waved a white flag and met with General George Armstrong Custer, who posted a guard outside the University to protect the Grounds while his troops camped on Carr's Hill.
After the war, the dormitories on Carr's Hill began to deteriorate. Most of these structures were closed in 1905. The only remaining building from this period is a small, two-story brick building behind the herb garden. President Edwin Alderman and his wife, Bessie, enlisted the prominent Gilded Age architectural firm McKim, Mead, & White to construct a house on the site that could accommodate social events. This firm was also responsible for designing three other buildings on the Lawn: Old Cabell Hall, Rouss Hall, and Cocke Hall. Constructed between 1907 and 1909, the Colonial Revival style brick building was intended to compliment the Jeffersonian architecture on the campus.
Since its construction, the dwelling on Carr's Hill has housed all eight University presidents. Surrounding the home are a number of gardens. Listed on the National Register of Historic Places, the property is also home to several contributing structures. These are the Guest Cottage (built ca. 1840), Buckingham Palace (1856), the Leake Cottage (c. 1870), carriage house (1908), the landscape (site), and two iron capitals (objects) salvaged from the ruins of the Robert Mills Rotunda Annex after it was destroyed by the fire of 1895. The capitals are located north of the terrace at the edge of the property.