Prospect House was built for John Potter, who was a wealthy merchant from South Carolina. In 1878, a couple from New York City bought the building and then deeded the house and the surrounding 35 acres to the College of New Jersey, the predecessor of Princeton. University presidents lived in the house until 1968, when another home designed by Notman was chosen to be the official residence. Prospect House became the dinner club at this time.
Interestingly, as the university expanded, students began to walk across the grounds surrounding the house. In 1905, a particularly bad event occurred when football fans walked across it, prompting Wilson to have a fence erected around five acres surrounding the house.