George Cabell, a friend to Thomas Jefferson, began construction on the home as soon as he gained access to the land. Cabell is best known for his innovations in medicine. The first of his family to graduate from medical school, Cabell studied at the College of Philadelphia. Cabell was also the physician who treated Patrick Henry during his last illness. Dr. Cabell was an early advocate for using ice and chilled drinks to help soothe fever and inflammation.
Point of Honor continued to pass through hands until 1928 when it was acquired by the city. While always maintained as a recreation facility, the home was also used as a soup kitchen during the Depression and a nursery during WWII. In 1968, the city of Lynchburg decided to make the home a museum. Restorations were completed by 1977. The Museum added furnishings that were common during the period Cabell owned the building.