Caldwell owned one enslaved person once he lived in North Carolina. This man was named November Caldwell, and he was Caldwell’s coachman. November was given the last name Caldwell to ensure Joseph Caldwell's ownership of him.
When Caldwell passed away on January 27, 1835, November worked as a servant under the ownership of the university. November's house was stoned by the Ku Klux Klan in 1869. November’s son Wilson went on to work as a janitor at the university. A monument at Caldwell’s grave was dedicated to him in 1858.
Caldwell’s original memorial was relocated and altered to honor the “faithful” slaves of the university, which included November and Wilson Caldwell, David Barham, and Henry Smith; all individuals who were enslaved by the university. In 1911, the University of North Carolina named a new building after Caldwell, which became Caldwell Hall.