Other prisoners included pirates, abolitionists, Union and Confederate soldiers, thieves, murderers, and others were imprisoned in the jail. A pirate named Jacque Alexander Tardy, also known as Tardy the Pirate, had an interest in poison and had a penchant of poisoning those who he had problems with. Another prisoner named Denmark Vesey, a freed slave who started a slave rebellion, was tried and hanged along with 35 other blacks. During the Civil War, Union soldiers from the 54th Massachusetts Regiment were held as prisoners of war. The movie Glory is based on the story of the regiment.
In 1911, a murderer named Daniel Duncan was the last prisoner to be hanged by the gallows until they were later dismantled. In the 1930s, the jail was in its decline and the city of Charleston wanted to build new low income apartment buildings across from the jail. After its decommission in 1939, the jail was bought by the Housing Authority of Charleston, but was left abandoned until it was bought by the American College of the Building Arts in 2000.