Fort Gilmer housed many troops that were commanded under the removal of the Native Americans. This fort was also the temporary headquarters to U.S General Winfield Scott. General Scott was responsible for more than 4,000 Cherokee Indians while held in the confinements such as Fort Gilmer. They were starved and beaten while held in these forts. This is in part is a major reason behind the Cherokee removal being called the Trail of Tears.
It is estimated that 15,000 Cherokee Indians were forced into stockades here while under orders of Scott. Scott had lost control of the U.S Soldiers and more specifically the Georgia Militia. After the removal of Indians, there was no longer a need for the fort and what remained. In late 1838 it became an abandoned site and auctioned off.