During the construction of the fort Sixty-one men from the Ninth Michigan Infantry died from disease and deprivation while building the fort. These men were buried on Memorial Hill. Today, a memorial cemetery stands therewith a thirty-four-star U.S. flag (like flown in 1861-1862) flying over it. The flag and flagpole were donated to Fort Duffield by the Second Kentucky Infantry in 1999.
As the war went on, it moved away from Fort Duffield. By 1863 the fort was no longer garrisoned at all. In fact, during 1864, Confederate guerrillas occupied it several times, deciding to burn the huts the Union soldiers had built while they were there. After the war, the fort went through various hands and had many uses. For example, it was used as a farm, rock quarry, and hunting lodge. The fort is now owned by the city of West Point and is known as a historic treasure.