With the of the outbreak of the Civil War the War Department was forced to take back Black Point rename it back to Point San Jose and evict all residents from the area. Point San Jose would be remade into the typical late 19th-century style fort with main parade ground, an open grassy square dedicated to drills, marches, parades and public ceremonies. The most significant military buildings, like the post headquarters, the hospital, the barracks and the mess halls were constructed on a rectangular grid around the main parade ground. Unlike other forts around the country, the officers quarters would become the homes of the evicted residents who were forced to leave their homes. Fearing an attack from Confederate ships on merchant shipping the Union in 1864 fortified Point San Jose with more coastal defenses. In 1882, the post was renamed Fort Mason to honor Colonel Richard Barnes Mason, the second military governor and commander of California.
By the start of WWII, Fort Mason would be transformed from a harbor defense post into a logistical and transport hub for American military operations in the Pacific. During World War II, Fort Mason served as the headquarters for the San Francisco Port of Embarkation, the nerve center of a vast network of shipping facilities that spread throughout the Bay Area. After the war's end and throughout the 1960s many forts along with Fort Mason would be deemed unfit for military use and be turned into national parks for recreational oppurtunities for all citizens.