In 1803, a United States trading vessel tried smuggling otter pelts to Spanish merchants, in violation to Spain's blockade of foreign goods. On March 22, the Leila Byrd, tried escaping from the bay's notorious Smugglers Cove, when it was fired on by guns from the fort. As the Byrd waited for favorable winds to help in its escape, the Byrd fired back, but in the end this battle resulted in a few shots from each side, no deaths, and only taking under an hour to finish. The United States ship suffered slight damage. This would be the only action between Spain and America along the Pacific Coast.
In 1828, now under Mexican control, the fort fired on another American vessel with similar results. In 1846, American forces with Californio allies took control of San Diego and the fort; the fort having been long abandoned prior to San Diego's occupation. Until 1873, the fort was maintained with its original purpose until modern and larger shore batteries were constructed. Fort Guijarros and the new batteries were renamed Fort Rosencrans. The batteries were active throughout the Spanish-American War, WW1, and WW2. During the Cold War, the batteries and the point became a nuclear submarine base. In 1890 a lighthouse was constructed nearby and decommissioned in 1957.
Now the fort itself is decommissioned, with remnants seen today as part of the Cabrillo National Monument. The fort's two original (and remaining) cannons are found in Fort Stockton and San Diego's Old Town Plaza. the Fort Guijarros Museum Foundation conducts archaeological digs and historical research into the history of Ballast Point.