While initially the proposed site of a train station, the Rincon Annex would become a major hub of shops, restaurants, and offices. Compromising an entire block, the land on which Rincon Annex stands was previously owned by the Southern Pacific Railroad. The original Rincon Annex building is a former United States Post Office. Designed by Gilbert Stanley Underwood in the Streamline Moderne style it was completed in 1940. It was listed on the National Register of Historic Places in 1979.
The interior of the building is renowned for its History of California mural, composed of 27 watercolor murals painted by the Russian immigrant muralist Anton Refregier. The murals, in the Social Realism style, depict the history and struggle of California and San Francisco's poor and working classes and were finished in 1948. As the murals were completed immediately following World War II, they generated fierce controversies. Refregier's detractors criticized his artistic style and questioned his political leanings. The controversy eventually reached the U.S. Congress, where critics called for the murals to be destroyed.
In the 1980s the building was made available by the United States Postal Service for development acquisition while the rest of the property was developed into what would become known as the Rincon Center. This 23-story mixed-use building was added on the south side of the block and contains a new post office, offices, and 320 apartments. In addition, two stories were added atop Underwood's original Rincon Annex building and a large atrium was cut into the interior. The whole complex was completed in 1988.
The center has an atrium with balconies connecting the office space on the upper levels with the food court on the lower levels. The atrium is topped by a 200-foot (61 m) long skylight and features the central and distinctive floor to skylight Water Column installation artwork.