The building was purchased by E. Hermann Altgelt in the 1860s. He lived in the residence at various points with his wife and family. After he passed away, his wife held the residence into the early 1900s. Afterward, the building fell into a state disrepair and was used for various commercial purposes, which did not serve its historical preservation. The building was only saved for preservation when it had deteriorated to the point of no longer being safe for residents.
By the 1920s, at various points, the different rooms had been the places of all sorts of industry. The front entrance had once been the Hole-in-the-Wall Bar, which advertised nickel beer, and the chapel had been a clothing store. The ball room had been a produce shop, and the room beyond it had been a tailor shop. The building had also housed a school at one point.
The city purchased the building in 1928 and restored it in 1929. Most of the rock and brick walls had remained intact and required only re-plastering. Walls that did require replacing were carefully laid on the basis of the still-existing foundations, to as closely mimic the original 18th century residence as possible. Much of the other features throughout the palace are originals, such as much of the tile flooring and the wooden lintels over the doorways. Other pieces which had to be recreated, were done as closely as possible so as to match the original, such as the hand-carved doors and the building's roof. The interior was furnished with Spanish colonial period pieces, allowing visitors to seemingly step back in time.