Charles Allis' father was Edward P. Allis, who came to Milwaukee from New York and founded the E.P. Allis Company, which supplied water pipes and pumping stations for Milwaukee's water system. By the time of Edward's death in 1899, the company had grown to 1,500 workers and was the largest producer of sawmills, flour mills and steam engines in the country. Charles Allis built upon his father's success, merging with Chicago-based Fraser & Chalmers, Dickson Mfg. Co., and the Gates Iron Works to become the Allis Chalmers Company in 1901, with Allis as its first president.
Charles Allis and his wife were prominent collectors of fine art. They traveled extensively, picking up pieces around the world to add to their collection. Eventually their collection required a home that would be protected from damage or destroyed by fire. Thus Allis built the Tudor-style mansion that today is known as the Charles Allis Art Museum in 1911. German architect Alexander Eschweiler designed the 2.5 story house with inspiration from Tudor prototypes, and the Tudor rose design features prominently on the inside. The walls were made from concrete to protect the collection from fires, with trimmings of sandstone and marble. The location and design of the house were built with full intention of bequeathing the house and contents to the people of Milwaukee after the Allis' deaths.
Sadly, Charles Allis did not get to enjoy the house for long. Seven years after moving into the house, he passed away in 1918. Sarah lived in the house until 1945, when she, per Charles' request, passed the house and entire collection to the public. At first the Milwaukee Public Library system assumed responsibility for the building and made it a library/art museum. In this iteration it was added to the National Register of Historic Places in 1975. In 1979, Milwaukee County took over the management of the house and collection, renaming it the Charles Allis Art Museum, which is its name today.