The Millers were both involved members of Owensboro society. Elmer was involved with the Masons, the Rotary Club, and the Chamber of Commerce, while Lizzy served in the Women's Club and the Red Cross. Both were active in their church.
Miller died in 1922, and his wife Lizzie stayed on at the house until her death in 1949. The couple had no children, so the house passed to one of her nieces. During the ‘70s, and early ‘80s, the house served as an apartment building. Eventually, the buildings became derelict, and the carriage house was tom down due to its neglect. Facing demolition, the house was bought and restored in 2006 by the Kirks. In 2007 a tornado struck the downtown area of Owensboro, and the building sustained extensive damage. More repair was necessary. The main house is now a restaurant, while the former billiards room in the basement is now a bourbon bar.