Brown married Elizabeth Gunn, the daughter of Calvin Gunn, a printer by trade, former Mayor of Jefferson City, and owner of the lot, in 1858. The property transferred to her after her father died and Benjamin, who was then governor of Missouri, became the trustee. Mrs. Brown grew up in a log house formerly on the lot where the rowhouses were built. Elizabeth was only 15 years old when she met Benjamin; she married the 32-year-old St. Louis legislator and editor of the Missouri Democrat newspaper three months later. The former governor's mansion was razed in October 1871 and the new mansion was not ready for occupancy until January 1872. In the interim, it is thought that Brown and his family lived in the row house building that Brown commissioned to be built. Brown's mother-in-law and sister-in-law are said to have lived in the rowhouses.
The three parts of the structure have 109, 111, and 113 Madison Street addresses. The three parts were sold separately after 1881. The 109 house became a boarding house and later was used for office space. The Cole County Historical Society purchased the building in 1946 and completed renovations two years later. Later, 111 Madison Street was added to the historical society's holdings.
Docents offer tours of the museum at a cost of $3 per person ($2 for age 55+, $1 for schoolchildren, free under 5). A new elevator makes all floors of the museum wheelchair-accessible. Many items in the Brown Room were donated by descendants of Benjamin and Elizabeth. Also displayed are furniture and items from the mansion of the first mayor, Thomas Lawson Price; that mansion was demolished to build the Missouri Supreme Court building. The society also houses the James Library, the DeLong Room which displays first ladies of Missouri's gowns, and the McHenry Room on the Civil War.
Behind the rowhouses, the Margaret Upschulte House now stands. The circa 1865 structure was moved to this location from the 300 block of W. High Street in 1980. The brick house is part of the historical society and is used for social functions.