Home to the Clay County Historical Society, the Austin Whittemore House was built in 1884 by Horace J. Austin, who was among the first group of settlers to arrive in the Dakota Territory in 1860. The house features very little of the original furnishings but, as the repository for the historical society, does contain local family, city, county, church, and organization's files. The society holds meetings every month, an ice cream social in June to help raise funds, and sponsors a tour of homes in the first weekend of December. The house was added to the National Register of Historic Places in 1972 for its association with Horace Austin (and later, the Whittemore family) and its American frontier Victorian style architecture.
Horace Austin served in the Dakota First Cavalry during the Civil War and became a professional surveyor afterwards. He stayed with many families for whom he was conducting survey work. He kept diaries of his experiences which are located in the house. He and his wife had a daughter, Pansy, who would marry Arthur Whittemore, the football coach at the University of South Dakota. Their son, musician Bucky Whittemore, inherited the home after Pansy died in 1968 but did not want deal with it. A group of local citizens approached Bucky to tell him they wanted to save the house and its contents, which were about to be sold at auction. Bucky refused and everything in the house except a kitchen cupboard was saved. Finally, after the group informed Bucky of the tax money he would save if he sold the house, he gave it to the historical society.