The house in which Elliott lived during his adult years was built in 1913 by Jasper developer T. R. Simmons as a simple six-room bungalow. The house was purchased by the Elliott family in September 1945, just after World War II. While raising four children, the Elliott's twice expanded their home. They remodeled in 1951 and in 1963, moving the main entrance to the north side of the building and adding three rooms and a bedroom. In 1974, the home and most of downtown Jasper was severally damaged in the famous April 3rd tornado outbreak.
After Mr. Elliott died in 1999, the house was acquired by Bevill State Community College. Under the leadership of Dr. Betsy Lavanna, grants were written to stabilize the home and develop exhibitions. In 2008, the Carl Elliott House was placed on the Alabama Register of Landmarks and Heritage. It has also been included as part of the City of Jasper’s planned Historic District.