Near the Oak Hill Cemetery, Catholic residents established their own cemetery on 40 acres of land around the 1850s after the first cemetery on Courthouse Hill closed. It was originally called the St. Patrick`s Cemetery until other Catholic congregations were established in Janesville. Then they changed the name to Mt. Olivet, in honor of the Mount of Olives in Jerusalem.
In 1901 the Mt. Olivet Cemetery constructed a small chapel. It was in the popular style of Romanesque Revival, which often emphasizes arches in door ways and windows and also corner buttresses. The Little Chapel, as it is often called, fell into neglect over the years until 1969 when a private donation was submitted in honor of the memory of the early settlers in Janesville. Because of this donation, the cemetery was able to fix and update the Little Chapel so it can continue to be used today.