Hill Memorial also has a cornerstone that was laid on May 23, 1916. Inside is a portrait of Joshua Hill, the proceedings from Sussex County's Sesqui-Centennial Anniversary from 1903, a program for the laying of the cornerstone, an image of Memorial Hill, a photograph of the building that was removed to build Hill Memorial, a list of life members of the historical society, and numerous other papers and photographs.
The Hill Memorial is made of two floors. The first floor exhibits local history. The first floor also features a fireplace that was built by George Sharp with stones that are historically and geologically significant. For instance, two large stones in the fire place are from the Thomas Anderson House before it was torn down. Anderson was a patriot during the Revolutionary era. There is a display on the second floor with objects like a partial skeleton of a Mastodon. On the second floor is also a display of veterans from the Revolutionary War, the Pennsylvania Insurrection of 1794, the Quasi War with France, the War with Tripoli, and the War of 1812. These veterans are buried in the Old Newton Burial Ground. Displays on the second floor are often known to rotate, so visitors can continue to visit the historical and see the objects that the Sussex County Historical Society has in its collections.