Literary Hall was built in 1869 to house the remaining books as the Romney Literary Society worked to reconstitute their collection. The Romney Literary Society held their last meeting in 1886. Following that, the building was used as a meeting place for both the Ancient Free and Accepted Masons and the Order of the Eastern Star.
In 1973, Literary Hall was purchased by Ralph Haines, a local lawyer. Under Haines, Literary Hall was restored and converted into a law office and museum. It was added to the National Register of Historic Places in 1979.