Dr. Isom first arrived in Oxford in 1835 as a young 19 year-old agent of an Indian trading company. He left to study medicine in Philadelphia and returned in 1839. As the Civil War approached, he argued against secession as a delegate at the Mississippi secession convention. During the war he served as a doctor with the 17th Mississippi regiment. After the war he became one of the more innovative doctors in northern Mississippi in terms of using drugs and surgical techniques. One of his children, Sarah, earned prominence as the first woman to be named to the University of Mississippi faculty.
The house itself reflects several periods of architecture. It is believed that the four major sections of the house were built at different times. The original two-story house was constructed first, and a one-and-a-half-story rear wing, a two-story side addition, and a one-story rear addition were added later on. There are no official records of when these additions were constructed, so it is difficult to date the respective parts of the house. The tentative dates rely on the details of the architectural style of each addition.
Today, Isom Place serves as the Barksdale Reading Institute. Owners Jim and Sally Barksdale changed the home into the institute in 2000. The institute focuses on assisting Mississippi children with their literacy skills. Children from birth to 3rd grade can improve both pre-literacy and reading abilities with help from the institute.