When the Civil War began, Smith moved to Washington DC with the goal of supporting the Union against the Confederate States of America. Smith became a recruiter for the all-black 5th Cavalry. After the war, Smith served three full terms in the Massachusetts House of Representatives. He was elected in 1868, 1869, and 1872. Smith was just the third African American to serve in this position. After his time in government, Smith moved to this house on Pinckney Street around 1878. At that time, he became the first African American to serve on the Boston Common Council.
As a council member, Smith worked to secure the first positions for African Americans on Boston's police force. Smith's influence continued to the next generation with his daughter, Elizabeth Smith, who became the first African American teacher to at an integrated public school in Boston at the Phillips School. Smith died at this home while surrounded by family on November 4, 1906.