This happened again in 1861, this time outside his church, for including Jefferson Davis, President of the Southern Confederacy, in his public prayers. This forced the doctor and his family to leave the city promptly, and they went by steamer to Panama, then to England, Paris and back to England before taking up a pastorate in New York City.
He returned to San Francisco in 1868 hoping to become pastor of Calvary Presbyterian once again, but ended up at Saint John's where he remained until his death in 1885. Most notably perhaps, in 1880, Dr. Scott married Robert Louis Stevenson and Fannie Osborne in his study.
At the beginning of the 20th century, the congregation decided to build a new church at Arguello and Lake Streets. Ground was broken on April 25, 1905 nad the first service was on Easter Sunday, April 15, 1906. Three days later the great earthquake struck, damaging much of the building quite badly. Fortunately, a year later, repairs were completed and the church was rededicated.