Life was tough for African Americans in New York City in the 1800s, as they were given very unfair treatment. Blacks were forced to stand on the desk of Hudson River Steamers even when it was raining as African Americans were not permitted to purchase cabins, and several other places were segregated as well, such as churches and schools. Tensions were particularly high between African Americans and newly arrived immigrants in New York City, as Congress, facing an ongoing Civil War, instituted a draft to bolster the ranks of the Union Army. Immigrants were particularly targeted for service as they were promised things like money and food, two things immigrants lacked when arriving in America. With the Emancipation Proclamation being passed, some saw the Civil War as a war for slavery and many immigrants knew they would have to compete with recently freed African Americans for jobs.
Tensions eventually boiled over on July 13th 1863, when a mob of over 500 people converged at the Ninth District's provost marshal's office on Third Avenue and 47th street. The mob ransacked the building and set it on fire, and dismantled the vehicles of the fire brigades who tried to put out the blaze. The rioters also cut telegraph wires to keep from other parts of the city from knowing about the riot. Since the city was lacking a militia at the time, the police force was the only defense the city had against the mobs. Many buildings were set ablaze in the riots, including the Bulls Head Hotel, the Mayor's office, and several police stations.
The rioters began targeting black people as their scapegoats and main sources of anger, with multiple African Americans being beaten, tortured, and lynched throughout the city. The Colored Orphan Asylum was one of the main targets of the rioters, claiming that the orphanage was a symbol of white charity to blacks and of black upward mobility. The mob looted and burned the building at 4 PM on July 13th, amazingly the 233 orphans who called the asylum home escaped both the blaze and the wrath of the mob. Two firefighters, Chief Engineer Decker and Paddy McCaffrey, attempted to put out the flames but it was no good, the Colored Orphan Asylum was lost.
The riots lasted until July 16th, when the New York State Militia returned to the city and fought back against the rioters. 120 people were killed in the riots, and many African Americans fled New York City. The orphanage was reestablished in a number of places throughout the years, first relocating to a hamlet in the Upper East Side, then going to a site on 143rd Street and Amsterdam Avenue in Harlem in 1868, then relocating one last time to 261st Street and Palisade Avenue in the Bronx in 1907. Facing many pressures, the Colored Orphan Asylum changed its name to the Riverdale Children's Association in 1944 and continues to operate under this name.