The citizens of Newburgh planned to stage a defense on their own. While the Confederates were busy emptying the arsenal the townsfolk gathered at a nearby hotel and armed themselves for battle. Johnson, realizing that Martin would be returning very soon, confronted the people gathered in the hotel and demanded them to disarm. During this confrontation he took the Union commander prisoner. A force called the Newburgh Home Guards were ready to wage battle against Johnson and his men, but Johnson called their attention to a small garrison of two cannons just across the river that were prepared to level the town on his command. The Guards backed down while Johnson and his men finished clearing the arsenal out. However, what the Guards didn't know was that the cannons were a complete ruse. The raiders had built them out of long stove pipes, wagon wheels, and axles. It was this ploy that earned Johnson the nickname Stovepipe. The ruse worked and the raiders started back across the Ohio River with their booty.
As the raiders began the last trip across the river, they happened across a Union gunboat and troop transport. The Union ship tried to maneuver to stop the Confederates from getting back to Kentucky, but Johnson and his men quickly made their way to the peninsula at the mouth of the Green River and opened fire on the Union ship. The Union soldiers thought they were facing a much larger contingent than they did, so they returned fire and fled the area. Two Union soldiers were wounded in the exchange, but none of the raiders suffered any injury at all.
In reality, the raid had the opposite effect that Johnson had hoped. Governor Oliver Morton telegraphed Washington, which in turn sent massive reinforcements to the area. These Union troops over the next several months started taking border towns down into Kentucky and paved the way for campaigns down into Tennessee and eventually Vicksburg.