Confederate Colonel William H.F. Payne was captured during the conflict by Union soldiers after his horse was killed and he was thrown off. Jeb Stuart seemed to have the advantage of approximately 6,000 men to Kilpatrick's 5,000, but he was forced forfeit his position and give up the town. While attempting to flee the town, Stuart himself was almost captured, but was able to escape on his horse named Virginia; jumping an approximately fifteen foot-wide ditch filled with water. He and his men set up a defensive position outside of town while three Union regiments were placed in an arc at the south edge of Hanover and barricades were formed across the streets leading to the town square.
Kilpatrick set up his headquarters at the Central Hotel while Stuart deployed men and artillery to cover the east and northeast around Hanover. Confederate artillery destroyed buildings in Hanover as citizens attempted to hide away in their cellars from the destruction. Union forces returned artillery fire throughout the day. Brigadier General George Armstrong Custer, who had been present since near the start of the conflict, attempted to break through Stuart's left flank. Stuart repelled Union troops successfully until he ran low on ammunition. He chose to retreat the battle altogether and marched toward Carlisle, Pennsylvania as he did not wish for the Hanover conflict in the first place. Total casualties amounted to approximately 330 to 350 men; Kilpatrick suffered around 213 while Stuart suffered at least 117 casualties.