There were few educational opportunities for former slaves and their children after the Civil War, although black churches and educators operated a variety of free and subscription schools. Funding for publicly-subsidized schools for black children lagged behind spending for white children throughout this time period. Taxpayers approved the construction of two schools for the 1898-1899 school year, one school for whites and Victoria Colored School. The principal of the school was African American educator named F.W. Gross. F.W. Gross was a graduate of Bishop University, attended three other colleges, and also earned his Masters Degree in 1902. He supervised eight staff members. F.W. Gross was a great leader in the school and encouraged students to go to college.
In 1907, C.H. McGruder became the new principal as F.W. Gross had become president of Houston College. Under the leadership of Mr. McGruder the school continued to flourish and he closely followed the district's policy and young men took industrial arts prior to graduating. In the year 1923, the school changed names and was called the Gross School. The name was in honor of F.W. Gross.
A few years after the name change, the attendance of pupils increased and soon the original building became too small. New construction soon took place in 1938 allowing for a larger, more accommodating building to be established. The property was listed as a National Register of Historical Places in 1986.