The building, with a facade of terra cotta and brick, was completed in 1913. Within a short period of time, the building filled with tenants, most of whom were milliners. The building's 20,000 square foot ground floor was leased by Winifred T. McDonald, a female entrepreneur, who used the space as a tearoom and restaurant. McDonald wanted to capitalize on the increasing amount of foot traffic in the area. She spent $7,000 in alterations to the space before opening the tearoom and restaurant in 1915.
The rapid development of the Garment District was not without its critics. The Save New York Movement grew out of the frenetic pace of development in the district, largely because well-heeled shoppers didn't care to rub elbows with hundreds of factory workers who spilled out onto Fifth Avenue each day. The movement established a restricted zone, which manufacturers were encouraged to avoid.
Eventually, the Colony Arcade Building felt the effects of this movement. In 1916, one of its largest tenants, Hollow & Perlow, moved out. The company manufactured silk shirtwaists (known as blouses today) and opted to move their operations further south in search of less expensive real estate and in support of the Save New York movement.
The building continued to house mostly millinery firms through most of the twentieth century. In 2012, the building, which remained largely unchanged from the time of its construction, reopened as the Refinery Hotel. The hotel incorporated elements that reference the building's past as a central part of the Garment District. The hotel features a bar and lounge known as Winnie's, a nod to Winifred McDonald. The building is listed on the National Register of Historic Places.