Henry Hirshfeld (born in Germany in 1834) emigrated to the U.S. at the age of 15. He subsequently embarked upon a military career, first in the Texas Military Brigade as a private under Gen. E.S.C. Robertson in Georgetown.
After the Civil War, in 1866, Hirshfeld settled in Austin and worked as the proprietor of Capital Clothing Co. He then married Jennie Melasky in 1868, whose father enjoyed financial success in the dry goods industry. As a result, Henry Hirshfeld evolved into a highly successful merchant so that by 1871, Governor Edmund Davis appointed Hirshfeld as the first vice president of the new Board of Trade. In 1890, Hirshfeld established the Austin National Bank. His two sons, Jake and Morris later served as directors at the bank.
Hirshfeld built the one-story limestone cottage on the west side of the lot in 1873, where his family lived for a dozen years. However, his financial success and eight children convinced Hirshfeld to build a much bigger home. Thus, in 1885, construction started on the grand, two-story brick and cut-stone house with Victorian and Eastlake styling; the family moved into the house in 1886. Exterior adornment highlights include stained glass, ornate woodwork, and elaborate limestone detailing.
In addition to his successful businesses, he also proved to be a leader among Austin's Jewish Community; he founded the Temple Beth Israel, the city's first Jewish congregation. As well, he was a charter member of the Mason's Hill City Lodge, and he participated in the German singing group known as Austin Sängerrunde.
Members of the Hirshfeld family continued to reside in the house until 1973 when Henry and Jennie's daughter, Leila Hirshfeld Bernheim passed away. The complex most recently served the offices for Texas A&M University in Austin from 1987 to 2016, where it got the nickname Aggie Embassy, which refers to the Texas A&M mascot (The Aggies). It was sold again in 2018 to a private group, but it remains protected from any noticeable renovations, and no building is allowed to built nearby that would deter from its viewing.