Poplar Hill Mansion was built as the manor house of a farm outside the small eighteenth century town of Salisbury,which had slowly expanded since the town's founding in 1732. Construction was begun by Major Levin Handy, originally from Newport, Rhode Island. In 1795 the Major took out a deed for 357 acres of the original 700 acre land patent called Pembertons Good Will. Construction of the house began in 1795-96. Unfortunately, because of severe medical conditions, the Major's architectural ambitions exceeded his ability to finance them. In 1803 the property was ordered to be put up for sale for payment of debts.
In 1805 the title was passed to a prominent physician in Salisbury, Dr. John Huston. When the Doctor died in 1828 there were fruit trees, vegetable crops, sheep, cattle, and eighteen slaves on the property, as well as $110 cash on hand. Huston's widow Sarah inherited the property. In the late 1840s to early 1850s, she partially subdivided it, overseeing the laying out of Poplar Hill Avenue and Isabella Street. The subdivision of Poplar Farm continued at a moderate rate before the Civil War. From the 1870s until World War One, the extended area experienced a building boom, creating Salisbury's first suburb, or Newtown. Mrs. Huston left Poplar Hill to her daughter Elizabeth, who initiated its sale to George Waller in 1881. (Elizabeth and her sister Isabella are commemorated in the names of neighborhood streets.)
George Waller and his family lived in the house from 1882 to 1945. Various early photos and memorabilia belonging to the Wallers are located in the Mansion’s archives. Fred A. Adkins, who purchased the property in 1945, undertook restoration work with the hope of turning the house into the local Masonic Lodge. In 1948, when this failed to materialize, the house passed to Mr. and Mrs. Ward A. Garber, who maintained an 18th century period antique shop downstairs and lived upstairs. Mrs. Garber sold the property to Wicomico County in 1970 and since 1974 the City of Salisbury has owned it as a house museum in the public trust.