The mansion was two stories, with five bays, fronted with a one-story columned portico. Two side wings flanked the central part of the house. At the rear of the structure was a large piazza, possibly inspired by the piazza at Mount Vernon.
After William's death in 1809, his son, another William (1792-1830) inherited part of the property, including the Ravensworth house. He married Anna Maria Sarah Goldsborough (1896-1794). They had no children, but adopted a Goldsborough niece. William sold some of the land in 1820, bequeathed another parcel to his adopted daughter, and his will left the rest of the Ravenstowrth estate to his widow, then to his niece Mary Anna Randolph Custis Lee.
The daughter of the Ravensworth-builder William and his wife Anna Bolling Randolph Fitzhugh (1747-1805), Mary Lee, married George Washington Parke Custis, the grandson of Martha Washington. They lived at Arlington House, and their daughter Mary Anna Randolph Custis Lee married Robert E. Lee. The mother of Robert E. Lee, Anne Hill Carter Lee, went to the estate in her final illness and was buried there after her death.
Enslaved persons operated the plantation. The last William Fitzhugh (son of the house's builder) supported the eventual end of slavery and colonization of Liberia by freed slaves, but he owned 208 people at the time of his death across several estates. In his will he manumitted his enslaved workers, though there were still a few enslaved persons still at the estate with his widow Anna Maria during the Civil War.
In 1861, when Robert E. Lee left Arlington House to take up command of Confederate troops, his wife Mary Anna fled to Ravensworth and her aunt, Anna Maria Fitzhugh, widow of the last William Fitzhugh. Mrs. Lee only stayed at Ravensworth for about a month during the war, but Fitzhugh remained there with several enslaved persons. Both Union and Confederate units made use of the estate during the war, cutting wood or camping at the site. Anna Maria Fitzhugh died in 1874; Robert E. Lee and Mary Anna Randolph Custis Lee's sons George Washington Custis Lee and Major General W. H. F. Rooney Lee inherited and lived at Ravensworth. Three years later, the U. S. government rejected a reimbursement claim for hay seized during the war.
After Rooney Lee's death in 1891, Ravensworth house passed to his sons Robert E. Lee III and Dr. George Bolling Lee. The latter eventually obtained sole ownership of the house. Ravensworth house burned in 1926 under mysterious circumstances, possibly arson, which also destroyed many portraits of the Lee and Custis families. The estate is nevertheless present in the Historic American Buildings Survey in a 1937 entry. It is possible that the survey used older photographs of the building. The survey also noted the unique brick stable built in the classic Jeffersonian style, which was constructed ca. 1805.
George Bolling Lee's widow Helen sold the estate to developers in 1957. The stable was razed in 1960. Historical markers near the former site of the estate tell the story of Ravensworth. Within the town of Springfield, the Census-designated area where the estate once stood bears its name.