Folly is a one story structure constructed in the Jeffersonian style, similarly to the buildings the former President was responsible for designing at the University of Virginia. The house also bears a resemblance to another nearby home, Edgemont, which has lead to speculation that Smith was an associate of Colonel James Cocke. Following the loss of the distinctive serpentine walls at the University of Virginia, Folly remains as the only example of them in Virginia.
After Smith's death in 1863, the house was passed down to his descendants. It has remained in the family to this day and remains in operation, though it no longer produces its former primary crop. It was added to the National Register of Historic Places in 1973.