Smith's borax operation grew throughout the 1870s and 1880s. He used a twenty-mule team to haul borax out of Death Valley to the nearest railroad, which gave rise to the Twenty Mule Team Borax trademark still in use today.
Smith and his wife Rebecca moved to Oakland in 1881. The couple began buying parcels of land with the intention of building an estate. While their grand new home was under construction, the couple lived in the old red house, a Victorian home that sat on a hill overlooking Lake Merritt. It wasn't actually an old house; it was likely built in the 1870s.
The new Smith home was under construction from 1893 to 1895. The new home, named Oak Hall, was a massive, three-story structure with 42 rooms, including 15 bedrooms, a ballroom, a bowling alley, and an attached conservatory. The house sat on a 35-acre estate called Arbor Villa, which was no less elaborate than the Smith's home. The grounds included a lily pond, rose arbors, a cactus garden, tennis courts, an observation tower, and a paddock for deer and rabbits. Maintenance of the vast estate required a small army of gardeners and groundskeepers. Remarkably, the mansion was demolished in 1931, following Smith's death. It was located across the street from the MacArthur and Park Blvd. location of the Oakland High School campus.
Though Smith's fortunes waxed and waned over the years, the old red house remains, although it is currently painted green. When Arbor Villa was under construction, the original Smith home was moved down the hill to make room for the new home. Following Smith's death, his second wife, Evelyn, moved the house a second time, to its current location on East 24th Street, where it was divided into apartments. The house remained in the Smith family until the 1970s.