Brown settled near Keeney’s settlement with his wife Dorinda Webster, who he married in Connecticut in 1785. She was from the same family as Governor Webster of Connecticut. They raised seven children together, most of them growing up on the cabin in Cuyler. Brown purchased uncleared government land and used it to farm. Brown was a member of the Baptist church, and for many years was deacon of that church. He also promoted the building of the Baptist church house of worship at Keeney’s settlement.
When Brown arrived in the area, there was nothing but unpaved roads and trees everywhere. According to Mrs. F. P. Whitmarsh, “Early in the summer of 1795, these men with axes on their shoulders, followed the rough roads, ‘state roads’, if you please, as far as Manlius, from there they followed blazed trees or blazed their own to a place three miles south of Fabius.” When Brown settled in Fabius, he had to cut down the land for himself. Brown cleared the land and later planted potatoes. Brown went the furthest south out of the men that he traveled with, traveling one mile further away. This was a drastic change for Benjamin and especially his wife Dorinda. They moved from Connecticut where there was a formal town and where the area was already settled, to a place in which nothing was around them. Shortly after, Brown became the first assessor of Fabius. As an assessor, he was trusted by the community to be called upon for advice. Brown also became the second town clerk. Benjamin had a productive farm, and both he and his wife, Dorinda, stayed in Fabius until their deaths.