William Brown had come to London Town about ten years earlier. In 1753 he began renting a property that had been used as an ordinary for years. He lived there and operated a tavern, while also working as a carpenter and ferry operator. In all likelihood, his wife Susannah managed the ordinary, while he tended to their other businesses. During the 1760s, Brown constructed the most distinguished structure in town, a large brick building that remains to this day. The Brown family included five children: Jane, Elizabeth, William Jr., Margarette, and the youngest, Willie Anne.

The Browns were well known in the community, but never as wealthy as they may have seemed. In fact, we know that Brown borrowed heavily to keep up the appearance of wealth and to attract even wealthier people to his businesses.