Daniel Fisher’s Very Bad Time in Williamsburg

As the capital of British North America’s largest colony, eighteenth-century Williamsburg drew in many people seeking opportunity. It didn’t work out for all of them. One of those who struggled was a merchant and almost-tavernkeeper named Daniel Fisher.

He left behind an extraordinary, and tantalizingly incomplete, diary that describes his perspective on his family’s troubled five-year stay in Williamsburg from 1750 to 1755. Its vivid tales of arson, conspiracy, and perjury speak to the often-undocumented ways that business, slavery, and marriage worked in eighteenth-century Virginia.

The World's Treachery

It is not entirely clear why Fisher left England. His diary, which he addressed to an unnamed friend, refers only vaguely to the decision. He explained that two friends had schemed to have him “stript of my employ,” though he also acknowledged that “whatever Ills have happened are mainly the result of my own Vain conduct.” The whole affair, he wrote, had convinced him “of the World’s Treachery.” He convinced his wife to move their family to Virginia in 1750.¹

As a merchant, Fisher needed introductions with the colony’s leading men if he hoped to succeed in Virginia. In the eighteenth century, it was common for gentlemen to enter a new place with a letter of introduction from someone prominent, vouching for the new arrival’s trustworthiness. His brief time in Virginia in the 1720s had acquainted him with some prominent men, including the merchant Thomas Nelson. Fisher went to some effort to get a “worthy recommendation” that would introduce him to Thomas’ sons William and Thomas.²

Upon his arrival, he visited Thomas Nelson and delivered his recommendation letter. But it turned out that his acquaintance with the Nelson family was not as helpful as he hoped. The Nelsons were close with a family that Fisher had once affronted. Almost thirty years before, Lawrence Smith had asked Fisher for a favor, and when he refused the disagreement escalated into a physical altercation. Fisher was the sort of person who was always getting into these kinds of scrapes. Perhaps still nursing this old grudge, Thomas’s elder brother William Nelson advised Fisher to return to London “or try my Fortune in some other Colony.”³

At this point in his diary entry, he recalled a pertinent piece of advice from the wealthy Virginian John Randolph, who he happened to travel across the ocean with. Randolph had told Fisher to avoid offending “any person of note” in Virginia, for “either by blood of marriage, we are almost all related.” Any enemy of one member of the gentry became an enemy of them all, until “his ruin is accomplished.”⁴

How His Ruin Was Accomplished

Shunned by the Nelsons, Fisher took up his second recommendation letter, addressed to Nathaniel Walthoe in Williamsburg. Walthoe was a well-connected man, serving in the important administrative role as clerk[MJ1] to the Governor’s Council. Walthoe introduced Fisher around town and helped his family find their footing in Williamsburg. Soon, Fisher agreed to rent a large building near the Capitol from local tavernkeeper Henry Wetherburn, which was the former site of Shields’ Tavern.⁵ Fisher intended to operate a merchant business, but was not successful. In April 1751, he indicated that his merchant business was not supporting his family, and sought “any Gentleman” to employ him, whether in Williamsburg or elsewhere. A few months later, apparently unable to find new employment, he announced that he was opening a tavern.⁶

According to Fisher’s account, the town’s elites soon began to squeeze him out. A member of the Lee family decided he wanted the building Fisher had rented from Wetherburn. Fisher had relied on a verbal bargain for his lease, which allowed Wetherburn to repeatedly change the terms. Despite earlier promises, he sold the building’s large billiard table and refused to make necessary repairs. When Fisher sought out the witnesses of their earlier agreement, he found them unwilling to recall these aspects of the agreement. He believed Wetherburn had swayed them. Early in 1752, citing “Difficulties and Impediments,” Fisher ended his business and turned most of the building into apartments and stables for rent.⁷

At this point, Fisher’s diary takes on a conspiratorial tone. The city’s merchants, he wrote, turned against him. In March 1754, Williamsburg Mayor John Holt, whom Fisher identified as a “friend and a known dependent” of William Nelson, accused him in court of “selling Rum to Negroes contrary to Law.” In Virginia, it was illegal to sell alcohol to enslaved people without their enslaver’s permission. A tavern could earn a reputation as a “disorderly House” if it served enslaved people without permission.⁸ Fisher protested in court that his family had been so scrupulous in securing permission before serving alcohol to enslaved people that they were “frequently ridiculed” for it. Indeed, he claimed that one day he “turned away” two enslaved people, who were then immediately served at Holt’s establishment. Fisher recorded the reaction to this accusation with evident satisfaction: “This put the Court in a flame.” When merchant John Greenhow applied for a license, Fisher accused him of also selling liquor to enslaved people without permission. It appears that Holt’s accusations were dropped.⁹ But according to Fisher, Holt did not forget.

Holt's Revenge

Fire was a constant threat in the eighteenth century. The first Williamsburg Capitol building had burned down in 1747, and its replacement had only been completed in 1753. A month after Holt and Fisher had accused each other of illegally selling liquor, a home near Fisher’s caught fire. When the flames reached a supply of gunpowder, it exploded and spread the fire to Walthoe’s home. To protect the nearby Capitol and the rest of the town, Mayor John Holt took command of the scene and ordered a crowd of Black spectators to “pull down” Fisher’s home “to the ground.” In the commotion, the crowd carried off most of his family’s furnishings and goods. When some of the burglars were later discovered, Williamsburg’s magistrates refused to prosecute them.¹⁰

Fisher wrote that he was “beginning now to perceive or experimentally feel the Truth of what Mr. Randolph had affirmed in our Passage,” that Virginia’s leading families would push out unwanted outsiders. He wrote to William Nelson, blaming him for “favoring the injurious assaults I had met with.” He told Nelson that Holt, Wetherburn, and many others would have never “dared to have wronged” him if Nelson had forbidden it.¹¹ Hoping to get out of Williamsburg and away from its unfriendly elite, he asked Nelson for help relocating to Philadelphia.

New Beginnings

This was all too much for Fisher’s wife. She had never wanted to move to Virginia. After he started antagonizing the Nelsons, one of Virginia’s most powerful families, she pled with him to stop. He ignored her. Their dispute came to the point “that she separated bed and board from me.” After Nelson and Walthoe decided to support Fisher’s move to Philadelphia, he left Williamsburg “without taking any leave of her, or in Truth, of any one of my Family.” He left town at four thirty in the morning on May 12, 1755, without waking or alerting anyone. He arrived in Philadelphia ten days later, and managed to gain the assistance of Benjamin Franklin, who offered him a job as a clerk.¹²

Before he took up the job with Franklin, though, Fisher received a letter from his wife indicating that she was willing to reconcile. He decided to return to Williamsburg to settle his affairs with his wife. His surviving diary ended while he was on the road to Williamsburg. It’s uncertain exactly how Fisher’s story ends. But we know he reached Williamsburg. He placed an advertisement in the Virginia Gazette in September 1755, which indicates that Fisher was robbed on his way back into town. Little went well for Daniel Fisher in Williamsburg.

The final documented example of Fisher’s misadventures appears in the records of the Virginia House of Burgesses, the colony’s representative assembly. In April 1756, Williamsburg resident and Burgess Benjamin Waller, who had once given Fisher legal advice and whom he described elsewhere as an “intimate friend” of the Nelson family, rose to share a letter from Daniel Fisher. According to Waller, Fisher had sent a letter containing an attack on the House of Burgesses along with “a most scandalous and defamatory Libel” against Waller. The actual letter was not entered into the record, and so we only have Waller’s description of it. But on the basis of this letter, the House immediately found Fisher “guilty of a Breach of the Privileges of this House” and directed that their Sergeant at Arms bring Fisher “into Custody . . . to answer for his said Offence.”¹³ His habit of writing impulsive, angry letters had finally led to his arrest.

Other Virginia Gazette advertisements document the presence of a person named Daniel Fisher in Virginia during the 1760s and 1770s.¹⁴ Given that his was a common name, it is impossible to say if these refer to the Daniel Fisher documented in this diary. But it is entirely possible that Fisher made a better life for himself and his family elsewhere in the colony. Williamsburg wasn’t for everyone.

Truth in Diaries

Daniel Fisher blamed his difficulties in Williamsburg on an elite class that worked together to force him out of town. Perhaps there was a conspiracy, as Fisher alleges. Perhaps, though, there was a simpler explanation. Fisher was clearly an abrasive individual. He seems to have alienated almost everyone he encountered in Virginia. It may be that the truth of Fisher’s story lies not only in his point of view, recorded in his diary, but in the unrecorded, forgotten reflections of men like William Nelson, John Holt, and Henry Wetherburn, the enslaved people of Williamsburg, and his own family. For while diaries can offer an extraordinarily rich window into the experience of one individual, they seldom tell the full and complete story.