Following a few years of his working on a variety of ships, Pyrrhus eventually secured a position aboard the Manhattan, an American whaling ship. His time with the crew of the Manhattan was notable, as it became the first vessel of American origin to travel to Tokyo at a time when few American vessels were allowed to travel and dock in this part of the world.
The Manhattan, a ship out of Pyrrhus's native New York State, set out on approximately a two year voyage in 1843. More than a year into this excursion, Pyrrhus, Captain Mercator Cooper and the rest of the ship's crew came across a total of 22 shipwrecked Japanese sailors in the waters of the Bonin Islands (Van Zandt, 1984). Despite no American vessel ever having previously been given permission to port in Japanese waters, The Manhattan sailed towards Tokyo (then Edo) with hopes of returning the sailors to their homeland.
On April 18, 1845, after sending some of the Japanese rescues with a message of intent to repatriate the remainder, the Manhattan was given permission to enter Tokyo Bay (Van Zandt, 1984). Upon examination of the vessel, as would have been customary for such an atypical arrival, Pyrrhus's appearance, as the only African American on board, was made of special note. Pyrrhus was the first man of such darkly colored skin that anyone in Japan had ever seen, and they were intimately intrigued. It is notable that the captain and other sailors have been lost to history while the presence of Pyrrhus was recorded in contemporary records. Towards the end of his life, the former slave was interviewed and shared his experience as one of the first Americans to be allowed to enter Tokyo by the Japanese shogunate.