Traditionally home of the Ramindjeri clan of the Ngarrindjeri people, the bay on which Victor Harbor sits was discovered by Matthew Flinders in HMS Investigator in April 1802. As part of the first circumnavigation of the continent, Flinders was surveying the then unknown southern Australian coast from the west. He encountered Nicolas Baudin in Le Geographe near the Murray Mouth several kilometres to the east of the present day location of Victor Harbor. Baudin was surveying the coast from the east for Napoleonic France. Although their countries were at war, each captain was given documents by the other nation’s Government, stating that the ships were on scientific missions, and were therefore not to be regarded as ships of war. Together, the ships returned to the bay and sheltered, while the captains compared notes. Flinders named the bay Encounter Bay after the meeting.
Around 1837 two whaling stations were established, one at Rosetta Head and the other near the point opposite Granite Island. Whale oil became South Australia’s first export. From 1839 the whaling station was managed for a time by Captain John Hart, a later Premier of South Australia. The town of Port Victor was laid out on the shores of Victor Harbor in 1863 when the horse-drawn tramway from Goolwa was extended to the harbour. The last whale was caught off Port Victor in 1872 and the town’s name changed in 1921.