Predatory interaction of marine mammals with longline fisheries is observed globally, leading to partial or complete loss of the catch and in some parts of the world to considerable financial loss. Depredation can also create additional unrecorded fishing mortality of a stock and has the potential to introduce bias to stock assessments. Here we aim to characterise depredation in the Patagonian toothfish (Dissostichus eleginoides) fishery around South Georgia focusing on the spatio-temporal component of these interactions. Antarctic fur seals (Arctocephalus gazella), sperm whales (Physeter macrocephalus), and orcas (Orcinus orca) frequently feed on fish hooked on longlines around South Georgia. A third of longlines encounter sperm whales, but loss of catch due to sperm whales is insignificant when compared to that due to orcas, which interact with only 5% of longlines but can take more than half of the catch in some cases. Orca depredation around South Georgia is spatially limited and focused in areas of putative migration routes, and the impact is compounded as a result of the fishery also concentrating in those areas at those times. Understanding the seasonal behaviour of orcas and the spatial and temporal distribution of “depredation hot spots” can reduce marine mammal interactions, will improve assessment and management of the stock and contribute to increased operational efficiency of the fishery. Such information is valuable in the effort to resolve the human-mammal conflict for resources.