Despite being one of the world’s oldest deserts, and the subject of decades of research, evidence of past climate change in the Namib Desert is extremely limited. As such, there is significant debate regarding the nature and drivers of climate change in the low-latitude drylands of southwestern Africa. Here we present data from stratified accumulations of rock hyrax urine that provide the first continuous high-resolution terrestrial climate record for the Namib Desert spanning the past 50,000 yr. These data, spanning multiple sites, show remarkably coherent variability that is clearly linked to orbital cycles and the evolution and perturbation of global boundary conditions. Contrary to some previous predictions of southwestern African climate change, we show that orbital-scale cycles of hydroclimatic variability in the Namib Desert region are in phase with those of the northern tropics, with increased local summer insolation coinciding with periods of increased aridity. Supported by climate model simulations, our analyses link this to variations in position and intensity of atmospheric pressure cells modulated by hemispheric and land-sea temperature gradients. We conclude that hydroclimatic variability at orbital time scales is driven by the combined influence of direct low-latitude insolation forcing and the influence of remote controls on the South Atlantic anticyclone, with attendant impacts on upwelling and sea-surface temperature variations.