The pollution risk of potentially toxic elements (PTEs) in soil is a common concern in the fields of environmental science and agriculture. The spatial distribution characteristics, pollution risk assessment and genetic sources of PTEs in soil in high geological background areas are crucial to the safe use of land. Based on the data for five PTEs in 921 surface soil samples from typical high geological background areas in SW China, we discuss their spatial distribution characteristics using geostatistics, the semivariance function and its model. We carry out pollution risk assessment using the influence index of comprehensive quality (IICQ) and analyse their genetic sources using the UNMIX receptor model. The spatial distribution of element content has a high spatial coupling with the stratigraphic spread, which is affected by external random factors, resulting in a weakened autocorrelation. The IICQ indicates that soil is dominated by mild and moderate pollution, accounting for 42.24 and 44.95%, respectively, with cadmium contributing up to 63.00% to the IICQ. Chromium and cadmium are controlled by natural sources, accounting for 83.98 and 69.39%, respectively. Arsenic and mercury have anthropogenic sources, accounting for 68.7 and 44.0%, respectively. Lead is controlled by both lead–zinc mining and geological background, accounting for 47.7 and 36.4%, respectively. The UNMIX model demonstrates that the spatial variability of PTEs is influenced by anthropogenic factors, such as coal combustion emissions, which account for 30.1% of the pollution sources. Soil parent material sources contribute 49.4%, followed by lead–zinc mining and smelting activities at 20.5%.