High‐frequency Rayleigh waves can be extracted from ambient seismic noises through noise correlation functions (NCFs), which provides a useful tool to image shallow structures in topographic regions, for example, landslides. Topography may affect signal‐to‐noise ratios (SNRs) of extracted Rayleigh waves. It is necessary to investigate the propagation features of Rayleigh waves passing a 3D topography. Based on the incident and scattered waves satisfying the free surface boundary conditions, we first derive the displacement responses of Rayleigh waves across a 3D elastic wedge. The results show that the particle motions of Rayleigh waves are an ellipse whose longer axis is always perpendicular to the topographic free surface. Therefore, the component, perpendicular to the topographic free surface, is a better choice to extract high‐frequency Rayleigh waves than the conventional vertical component. To verify the choice, we carry out numerical simulations to extract high‐frequency NCFs for a typical 3D massif model. Finally, we apply this approach to extract high‐frequency Rayleigh‐wave NCFs on the Xishancun landslide in southwestern China. The NCFs obtained using the component have more coherent waveforms and higher SNRs than those using the vertical component. We conclude that the component has advantages in extracting high‐frequency Rayleigh waves over the conventional vertical component.