In this paper, an optimized solution method is proposed for the 3D stability analysis of rock slopes subject to toppling failure based on their geometric and mechanical properties. It was verified by a 3D block system that focused on the geometric properties of toppling slopes as a research object, considering the force and its action point on the interface of the block system as unknown variables, as well as introducing the definition of a safety factor considering both tension and shear strength reduction. The proposed method implied setting constraints, such as the balance equation corresponding to block force and moment, as well as non-violation of the yield criterion, considering the minimum value of the safety factor as the objective function. It was applied to the analysis of two typical 3D models simulating toppling failure on slopes. The example of a 3D spherical toppling slope was reconstructed and corroborated by calculations. The experimental results demonstrated that the proposed method could appropriately reflect the mechanical properties and stability behavior of a 3D toppling slope, thereby facilitating the analysis of the stability of 3D toppling rock slope model.