Shear-wave (S-wave) velocity is considered an essential parameter for the study of the earth, and Rayleigh wave inversion has been widely accepted and used to determine it. Given high-quality measured dispersion curves, the inversion performance depends on the applied optimization algorithm inside the inversion process. We propose a novel inversion framework to promote efficient and accurate inversion, i.e., a two-stage broad learning inversion framework (TS-BL). The proposed TS-BL not only inherits the powerful mapping capability and simple configured structure of broad learning (BL) network but also makes two significant improvements to better acclimatize itself to Rayleigh wave inversion. First, TS-BL adopts a two-stage inversion strategy to perform optimizing two times. It does not yield the same search space in the two inversion stages. In the first stage, because the inversion aims to find an approximation rather than the accurate value of model parameters, the difficulty in constructing the mapping model is reduced by sacrificing accuracy. Then, an effective BL network can be established using smaller sample sizes. In the second stage, the search space becomes much narrower, commencing with the approximation results obtained in the prior stage. This helps the final BL network to easily and quickly model the actual relationship between measured dispersion curves and unknown model parameters. After that, the forward modeling of measurements rather than the validation data set is exploited for tuning the network’s hyperparameters. The physical model is superior to the validation data set for selecting a suitable network complexity to adapt to the measured dispersion curves because the latter only describes an overall relationship. As a result, accurate S-wave velocities can be efficiently acquired by using the proposed TS-BL with a low cost of training samples. The efficiency and reliability of TS-BL have been demonstrated in numerical and field data examples.