Structural phase transformations of MgSiO3 protoenstatite at high pressures are studied by atomic scale simulation techniques. Molecular-dynamics simulations and electronic-structure calculations reveal two metastable polymorphs with space groups P21cn and Pbcn, respectively. They are related to protoenstatite by displacive transition mechanisms via subsequent change of the silicate chain rotations from O- to S-type. Metadynamics simulations in combination with molecular dynamics reveal possible mechanisms for the martensitic transition from protoenstatite to high-pressure clinoenstatite. Two different shear mechanisms in the (100) plane are activated during the transition. The first consists of four partial displacements in (100) and (100), whereas in a second step only a single shear in (100) is observed.