Most natural and synthetic rubbers have inherently high flammability, a property which limits their uses. The aim of the present work was to study the effect of organo-montmorillonite (OMMT) and modified OMMT on the flame-retardance and mechanical properties of natural rubber (NR) composites. The OMMT was modified with hyper-branched polymer via condensation polymerization between the intercalation agent, N,N-di(2-hydroxyethyl)-N-dodecyl-N-methylammonium chloride, and the monomer, N,N-dihydroxyl-3-aminomethyl propionate. This modified OMMT was then reacted with phosphate, and a novel flame-retardant hyper-branched organic montmorillonite (FR-HOMMT) was thus obtained. The surface morphology, interlayer space, interlamellar structure, and thermal properties of these modified clays were investigated by Fourier-transform infrared spectroscopy, scanning electron microscopy, X-ray diffraction, and thermogravimetric analysis. The FR-HOMMT showed increased basal spacing and better thermal stabilities due to the different arrangement and thermal stability of the novel organic macromolecular surfactant. Natural rubber NR/OMMT and NR/FR-HOMMT composites were prepared by conventional compounding with OMMT and the phosphorus-based organo-montmorillonite. The cure characteristics, tensile strength, wear resistance, thermal stabilities, and flame-retardant properties were researched and compared. The best dispersion of this modified clay was observed for 20 phr (parts per hundred of rubber) of FR-HOMMT-filled composite, which resulted in the best mechanical performance with an increase of 47% in tensile strength, of 40% in elongation at break, and decrease of 140% in abrasion loss compared with 20 phr of the OMMT-filled matrix. A mechanism for reinforcing and flame retardance is proposed here. The ‘anchor’ effect caused by the hyper-branched polymer may decrease the number and size of the voids in the NR matrix, and thus increase the crack path during tensile drawing. Meanwhile, the flame retardance of the OMMT and the phosphate may increase the number of carbonaceous layers, thus inhibiting the degree of pyrolysis of the NR matrix during burning.