Illite-smectite (I-S) minerals isolated from Upper Jurassic oil-source rock shales from Denmark and the North Sea have been investigated by X-ray diffraction, thermal analysis, infrared, Mössbauer, and solid-state nuclear magnetic resonance spectroscopies and chemical analysis. Detailed structures have been determined in order to reveal the diagenetic transformation mechanism in these shales. Generally, in oil-source rocks of sedimentary basins, oil generation takes place simultaneously with the diagenetic transformation of I-S. We demonstrate a link between the two reactions: NH3 released from kerogen during maximum oil generation is fixed as NH4+ in the NH4-bearing mica or tobelite layers formed from smectite or vermiculite layers in I-S, in a diagenetic interval which we name the ‘tobelitization window’. Due to this solid-state transformation, mixed-layer structures have been formed consisting of interstratified illite, tobelite, smectite and vermiculite layers (I-T-S-V) and having maximum ordering of illite + tobelite and smectite layers for R = 1. The tobelitization of smectite in I-S is probably typical for all oil-source rock shales.