Understanding pore growth is of great significance to investigating reservoir performance in shale-gas systems. However, different from the marine shale reservoir, the lacustrine shale reservoir is commonly rich in clay minerals, resulting in a complicated and poorly understood pore system. We have investigated the impact of coexisting clay mineral and organic matter on pore growth in the Lower Jurassic Da’anzhai Shale in the Northeast Sichuan Basin, West China, through performing total organic carbon (TOC) analysis, X-ray diffraction, field-emission scanning electron microscopy, focused ion beam-scanning electron microscopy (FIB-SEM), and adsorption experiment, and high-pressure mercury intrusion porosimetry. Our results indicate that the Da’anzhai Shale is dominated by clay-mineral-hosted pores, which are commonly filled or partly filled by pyrobitumen. Controlled by organic maceral, organic pores are poorly and heterogeneously developed in pyrobitumen, and minor or even no organic pores grow in vitrinite. Mesopore and macropore are popular in the Da’anzhai Shale reservoir with complex shapes, e.g., slit- or plate-like shapes combined with “ink-bottle” shapes, confirming a pore system dominated by clay-mineral-hosted pores. The weak positive correlation between the clay mineral content and the meso/macropore volume confirms that the clay mineral is a positive contributor to the storage space, and the weak negative correlation between the TOC and the mesopore volume suggests that infilling of pyrobitumen decreases the pore volume significantly. Similar correlations occur between specific surface area and clay mineral/TOC. FIB-SEM observation confirms that the pore system, e.g., the pore size, pore shape, and pore volume, is controlled by the coexisting clay mineral and pyrobitumen filling in a later stage. The calculated plane porosity of the initial inorganic pore and the unfilled inorganic pore in the Da’anzhai Shale is in the range of 3.66%–10.95% and 0.79%–1.46%, respectively, suggesting that 76.66% of inorganic pores is inactive due to pyrobitumen filling. All of this evidence suggests that pore growth in the Da’anzhai Shale is positively contributed by clay minerals, but it is negatively contributed by pyrobitumen filling. Further discussion suggests that pyrobitumen infilling between clay minerals in the Da’anzhai lacustrine shale can decrease the original pore volume significantly, which work together to govern the pore system in shale reservoirs.