Extracting horizons from a seismic image has been playing an important role in seismic interpretation. However, how to fully use global-level information contained in the seismic images such as the order of horizon sequences is not well-studied in existing works. To address this issue, we have developed a novel method based on a directed and colored graph, which encodes effective context information for horizon extraction. Following the commonly used framework, which generates horizon patches and then groups them into horizons, we first build a directed and colored graph by representing horizon patches as vertices. In addition, edges in the graph encode the relative spatial positions of horizon patches. This graph explicitly captures the geologic context, which guides the grouping of the horizon patches. Then, we conduct premerging to group horizon patches through matching some predefined subgraph patterns that are designed to capture some special spatial distributions of horizon patches. Finally, we have developed an ordered clustering method to group the rest of the horizon patches into horizons based on the pairwise similarities of horizon patches while preserving geologic reasonability. Experiments on real seismic data indicate that our method can outperform the autotracking approach solely based on the similarity of local waveforms and can correctly pick the horizons even across the fault without any crossing, which demonstrates the effectiveness of exploring the spatial information, i.e., the order of horizon sequences and special spatial distribution of horizon patches.