Human activity has changed land surfaces by converting native ecosystems to urban land uses, which has caused increases in soil temperature. In order to understand the effect of different ground covers on soil temperature, observation stations were built under concrete, bare soil, and grass ground covers located in urban and rural areas of Nanjing City, China. By analyzing continuously recorded soil temperature data from June 2009 to June 2010, it was found that the average daily soil temperature in both urban and rural areas shows a tendency of concrete > bare soil > grass ground cover. Soil temperature under concrete ground cover in summer is significantly higher than that under bare soil and grass, consistent with the urban heat island effect. The cover thermal-physical properties and soil structures lead to differences in soil temperature between urban and rural areas. As compared to bare or grass-covered soils, concrete-covered soils exhibit lower values of evaporation rate, reflectivity, and specific heat capacity and higher values of thermal conductivity. As such, the initial heating rate of the soil under concrete ground cover is faster, and the final temperature is higher. These findings provide a basis with which to reduce the heat island effect through the provision of increasing grass-covered open space.