A stochastic method for investigating site effects by means of borehole array observation is proposed. This method is based on a stochastic time-series model that is derived from Haskell's matrix procedure for a layered medium and is applicable to transverse-component recordings simultaneously obtained on the ground surface and at a depth. The time-series model not only accounts for the amplitude but also for the phase characteristics of the recordings. By applying this method to actual downhole array recordings, the following aspects of site effects can be investigated: (1) An estimation of the wave transfer function for SH waves, (2) the separation of upcoming and downgoing waves on a borehole seismogram at a depth, that is, an estimation of purely incident waves, and (3) the enhancement of fundamental mode Love waves on a surface seismogram are examined by applying this method to two borehole array recordings obtained at the Fuchu observatory in Japan. A noteworthy point is that these aspects of site effects are examined using this method without any knowledge of the velocity structure beneath the site.