We investigate an algorithm for 3D time-domain airborne electromagnetic (AEM) inversion based on the finite-volume (FV) method and direct Gauss-Newton optimization, where we obtain high efficiency by constraining the modeling volume to the AEM volume of influence (VOI) of a 3D source within the earth, rather than using the larger VOI of the AEM system. A half-space or layered earth is used to model the background field in the time domain, taking into account the transmitter waveform through convolution. Assuming that the 3D source of any secondary field detected at a survey point lies within the moving VOI of the airborne system, we conduct time-domain forward modeling and Jacobian calculation using an FV method within the 3D source VOI that requires a small number of cells for discretization. A local mesh and direct solver are shown to further speed up the computation. A synthetic isolated synclinal conductor inversion shows good agreement with the model geometry and provides a good fit to the data contaminated with noise. A synthetic multiple-body model inversion was also quite successful, showing that our algorithm is effective and about four times faster than inversion using the total-field method. Finally, we inverted GEOTEM data over the Lisheen deposit, where our inversion result was consistent with the published geology.