We developed a compressed implicit Jacobian scheme for the regularized Gauss-Newton inversion algorithm for reconstructing 3D conductivity distributions from electromagnetic data. In this algorithm, the Jacobian matrix, whose storage usually requires a large amount of memory, is decomposed in terms of electric fields excited by sources located and oriented identically to the physical sources and receivers. As a result, the memory usage for the Jacobian matrix reduces from O(NFNSNRNP) to O[NF(NS + NR)NP], where NF is the number of frequencies, NS is the number of sources, NR is the number of receivers, and NP is the number of conductivity cells to be inverted. When solving the Gauss-Newton linear system of equations using iterative solvers, the multiplication of the Jacobian matrix with a vector is converted to matrix-vector operations between the matrices of the electric fields and the vector. In order to mitigate the additional computational overhead of this scheme, these fields are further compressed using the adaptive cross approximation (ACA) method. The compressed implicit Jacobian scheme provides a good balance between memory usage and computational time and renders the Gauss-Newton algorithm more efficient. We demonstrated the benefits of this scheme using numerical examples including both synthetic and field data for both crosswell and controlled-source electromagnetic (CSEM) applications.