Using effective field theory methods, we calculate for the first time the complete fourth-order term in the Fermi-momentum or $k_F{a}_s$ expansion for the ground-state energy of a dilute Fermi gas. The convergence behavior of the expansion is examined for the case of spin one-half fermions and compared against quantum Monte-Carlo results, showing that the Fermi-momentum expansion is well-converged at this order for $\left|k_F{a}_s\right|\lesssim 0.5$.