We point out that the odd-in-velocity contribution to the $\mathcal{O}(G^4)$ radiated angular momentum for two-body scattering is determined by the radiation-reaction (RR) term in the one-loop waveform. This RR term is actually proportional to the tree-level waveform, and this reduces the calculation of the odd-in-velocity contribution to the $\mathcal{O}(G^4)$ angular momentum loss, $J_{2\mathrm{rad}}$, to two loops, instead of three loops as one would expect by power counting. We exploit this simplification, which follows from unitarity, to obtain a closed-form expression for $J_{2\mathrm{rad}}$ for generic velocities, which resums all fractional post-Newtonian (PN) corrections to the $\mathcal{O}(G^4)$ angular momentum loss starting at 1.5PN.