A precise link is derived between scalar-graviton S-matrix elements and expectation values of operators in a worldline quantum field theory (WQFT), both used to describe classical scattering of black holes. The link is formally provided by a worldline path integral representation of the graviton-dressed scalar propagator, which may be inserted into a traditional definition of the S-matrix in terms of time-ordered correlators. To calculate expectation values in the WQFT a new set of Feynman rules is introduced which treats the gravitational field h μν (x) and position $x_i^{\mu }\left({\tau }_i\right)$ $$ {x}_i^{\mu}\left({\tau}_i\right) $$ of each black hole on equal footing. Using these both the 3PM three-body gravitational radiation 〈h μv (k)〉 and 2PM two-body deflection $\Delta p_i^{\mu }$ $$ \Delta {p}_i^{\mu } $$ from classical black hole scattering events are obtained. The latter can also be obtained from the eikonal phase of a 2 → 2 scalar S-matrix, which we show corresponds to the free energy of the WQFT.