We present new experimental measurements of resonance strengths in the astrophysical $^{23}$Al($p,\gamma$)$^{24}$Si reaction, constraining the pathway of nucleosynthesis beyond $^{22}$Mg in X-ray burster scenarios. Specifically, we have performed the first measurement of the ($d,p$) reaction using a radioactive beam of $^{23}$Ne to explore levels in $^{24}$Ne, the mirror analog of $^{24}$Si. Four strong single-particle states were observed and corresponding neutron spectroscopic factors were extracted with a precision of ∼20%. Using these spectroscopic factors, together with mirror state identifications, we have reduced uncertainties in the strength of the key ℓ = 0 resonance at $E_r$ = 157 keV, in the astrophysical $^{23}$Al($p,\gamma$) reaction, by a factor of 4. Our results show that the $^{22}$Mg($p,\gamma$)$^{23}$Al($p,\gamma$) pathway dominates over the competing $^{22}$Mg($\alpha ,p$) reaction in all but the most energetic X-ray burster events ($T>0.85$ GK), significantly affecting energy production and the preservation of hydrogen fuel.