Inclusive hadronic decays of the $\tau$ lepton are very interesting from the phenomenological point of view since they give access to the Cabibbo-Kobayashi-Maskawa matrix elements $V_{ud}$ and $V_{us}$. In this paper, for the first time, by employing the method of [M. Hansen, A. Lupo, and N. Tantalo, Phys. Rev. D 99, 094508 (2019)] for hadronic smeared spectral densities, we compute on the lattice the inclusive decay rate of the processes $\tau \to X_{ud}{\nu }_{\tau }$, where $X_{ud}$ is a generic hadronic state with $\overline{u}d$ flavor quantum numbers. Our computation, which avoids any recourse to operator product expansion and/or perturbative approximations, is carried out in isospin symmetric $N_f=2+1+1$ lattice QCD at physical quark masses, using ensembles produced by the Extended Twisted Mass Collaboration at three lattice spacings and two volumes. All uncertainties, except for isospin breaking effects, are taken into account and a result with a subpercent error is obtained for $\left|V_{ud}\right|$, which is nicely consistent with the current world average. These findings validate our approach and also motivate the inclusion of isospin breaking corrections and its extension to the inclusive decay $\tau \to X_{us}{\nu }_{\tau }$, paving the way toward a high-precision first-principles determination of $\left|V_{us}\right|$ and $\left|V_{ud}\right|$ from inclusive $\tau$ decay.