We explore the possibility that the initial transverse momentum distribution of unpolarized and polarized partons within unpolarized nucleons, both with and without the anisotropy of unpolarized hadrons produced in the fragmentation of outgoing partons, could lead to the observed azimuthal anisotropy of high transverse momentum (high-$p_T$) hadrons produced in high energy proton-proton (p-p) or proton-ion (p-A) collisions. Including simple Gaussian forms for transverse momentum dependent parton distribution functions (PDF) and fragmentation functions, and assuming an $A^{1/3}$ enhancement of a PDF in p-Pb collisions, we show that the observed anisotropy, with no modification to the angle integrated spectra ($R_{pA}=1$) for 5 GeV $\lesssim p_T\lesssim 50$ GeV, can be straightforwardly understood as arising from a few processes dominated by gluon-gluon to gluon-gluon scattering.