Motivated by the problem of expanding the single-trace tree-level amplitude of Einstein-Yang-Mills theory to the BCJ basis of Yang-Mills amplitudes, we present an alternative expansion formula in gauge invariant vector space. Starting from a generic vector space consisting of polynomials of momenta and polarization vectors, we define a new sub-space as a gauge invariant vector space by imposing constraints on the gauge invariant conditions. To characterize this sub-space, we compute its dimension and construct an explicit gauge invariant basis from it. We propose an expansion formula in this gauge invariant basis with expansion coefficients being linear combinations of the Yang-Mills amplitude, manifesting the gauge invariance of both the expansion basis and coefficients. With the help of quivers, we compute the expansion coefficients via differential operators and demonstrate the general expansion algorithm using several examples.