As an advanced control method that could bring extra inertia and damping characteristics to inverter-based distributed generators, the virtual synchronous generator (VSG) has recently drawn considerable attention. VSGs are expected to enhance the frequency regulation capability of the local power grid, especially the AC microgrid in island mode. However, the cost of that performance promotion is potential instability. In this paper, the unstable phenomena of the islanded microgrid dominated by SGs and distributed generators (DSs) are addressed after mathematical modeling and detailed eigenvalue analyses respectively. The influence of VSG key parameters, e.g., virtual inertia, damping factor, and droop coefficient on system stability is investigated, and the corresponding mathematical calculation method of unstable region is obtained. The theoretical analysis is well supported by time domain simulation results. The predicted frequency oscillation suggests the consideration of stability constrain during the VSG parameters design procedure.