We consider a hybrid quantum many-body system formed by a vibrational mode of a nanomembrane, which interacts optomechanically with light in a cavity, and an ultracold atom gas in the optical lattice of the out-coupled light. The adiabatic elimination of the light field yields an effective Hamiltonian which reveals a competition between the force localizing the atoms and the membrane displacement. At a critical atom-membrane interaction, we find a nonequilibrium quantum phase transition from a localized symmetric state of the atom cloud to a shifted symmetry-broken state, the energy of the lowest collective excitation vanishes, and a strong atom-membrane entanglement arises. The effect occurs when the atoms and the membrane are nonresonantly coupled.