Recently, source data-free unsupervised domain adaptation (SFUDA) attracts increasing attention. Current work shows that the geometry of the target data is helpful to solving this challenging problem. However, these methods define the geometric structures in Euclidean space. The geometry cannot completely draw the semantic relationship between the target data distributed on a manifold. This article proposed a new SFUDA method, semantic consistency learning on manifold (SCLM), to address this problem. Firstly, we generated pseudo-labels for the target data using a new clustering method, EntMomClustering, that enhanced k-means clustering by fusing the entropy momentum. Secondly, we constructed semantic neighbor topology (SNT) to capture complete geometric information on the manifold. Specifically, in SNT, the global neighbor was detected by a developed collaborative representation-based manifold projection, while the local neighbors were obtained by similarity comparison. Thirdly, we performed a semantic consistency learning on SNT to drive a new kind of deep clustering where SNT was taken as the basic clustering unit. To ensure SNT move as entirety, in the developed objective, the entropy regulator was constructed based on a semantic mixture fused on SNT, while the self-supervised regulator encouraged similar classification on SNT. Experiments on three benchmark datasets show that our method achieves state-of-the-art results. The code is available on https://github.com/tntek/SCLM.