Femtosecond X-ray irradiation of solids excites energetic photoelectrons that thermalize on a timescale of a few hundred femtoseconds. The thermalized electrons exchange energy with the lattice and heat it up. Experiments with X-ray free-electron lasers have unveiled so far the details of the electronic thermalization. In this work we show that the data on transient optical reflectivity measured in GaAs irradiated with femtosecond X-ray pulses can be used to follow electron-lattice relaxation up to a few tens of picoseconds. With a dedicated theoretical framework, we explain the so far unexplained reflectivity overshooting as a result of band-gap shrinking. We also obtain predictions for a timescale of electron-lattice thermalization, initiated by conduction band electrons in the temperature regime of a few eVs. The conduction and valence band carriers were then strongly non-isothermal. The presented scheme is of general applicability and can stimulate further studies of relaxation within X-ray excited narrow band-gap semiconductors.