Relativistic forward scattering of electrons at finite temperature involves the incoherent superposition of diffraction patterns formed by different snapshots of thermal atomic displacements. In experiments, thermal vibrations lead to thermal diffuse scattering (TDS), partly dominating diffraction patterns of thick specimens. This study sheds light on the effects of TDS on solutions to the inverse scattering problem using combined realand diffraction-space information acquired in a scanning transmission electron microscope (STEM) to retrieve the object's phase. Using frozen phonon multislice within the Einstein approximation, realistic ground truth data of 20-nm-thick SrTiO3 is generated and subjected to contemporary inverse multislice schemes to retrieve the projected Coulomb potential slicewise. We first classify phase retrieval algorithms as to their assumptions on periodicity along the incident beam direction, as well as pixelwise and parametrized reconstruction methods. It is found that pixelwise object reconstructions are capable of retrieving structural details qualitatively while being prone to contain TDS-related artifacts which can result in unphysical potentials. For pixelwise reconstructions of multiple independent specimen slices, we observe that the origin of TDS, i.e., thermal atomic displacements, starts to emerge naturally. However, the quantitative assessment tends to too small mean squared thermal displacements, also when reconstructing multiple object modes. Using an atomistically parametrized inversion strategy which exploits the explicit separation of thermal vibrations and potentials, temperature and chemistry of the specimen can be retrieved quantitatively with high accuracy.