Even though the speed of sound (SoS) is non-homogeneous in biological tissue, most reconstruction algorithms for optoacoustic imaging neglect its variation. In addition, when heavy water is used as coupling medium to enable imaging of certain biological chromophores such as lipids and proteins, the SoS also differs significantly between couplant and tissue. While the assumption of uniform SoS is known to introduce visible deformations of features in single-wavelength optoacoustic images, the spectral error introduced by the assumption of uniform SoS is not fully understood. In this work, we provide an in-depth spectral analysis of multi-spectral optoacoustic imaging artifacts that result from the assumption of uniform SoS in situations where SoS changes substantially. We propose a dual-SoS model to incorporate the SoS variation between the couplant and the sample. Tissue-mimicking phantom experiments and in vivo measurements show that uniform SoS reconstruction causes spectral smearing, which dual-SoS modeling can largely eliminate. Due to this increased spectral accuracy, the method has the potential to improve clinical studies that rely on quantitative optoacoustic imaging of biomolecules like hemoglobin or lipids.