We propose a new algorithmic approach to the non-smooth and non-convex Potts problem (also called piecewise-constant Mumford-Shah problem) for inverse imaging problems. We derive a suitable splitting into specific subproblems that can all be solved efficiently. Our method does not require a priori knowledge on the gray levels nor on the number of segments of the reconstruction. Further, it avoids anisotropic artifacts such as geometric staircasing. We demonstrate the suitability of our method for joint image reconstruction and segmentation from limited data in x-ray and photoacoustic tomography. For instance, our method is able to reconstruct the Shepp-Logan phantom from $7$ angular views only. We demonstrate the practical applicability in an experiment with real PET data.