Purpose: To longitudinally assess the impact of high-risk structural biomarkers for natural disease progression in non-exudative age-related macular degeneration (AMD) on spatially resolved mesopic and scotopic fundus-controlled perimetry testing. Methods: Multimodal retinal imaging data and fundus-controlled perimetry stimuli points were semiautomatically registered according to landmark correspondences at each annual visit over a period of up to 4 years. The presence of sub-RPE drusen, subretinal drusenoid deposits, pigment epithelium detachments (PEDs), hyper-reflective foci (HRF), vitelliform lesions, refractile deposits, and incomplete RPE and outer retinal atrophy (iRORA) and complete RPE and outer retinal atrophy (cRORA) were graded at each stimulus position and visit. Localized retinal layer thicknesses were extracted. Mixed-effect models were used for structure-function correlation. Results: Fifty-four eyes of 49 patients with non-exudative AMD (mean age, 70.7 ± 9.1 years) and 27 eyes of 27 healthy controls (mean age, 63.4 ± 8.9 years) were included. During study course, presence of PED had the highest functional impact with a mean estimated loss of -1.30 dB (P < 0.001) for mesopic and -1.23 dB (P < 0.001) for scotopic testing, followed by HRF with -0.89 dB (mesopic, P = 0.001) and -0.87 dB (scotopic, P = 0.005). Subretinal drusenoid deposits were associated with a stronger visual impairment (mesopic, -0.38 dB; P = 0.128; scotopic, -0.37 dB; P = 0.172) compared with sub-RPE drusen (-0.22 dB, P = 0.0004; -0.18 dB, P = 0.006). With development of c-RORA, scotopic retinal sensitivity further significantly decreased (-2.15 dB; P = 0.02). Thickening of the RPE-drusen-complex and thinning of the outer nuclear layer negatively impacted spatially resolved retinal sensitivity. Conclusions: The presence of PED and HRF had the greatest prognostic impact on progressive point-wise sensitivity losses. Higher predominant rod than cone-mediated localized retinal sensitivity losses with early signs of retinal atrophy development indicate photoreceptor preservation as a potential therapeutic target for future interventional AMD trials.