Background: Neurofilament light chain (NfL) is a cytoskeletal protein component whose release into blood is indicative of neuronal damage. Tau is a microtubule-associated protein in neurons and strongly associated with overall brain degeneration. NfL and tau levels are associated with mortality in different neurological diseases, but studies in the general population are missing. We investigated whether NfL and tau serum levels could serve as prognostic markers for overall mortality in elderly individuals without pre-defined neurological conditions. Further, we investigated the cross-sectional associations between NfL, tau, neuropsychological functioning, and brain structures. Methods: In 1997, 385 inhabitants of Augsburg who were aged 65 years and older were included in the Memory and Morbidity in Augsburg Elderly (MEMO) study. They participated in a face-to-face medical interview including neuropsychological tests and magnetic resonance imaging (MRI) of the brain. NfL and tau were measured from non-fasting blood samples using highly sensitive single molecule array assays. To assess the prognostic accuracy of the biomarkers, concordance statistics based on the predicted 5-year survival probabilities were calculated for different Cox regression models. Associations between the biomarkers and the neuropsychological test scores or brain structures were investigated using linear or logistic regression. Results: NfL (HR 1.27, 95% CI [1.14–1.42]) and tau (1.20 [1.07–1.35]) serum levels were independently associated with all-cause mortality. NfL, but not tau, increased the prognostic accuracy when added to a model containing sociodemographic characteristics (concordance statistic 0.684 [0.612–0.755] vs. 0.663 [0.593–0.733]), but not when added to a model containing sociodemographic characteristics and brain atrophy or neuropsychological test scores. NfL serum levels were cross-sectionally associated with neuropsychological test scores and brain structures. Conclusions: The association between NfL serum levels and brain atrophy and neuropsychological performance in individuals without overt neurological disease is similar to that seen in patients with neurodegenerative diseases. These findings support the concept of a continuum of physiological aging and incipient, subclinical pathology, and manifest disease. NfL, but not tau, serum levels might serve as a prognostic marker for all-cause mortality if no other clinical information is available.