Iron-related disorders of the liver can result in serious health conditions, such as liver cirrhosis. Evidence on the role of modifiable lifestyle factors like nutrition in liver iron storage is lacking. Thus, we aimed to assess the association of habitual diet with liver iron content (LIC). We investigated 303 participants from the population-based KORA-MRI study who underwent wholebody magnetic resonance imaging (MRI). Dietary habits were evaluated using repeated 24 h food lists and a food frequency questionnaire. Sex-stratified multiple linear regression models were applied to quantify the association between nutrition variables of interest and LIC, adjusting for liver fat content (LFC), energy intake, and age. Mean age of participants was 56.4 ± 9.0 years and 44.2% were female. Mean LIC was 1.23 ± 0.12 mg/g dry weight, with higher values in men than in women (1.26 ± 0.13 and 1.20 ± 0.10 mg/g, p < 0.001). Alcohol intake was positively associated with LIC (men: β = 1.94; women: β = 4.98, p-values <0.03). Significant negative associations with LIC were found for fiber (β = −5.61, p < 0.001) and potassium (β = −0.058, p = 0.034) for female participants only. Furthermore, LIC was highly correlated with liver fat content in both sexes. Our findings suggests that there are sex-specific associations of habitual dietary intake and LIC. Alcohol, fiber, and potassium may play a considerable role in liver iron metabolism.