Background Reduced muscular strength in the old age is strongly related to activity impairment and mortality. However, studies evaluating the gender-specific association between muscularity and mortality among older adults are lacking. Thus, the objective of the present study was to examine gender differences in the association between muscular strength and mortality in a prospective population-based cohort study. Methods Data used in this study derived from the Cooperative Health Research in the Region of Augsburg (KORA)-Age Study. The present analysis includes 1,066 individuals (mean age 76 ± 11 SD years) followed up over 3 years. Handgrip strength was measured using the Jamar Dynamometer. A Cox proportional hazard model was used to determine adjusted hazard ratios of mortality with 95% confidence intervals (95% CI) for handgrip strength. Potential confounders (i.e. age, nutritional status, number of prescribed drugs, diseases and level of physical activity) were pre-selected according to evidence-based information. Results During the follow-up period, 56 men (11%) and 39 women (7%) died. Age-adjusted mortality rates per 1,000 person years (95% CI) were 77 (59–106), 24 (13–41) and 14 (7–30) for men and 57 (39–81), 14 (7–27) and 1 (0–19) for women for the first, second and third sex-specific tertile of muscular strength, respectively. Low handgrip strength was significantly associated with all-cause mortality among older men and women from the general population after controlling for significant confounders. Hazard ratios (95% CI) comparing the first and second tertile to the third tertle were 3.33 (1.53–7.22) and 1.42 (0.61-3.28), respectively. Respective hazard ratios (95% CI) for mortality were higher in women than in men ((5.23 (0.67–40.91) and 2.17 (0.27–17.68) versus 2.36 (0.97–5.75) and 0.97 (0.36–2.57)). Conclusions Grip strength is inversely associated with mortality risk in older adults, and this association is independent of age, nutritional status, number of prescribed drugs, number of chronic diseases and level of physical activity. The association between muscular strength and all-cause mortality tended to be stronger in women. It seems to be particularly important for the weakest to enhance their levels of muscular strength in order to reduce the risk of dying early.