Physical activity (PA) during pregnancy may have a positive effect on the fetal cardiac maturation which is reflected in a decreasing resting heart rate and increasing heart rate variability (HRV). Different types of PA, for example during leisure or work time, have differential effects on HRV; however, this relationship has not yet been investigated in pregnancy. In our work, we related different types of PA during pregnancy with maternal and fetal HRV. We assessed the levels of PA in 95 pregnant women between 28 and 32 weeks of gestational age using the Baecke Physical Activity Questionnaire. Maternal and fetal heart rate and HRV were extracted from magnetocardiography recordings at rest, and maternal anthropometric and metabolic parameters were measured, such as fasting glucose and insulin levels, body mass index, and blood pressure. Pearson correlations were calculated between HRV, PA, and maternal parameters. Principal component analysis and generalized linear models were implemented to further investigate these relationships. Our findings indicate that habitual physical activity, whether during leisure or work, has no significant effect on maternal or fetal HRV at rest. However, leisure-time physical activity, unlike work-related activity, is associated with improved maternal insulin sensitivity. Additionally, our exploratory analyses revealed that lower HRV in both the mother and the fetus is associated with poorer maternal metabolic health quantified through higher fasting insulin levels, triglycerides, and adiposity. Finally, male fetuses showed higher HRV compared to females, highlighting the difference in cardiac development between the two biological sexes.