Adipose tissue is essential for energy storage and endocrine regulation of metabolism. Imbalance in energy intake and expenditure result in obesity causing adipose tissue dysfunction. This alters cellular composition of the stromal cell populations and their function. Moreover, the individual cellular composition of each adipose tissue depot, regulated by environmental factors and genetics, determines the ability of the depots to expand and maintain its endocrine and storage function. Thus, stromal cells modulate adipocyte function and vice versa. In this mini-review we discuss heterogeneity in terms of composition and fate of adipose progenitor subtypes and their interactions with and regulation by different immune cell populations. Immune cells are the most diverse cell populations in adipose tissue and play essential roles in regulating adipose tissue function via interaction with adipocytes but also with adipocyte progenitors. We specifically discuss the role of macrophages, mast cells, innate lymphoid cells and T cells in the regulation of adipocyte progenitor proliferation, differentiation and lineage commitment. Understanding the factors and cellular interactions regulating preadipocyte expansion and fate decision will allow the identification of novel mechanisms and therapeutic strategies to promote healthy adipose tissue expansion without systemic metabolic impairment.