HIV-1 Nef-mediated CD4 downmodulation involves various host factors. We investigated the importance of AP-1, AP-2, AP-3, V1H-ATPase, β-COP and ACOT8 for CD4 downmodulation in HIV-1 infected shRNA-expressing CD4+ T cells and characterized direct interaction with Nef by FRET. Binding of lentiviral Nefs to CD4 and AP-2 was conserved and only AP-2 knockdown impaired Nef-mediated CD4 downmodulation from primary T cells. Altogether, among the factors tested, AP-2 is the most important player for Nef-mediated CD4 downmodulation.