Several examples of allosteric modulators of GPCRs have been reported recently in the literature, but understanding their molecular mechanism presents a new challenge for medicinal chemistry. For the specific case of the cellular receptor CXCR4, it is known that pepducins (lipidated fragments of intracellular GPCR loops) such as ATI-2341 modulate CXCR4 activity agonistically via an allosteric mechanism. Moreover, there are also examples of small organic molecules such as AMD11070 and GSK812397 which may also act as allosteric antagonists. However, incomplete knowledge of the ligand-binding sites has hampered a detailed molecular understanding of how these inhibitors work. Here, we attempt to answer this question by analysing the binding interactions between the CXCR4 receptor and the above-mentioned allosteric modulators. We propose two different allosteric binding sites, one located in the intracellular loops 1, 2 and 3 (ICL1, ICL2 and ICL3) which binds the pepducin agonist ATI-2341, and the other at a subsite of the main extracellular orthosteric binding pocket between extracellular loops 1 and 2 and the N-terminus, which binds the antagonists AMD11070 and GSK812397. Allosteric interactions between the CXCR4 and ATI-2341 were predicted by combining different modeling approaches. First, a rotational blind docking search was applied and the best poses were subsequently refined using flexible docking methods and molecular dynamic simulations. For the AMD11070 and GSK812397 antagonists, the entire CXCR4 protein surface was explored by blind docking in order to define the binding region. A second docking analysis by subsites was then performed to refine the allosteric interactions. Finally, we identified the binding residues that appear to be essential for CXCR4 allosteric modulators.