Academic research on the question is discussed here. The answer seems to be the one given (inadvertently) by Brice Hortefeux (indeed, cited as an epigraph in the paper): "Il en faut toujours un. Quand il y en a un, ça va. C'est quand il y en a beaucoup qu'il y a des problèmes." Or, as Henry Farrell puts it:
Adida, Laitin and Valfort conduct a variety of field experiments, and find that 'rooted' French people (those with four native grandparents) are less likely to be generous when the 'salience' of Muslims in the group increases. They furthermore suggest that this is best explained by a taste for discrimination rather than some rational system of beliefs about how Muslims will reciprocate or not reciprocate towards them.