To answer the question, "are all men pedophiles?," we must first define the term 'pedophile'. In a certain colloquial sense, 'pedophile' is used to refer to anyone who expresses an attraction to girls (or boys) under the legal age of majority, which in the U.S. is 18. In that case, I think it is not at all inaccurate to state that most (though perhaps not all) men are pedophiles - strictly by that definition of pedophilia.
Girls reach puberty, on average, by the age of 12, and even though they are not fully developed by that time, a great many girls are by the age of 16 or so (and certainly some reach that stage even earlier than that), which is reflected in the fact that many states have set the age of consent at 16. It is not abnormal at all for a full grown man (no matter how old) to be attracted to a physically-developed 16 year old girl, and I don't see this fact as cause for alarm.
Now, on the other hand, this usage of the term 'pedophile' is not technically accurate, and that poses a different problem. Pedophilia is, very specifically, an unusual form of sexual attraction whereby a person (even into adulthood) is attracted to prepubescent children. The 'prepubescent' part of that is very important, as it is the part that separates the pedophile from everyone else, and indicates his attraction as abnormal. A normal man would not normally be attracted to prepubescent children, as it is the sexual characteristics of the post-pubescent form that he responds to. That is, precisely, what makes the pedophile unusual.
So how do we reconcile these two perspectives on the phenomenon of pedophilia? I believe it is important to maintain a meaningful definition of the term pedophilia, by restricting it to cases of attraction to prepubescence, for the sake of saying what we mean and meaning what we say. At the same time, the term is widely used in inappropriate contexts, in an attempt to disparage a natural part of sexual attraction whereby an adult may be attracted to the virility and nubility of youth, but not beyond the [pubertal] line that would mark that attraction as abnormal.
I see this as an extension of a broader agenda of sex-negativity, where sex is cast as a vile and impure act, that has the power to corrupt innocence, and that is conceptualized as a weapon used most often by men to degrade and subjugate weak and impressionable women and children. As you might guess, I do not view sex in this manner, and I stand staunchly opposed to those who would.
And as to the question of "are all men pedophiles?" The answer is yes and no. A great many - if not all - men are capable of being attracted to underage girls; this should not be alarming, and it need not be problematic. But those men who are literally (as opposed to legally) attracted to children (distinct from adolescents - a time in one's life that is defined by sex at least as much as childhood is defined by its absence - and young adults) probably remain a minority, which should also not be alarming. Although whether or not that remains problematic is a topic worthy of careful consideration.