Well, quite a few people come to cosmology from a particle physics angle; many also come having specialised more in astronomy as undergraduates. That applies especially to cosmologists who don't focus on the very early Universe. Sometimes it seems to me that when someone says "cosmology", a lot of people (even physicists outside astronomy) think straight away of very early Universe cosmology and all the relatively speculative things that go with it.

A great deal of what's considered to be cosmology is instead to do with the subsequent evolution of the contents and structure in the Universe, which depends far less on speculative physics, and more on 'standard' physics but complicated astrophysics (the details of galaxy formation, for example). This is why I have a problem with the generalizations you sometimes see about 'problems' with the subject. To assess a lot of the ideas and results in modern cosmology, a good knowledge of astronomy would be more important than detailed knowledge of particle physics. I'd agree with bethe123 that general relativity is required, of course. I only quibble because I think that the average student who's taken physics to a high level is more likely to have missed out on advanced astronomy courses than advanced particle physics or GR courses. It's certainly the former that I had to catch up on when I started research in cosmology, having come from an applied maths background.