Lupin: The dementors affect you more than others because there are true horrors in your past, horrors your classmates can scarcely imagine. You are not weak, Harry.
I think you have the question backwards. The dementors affect Harry more because he has suffered more than the others. Dementors make their victims relive their worst memories. For Harry, this was seeing Voldemort murder Lily right in front of him. It would be enough for anyoned to faint.