I feel unhappy, I'm a teenager, help!?

Ok I have a lot of family issues that I deal with in my own way, however there are some issues which I can't solve. At school I have a group of friends, we all laugh and joke however I would consider myself the joker of the group and with that take a heck of a lot of abuse. Sometimes (I know this sounds stupid) they attack me for fun and I don't retaliate, that is just me, I can't hit people back I am just not like that. Maybe some people take that as a weakness but I take that as a strength, I tend to forgive people. However I get very upset with this, some of the comments people say are also horrible and dig deep. Today this person annoyed me, this person annoys me a lot, however I tend to think of him as a outcast to the group but I just really wanted to attack him badly, and I mean seriously, how do I stop my self, I have stopped my self in the past but now it is reaching breaking point. I do have a lot of friends who do not participate in annoying me, some of them get the same treatment, however I think I get it worst. Please help me.