Am I undermounted?

My pony is 12.3 with shoes and pads and I play mounted games through pony club and MGAA. I have worked very hard with this pony and have turned him into everything he is today. When I got him he was a driving pony and that was all he knew. He spooked at everything. He still freaks out and I ride him in a hackamore and a martingale when playing. He has learned that when I put all my wheight down through my and heels that he is to stop. He is a very hard to ride pony. I am 5'6" and about 140. I know this sounds way to big but this pony has never had an issues carrying me and we have gone on 10 mile gallops through all sorts of terrain and he hasn't been sore. Some people have been giving me crap about me being to big for him even though I am in the medium pony wheight limit. What do you think? This pony couldn't be ridden by a child as he is very spooky and it takes a lot for him to trust. What should I tell these people if you don't think I am to big.