You sound just like me! I was always called skinny as a child so after I had my children I just went back to being thin again. It took me a very very long time to realise this but a lot of the people who keep picking on you about your weight are jealous. Most women struggle with losing pregnancy fat so consider yourself lucky Also, I suggest you decide for yourself what looks like a nice weight to be at and aim for that. You'll realise that being thin is actually a plus
Jealousy makes people say petty things, so just ignore them, and be confident with how you look: you can eat what you like, when you like, and still look thin! Take advantage!
Your weight is at the right proportion with your height. It's all about the genes. Do not mind those people who make nasty comments about your physique. It goes to show that they're just jealous of your figure.