I don't drink or smoke and feel bad about it...?

I'm 19 and don't drink or smoke. Drank a couple times and it hits me easily and I might do something stupid so I quit. I feel like a loner cause all of my friends drink or smoke and my friend told me dude you can't drink right. I feel like wtf, it's just not my thing what should I do about this? Build up a tolerance or just say screw it and live a sober life?