I often wonder if I spend too much time defending myself. But while in the right context, "live your life and don't worry what others think" can be very good advice, it's not always applicable (like when what others think directly obstructs your ability to live your life), and it can sometimes be used insensitively in the sense of, "shut up and quit whining".
The fact is, sometimes (and most times to some extent), it does matter what other people think. If we didn't care at all what other people thought, we'd be a far more antisocial species. But even beyond that, when the way you live your life is so alternative that it offends or contradicts mainstream sensibilities, you may encounter very real dangers just living your life without care to how others might react and respond.
If you're gay, for example, you don't just go and make out with your same sex partner in public when you know full well that the reigning sentiment in your small town is that homosexuals are fair game for abuse. First you take up the cause and fight for acceptance for homosexuality, and then only after enough people's opinions have changed will you feel safe enough to not let the haters dictate the course of your life and comfortable enough to just be yourself.
I wish I could go out in the sun naked regularly (without driving hundreds of miles to pay admission for entrance into a small, privileged community), but I don't have a private garden. Who knows what would happen if I went outside naked, in full view of others - but I'm sure it wouldn't be good for me. The cops may even get involved.
Now if I went to the store wearing a dress (contradicting what mainstream society says are the clothes my gender is allowed to wear), I doubt the cops would get involved (unless a narrow-minded bigot wanted to make a scene, in which case I would hope the cops were on my side, although I don't suppose that's a guarantee). I've been doing it a lot, actually, and have had little trouble so far, although I am frequently "on edge" for the worst case scenario - and that's because it's been drilled into my head (and the heads of everyone else, too) that if you refuse to conform to mainstream cultural dictates, you are leaving yourself vulnerable to socially-sanctioned retaliation ranging from criticism all the way to lethal assault.
What I want is a world not just where I will be begrudgingly allowed to pursue my quirky lifestyle so long as I keep most of it in private and try not to offend too many people, and as long as I don't cross any really egregious social barriers (there was a time, you know, when totally private, totally consensual homosexual sex was considered a serious crime). I want a world where I can be free to be myself and not put too much stock into the opinions of people who don't like the way I live my life, because I have trust in society on the whole to protect me, as I am, despite having strange beliefs and alternative habits.
And the only way to get there is to preach my humanity, demand greater tolerance for diversity, and show how my being weird doesn't make me a dangerous or altogether unlikable person. Undoubtedly, part of that is exposing myself and my views to the world, but that includes being concerned for what others think about the way I live my life, and not carelessly breezing through life, unconcerned for the hatred and fear and bigotry that's out there, directed toward me and others more and less like me who don't follow the guidebook on how you're supposed to live your life.
Absolute, recommended reading: Natalie Reed's followup to Shut Up, That's Why, especially the sections under the headlines "Why do you care what other people believe?" (answer: because people act on their beliefs) and "Can’t we just get along?"