Sure, I felt that way. It's a completely natural reaction.
I used to be very self-conscious, but after awhile, that faded.
I came to a realization that it was pointless to worry about the part of my appearance over which I had no control.
Plus, the truth is, most people aren't being polite, it's much more likely that they are just unaware. Most people are so caught up with what's going on in their own lives, that they don't notice a lot of the details of anyone else's life unless there is some kind of figurative neon sign attached to it, pointing out to them what seems blatantly obvious to you.
I used to apologize to people for my neck being fused, until sooo many people gave me that puzzled look that says, they hadn't even bothered to notice. Now I only explain when someone asks or comments on it.