I don't care if you don't want to wear one but carrying one around without using it would indicate to me that:
A: Your significant other/parents have told you to wear one and you carry one so that when you leave the house they see you carrying one and assume you're going to wear it.
B: Part of you thinks you should wear one but you just can't bring yourself to strap the polystyrene monstrosity to your head... in public.
C: You are under the misguided impression that, when one came with your bike, it was some sort of technical thing that belongs with said bicycle and that to separate them by more than four feet will render the bike useless.
To all of you, be it A, B or C:
You are lame. Make the decision and stick to it, grow a pair while you're at it.