I wouldn't feel anything, why should you ?? they are probably whispering because what they are talking about isn't work related and the supervisor probably thinks it doesn't look very professional to be discussing last night's TV in front of staff.

Unless you are totally unsecure about yourself and paranoid or indeed that big-headed that you assume they are interested enough in you to be discussing you in whispers, then perhaps you have a problem with yourself you need to sort out.