Not telling people

When going somewhere in the summer, I often drive in just shorts and put my shirt and sandals on when I get there, if it is a place where I need them. I play in three community bands, and have been seen putting on or taking off shirt and shoes by other band members on many occasions. One time one of the men said to me in a pleasant way that he sometimes thinks that I wish I could take everything off and just live nude.
I gave him a vague answer.
In contrast, the only other person in all 3 bands sometimes wears shirts advertising his nudist lifestyle.