Is this wrong of me not to tell him?

It's no big deal unless it's bothering you so much that you have to tell him. You shouldn't feel the need to, but if you must get it off your chest, he should brush it off. It really isn't any of his business, though.