English is a wonderfully expressive and comprehensive language but its piecemeal construction and giant inconsistencies make it very difficult to master. I sympathise with the less/fewer conflation a lot of the time. In actual speech, I'm sure I sometimes mix them up as well.That's one that that a lot get wrong.
After all, more or less is a phrase, so it's easy to misunderstand them as being opposites in all situations. You'd say less time, rather than fewer time etc.
Much less annoying that lose/ loose, your/you're etc