A key point. The Tories have often been mistrusted on key issues such as the NHS and education, and they've never quite shirked of the tag as a nasty, privileged party many give them, but their stomping ground, so to speak, has almost always been the economy - Thatcher painted herself as a tough, no nonsense figure coming in to fix Labour's irresponsible mess, and Cameron's Tories painted Labour as fiscally irresponsible during the Blair and Brown era, and aimed to appear more sensible through austerity. They've often created this image of a party that's kind of like a teacher or boss you don't like, but whose authority and ability you can respect all the same.
Brexit blows that out the water. As a party they're jumping into the unknown to an extent and taking a massive, irresponsible plunge, the sort of move they'd persecute an opponent for doing.