Jacob Rees-Mogg has suggested the Conservatives introduced voter ID to boost their election chances, but it came "back to bite them".
The former minister said it had "made it hard for our own voters" to take part in England's local elections.
The polls on 4 May were the first in Britain where people had to show photo ID, such as a passport or driving licence, to vote.
Mr Rees-Mogg said the change had "upset a system that worked perfectly well."
The government says the new voter ID rules are necessary to tackle voter fraud, and boost public confidence in elections.
But Mr Rees-Mogg told the BBC there was no evidence that the issue of voter fraud was a "serious problem" - something the government's opponents have long argued