To me this is obvious, but maybe not... Please consider the example of new countries joining the EU and the transitional arrangements that are put in place.
https://ec.europa.eu/social/main.jsp?catId=466&langId=en
As you can see, for 7 years people from Croatia (in this example) are free to travel throughout the EU unrestricted, but Croatians do not have the right to work in another EU country. This shows that it is of course technically possible and mirrors the situation that would occur in a European Union where member states were free to set their own rules on eligibility to work, without
automatic Freedom of Movement for Workers. Like the Croatians, people could travel around as they please, unrestricted, but would not be permitted to work without first obtaining permission from the nation state.
Yes, that would be impossible as people from Manchester have exactly the same documents (passports) as the people from London. And for economic reasons you don't want to restrict movement of workers within a currency block, in this instance the Pound Sterling zone.