The Remain camp would rather live in a world where the public has no voice