Is it ever acceptable for a man to wear a vest in public?

As a University Challenge contestant flouts the show's strict dress code, Stephen Doig asks: is a man in a vest the ultimate fashion crime?