Joe Alwyn admits he just tried to 'control' what he could amid relationship with Taylor Swift