The pastor of the London church that Sir Keir Starmer apologised for visiting says he is “disturbed” by the Labour leader’s comments.