I wouldn't switch but I'd keep an eye out, it's also possible the loan gets sold before you even get a chance to have an encounter with them.
FWIW people in this thread I think are considering customer service too broadly. Things like misapplied payments, incorrect processing times or timelines, non-compliance with regulatory requirements (such as not responding to a customer within legal timeframes), not paying someone's homeowner's insurance, etc are all things my very brief searching can find and happening with enough clients. So I'm inclined to understand the concerns of the OP but would hold off on switching immediately.