I completely agree, I don't think he ever loved any of them.
As an outsider looking in / my opinion, he married the 1st three because that is what he was supposed to do. Gather wives, have children, populate the earth and gain accolades in heaven. "Love" was never a factor for Cody. The marriages were tasks to be completed.
Now his needs are fulfilled with Robin, she is giving him young children, and he is done with the other three but does not want to give them up as part of his family.
It is a hard show to watch.