You can swim if you're from the great white north!! We've travelled early January twice, and my kids were in the pool almost every day. I, on the other hand, would only go in if the temperature hit 70. I think I was in the pool twice both trips.
We had a real mix of weather both times. A few days were balmy in comparison to what we'd left behind (low to mid 70's), a few days were 60'ish, and then a couple of days were downright cold! In fact, the evening we sat and waited for Fantasmic, it dropped to freezing. We got soaked at AK and had to stand in front of these huge dryers for about half an hour - it was too cold to leave without being perfectly dry. So pack a jacket, gloves and hats just in case. The worst thing that can happen is you won't need them! The benefit of putting up with the unreliable weather is the lack of crowds.