We have been the last week of March the few times because that is always our local school's spring break. The crowds have varied over the years, but never been light! To put this in perspective, there have been years where we were able to pick up several FPs in the evening at parks like each HS and EP and big ticket rides at MK like Space Mountain. Last year, there was no getting evening FPs anywhere except for MK, and even then it was little rides like Dumbo, Barnstormer, or Ariel. We are a travel party of two, so I worked refresh quite a bit and still had trouble finding FP in the evenings.
Also, the weather has varied. One year it was mid-90s the whole week and we melted. Then this last year, the weather was amazing, and we actually had to bring jackets with us for the evening shows! Either way, swimming in the afternoon was fine. The only time I've had difficulty swimming was one November at POFQ when the pool did not feel heated at all.