I used to serve for 4 years, so I'm reading OP's post just cringing. While serving, one thing I made a point to do with each table was COMMUNICATE! If I was slammed and unable to tend to someone's needs immediately, I would literally just tell them that I was tending to someone else but if they could sit tight I'd be right with them. If I was really too busy, I'd reach out to a co-worker for assistance. If ticket times were running longer than usual, I'd tell them. I'd offer them something small to show appreciation for their patience. I'm really confused as to why servers don't just speak up about these very basic things. If you can communicate with your tables, your guests will be a lot less annoyed/unhappy. Rational people are able to tell the difference between what is out of the server's hand (food errors, long waits, prices, etc) and what's the server's responsibility (catering to the guests needs, checking in with their guests, filling drinks, clearing plates, offering suggestions, making sure the check is correct, etc). So, as I'm sure you all could guess, when you did a job for an extended period of time, you can't HELP but have a more fine-tuned observation of scenarios like this.
If the server that this OP had was truly just socializing with a table and neglecting others, that's reason enough to complain. I'm sorry, but it is. When I am in a restaurant, busy or slow, and I see someone ON THEIR PHONE, or socializing with co-workers, I instantly become annoyed. It's hard for me to like my server after I've watched them ignore me as they sat on facebook or gossiped with their friend for 10 minutes as i sat there needing something. The server mentioned originally should be held accountable for giving you and other tables the cold shoulder during your visit. He shouldn't have a job there. Work is not social hour. There's nothing wrong with making small talk and making your guests feel welcomed and relaxed, but by the sound of OP's description, he wasn't doing that. Maybe to the table with the girls, but not the rest of his guests.
In regards to whether or not you should complain now, hmm....I guess you could. However, I'd make it known in your email the reasons you didn't report at the time. When I was serving, people would complain 2-3 days later about them not liking their food, expecting a full refund. Lol. *Scratches head*. When there is a delayed response to something that is angering you enough to reach out and bring it up, it can raise suspicion to managers. It looks like a money grab. But in OP's situation, the food wasn't necessarily the complaint so much as the service and overall experience. Again, lay out WHY you're writing after the fact and stress the reason you were irritated by this server's lack of attention paid to you and others.
It pains me to hear that this server didn't care, because honestly, Disney trips are looked forward to by SO MANY for so LONG! I know for me, I count down the days til I get to go back, and I peruse the menus of the restaurants I have reservations for. I look up photos and reviews. I build up my excitement so much that by the time I get there, I'm only expecting the best because of the way I built up the excitement in my own reality. I know for these servers, this is their everyday routine and the magic may start wearing off a few shifts into their time there, but they need to keep in mind that the people they're waiting on have been looking forward to this meal and are hoping for the best. I'm completely understanding of things that are out of their control, but the LEAST a WDW server could do is have a good attitude and a smile on. If they can't do that, they really shouldn't be serving. When people came into my work for their anniversary, birthday, retirement, etc., I'd be so honored to be able to make their night special. Seriously, it sounds cheesy, but it's true. I know my response was long-winded, but I empathize with the OP because I know how annoying it is to have high hopes for a place and then be given a server like that. It just sucks. It makes you wonder why the heck this person serves if they don't like it or care to do it. Thankfully, I've yet to really truly have a bad dining experience at WDW...did I just jinx myself?