I feel exactly the same way you do. We can afford the deluxe resorts, but I just get such an uncomfortable feeling when I think about what we could do with that money. It nags at me the whole time we're there and prevents me from even enjoying it properly, which is ridiculous but I can't help it.
We've stayed in all levels, and personally I prefer the moderates. Nice pools, nice enough food, nice atmosphere, and a pretty nice price.
Unless you're planning to spend a lot of time at the resort itself, to me it's not worth it. We're coming up for one at Yacht Club next month and we're not going to visit the parks at all. We're just going to enjoy Stormalong Bay and get our money's worth, haha. We're staying club level because we'll be there all day to get our money's worth out of that, too. Hmmm...maybe I'm cheap.
If it wouldn't be too much of a hastle for you, you could always stay at POR for the first part of your trip, and then move over to WL for your last night, as a treat. That's the only way I can make myself stay in a deluxe, if it's only a one-night stay.
But then again, we live fairly nearby and don't consider Disney trips our "vacation," they're just extras. If this is your big vacation for the year maybe it's worth the splurge. Haha, I'm no help!
You should really just do what makes you most comfortable and know that you'll have a great time regardless of where you stay!