I've got 4 different places around me in Birmingham AL where kids can drive Go Carts of varying speeds on varying tracks. From ones the parents need to accompany but tykes can drive straight through the 50 mph scare fests. On I Drive alone off the top of my head you can go to Nascar Karts, Magical Midway, and Fun Spot America and drive go karts. The first two are even... battery powered ... so they don't stink. Then there is K1 Speed and Orlando Kart Center not far away.
Speedway has nostalgia. But really, for a park that is (artificially) constrained by space, Speedway just is a poor option given the size, the low capacity, and the high maintenance (ever try and keep gas go kart motors running? It's a chore I promise you!). Further, I think most people go to Disney to experience things they can't get anywhere else. Your local amusement park simply doesn't have the quality attractions, rides, and shows you get at Disney. Even the basic spinners are better themed than anything you get anywhere else (Astro Orbiter anyone?). Plus the place is clean, with wonderful music and smells that they actually sell in terms of scented candles.
When you get right down to it, Speedway is the antithesis of all of the above. Every family fun park can do it. It's not themed at all. It is dirty in both sound, smell, and environmental pollution, so it drowns out the music and drags on the atmosphere.
The hate? Well I wouldn't say I hate it. It just isn't Disney. Absolutely nothing about Speedway to me says unique to Disney. The only Disney thing about it is that it has always been there so we have memories of it being there. I don't want Speedway replaced by something for adults. But I think you could absolutely replace it with 2 unique and interesting attractions that fit the Disney image and accommodate the toddler through tween customer, add capacity and better the Park.
But I get that other people have other opinions. I just don't understand them.