It doesn't matter which resort the OP is having lunch at, unless they are willing to walk to the Contemporary from the MK and back from the Contemporary to the MK.
Using the Monorail is a zero sum game for all of the resorts on the beam.
In all cases
You will have to wait to board the monorail at the MK
For a restaurant at Contemporary
. If you decide to have lunch there, you will get off the monorail fairly quickly, have lunch, then wait for the monorail and have stops at the TTC, Poly, and Grand Floridian
For a restaurant at Poly
If you decide to have lunch there, it will stop at the Contemporary, the TTC and then Poly, have lunch, then you will have stop at Grand Floridian.
Etc.
So if depending on where you want to have lunch there may be minimal front in time cost but with a back end time cost and vice versa.
This doesn't take into account the "holds" that may occur.