The biggest downside to Apple pay is training store clerks that you have already paid (via your watch proximity) with Apple pay, it takes a bit but eventually they figure it out.

Possibly in the future apple watch pay users might develop sore shoulder muscles due to barely noticable arm movements. But once you have exercised that muscle you will be good to go. :-)