You are not alone. A few people are troubled by wasps, and have asked about them here. These are a couple of threads that may offer helpful information:
A wasp trap will work. The bees are attracted to the sugar mix that is used to attract the birds, The traps which is not much more then an upside down glass jar or soda bottle that holds more of the sweet stuff will allow the bees and not the birds to enter and prevent them from coming back out.
You can pick these up pretty much at any home center, or farm supply such as Tractor Supply. http://tsc.tractorsupply.com/tractor/Wasp-Traps
Thanks for the answers, but I'm still hoping for more ideas.. I've been using wasp spray, but I'm afraid that spraying the wasps with wasp spray onto the feeders (usually about 2-3 wasps per feeder. I have 7 feeders up. I don't want to hurt my hummningbirds with spraying the hummingbird feeders. I try to spray them when they fly away, but, some have to be sprayed on the feeder. Will wasp spray hurt my hummingbirds from spraying the feeders?
Yes, you do not want to risk getting the chemicals on the feeders as it may discourage the birds from feeding there again.
Try the traps. Your fighting a up hill battle in a rain storm trying to keep the wasps away. Unless you can find their nests, which are very difficult as they can be quite far away. They will simply keep coming back. Might as well catch them in the act.