That would require a lot of "elbow grease" (so to speak) so I would power wash it. You can rent a power washer by the day or half a day, BUT home improvement stores have great ones now that attach to your hose. If that does not work, try a self contained sprayer like Walmart has (not the $8 one like we got for lightly spraying) but the better ones with more power in them and you can add your choice of cleaner in them too.
I've had vines growing on it. Not sure a pressure washer will take them off.
I sympathize Doug, I had a fence full of ivy my neighbor wanted gone... took forever picking it off with small pruning shears.
Since you've had vines, that may have actually pulled some of the paint. You might want to clean and spray paint once it's dry.