Hi Winie, If a scrub brush doesn't do the trick, you may need to powerwash. Hope this helps!
You will have to pull it off, no way around it. We had the same problem when we bought our house. Ours had a colony of hornets infesting the ivy that had grown into the brick walls. if you have these insects, spray them at night so if they have less reason to attack you.
Pull off what you can. Left over tendrils can be brushed off with a stiff broom.
I have Poison Ivy growing up with my English Ivy. I kill the poison ivy (OTC Pesticide) which kills both, it dries up, I pull them off and the next year they are both back! Right now they are overtaking my kitchen window! I guess this is not an answer, just empthazing with your situation!