Try using a fine-tooth comb or a pair of tweezers.
I vacuum out what I can first, and then I do what Ann does, with the tweezers...time consuming, but it works.
John, since metal BBQ grill cleaners have been exposed as leaving behind the metal that gets in the cooking food people have been throwing them away. However, I find them handy for things that have nothing to do with food, like velcro covered in lint and fuzz! I used our old metal brush made for a grill to come in very handy on the velcro on my coat which would not close due to the fuzz and lint. Something similar would work. Perhaps you can find a small version of one at the local home improvement stores? I am looking for one next shopping trip.
I use a metal flea comb, works like a charm.