have you tried an electric razor? cleans it off real nice
A shaver you know the ones like 5 for a dollar tree. Then use a lint brush to clean up.
Make sure sweater is laying flat, Lint brush not the roller kind but the other one.
What is the sweater made out of?
Depending on the density of the knit, and the thickness of the yarn, sometimes razors are not only ineffective, they can cause more harm.