Yes. Chalk paints aren't meant to be washed. That's why I tell folks not to use on cabinets, counters or table tops.
Chalk paints are really hard to wash, unless they have a seal coat on them
If you painted a wall with chalk paint and didn't finish with a wax or poly then you would have this problem. If you never sealed your chalk paint on the wall with anything then you can treat the chalk paint like a primer and just paint over it with latex paint. If your wall was sealed with a wax or poly after you used chalk paint on the wall then you have some more work to do before you can paint. I am guessing that you only used chalk paint on your wall with no finish coat of wax or poly.