Firstly sand down to remove loose rust, use a metal primer to seal then repaint. I have done a similiar project https://vickymyerscreations.co.uk/upcycling-2/transforming-rust-covered-bathroom-cabinet-pintyplus/
Vickymcreations suggestion is great if you want to paint a different color. You can remove the rust with a bunch of crumpled aluminum foil and seal with Rustoleum 2X clear spray paint or metal spray paint.
I would definitely do as the other comments suggest and sand it down to remove as much rust as you can first then you can paint or spray paint any color you want. I would recommend something like mentioned above a rustolem, or a spray paint/paint that is meant for outdoors as it will stand up to the moisture in a washroom
Sand prime and repaint. Sadly this is a common occurrence I wish I kept my plexiglass caddy.
I would replace it with a plastic caddy. There are some really great ones out there at low prices. Cheaper than fixing the one you own.