How to smooth out bathroom walls?

You can skim coat the walls with sheet rock mud. Once dry, smooth with a damp sponge. You might have to do this a couple of passes before it's perfectly smooth.

If you decide to remove the wallpaper, here is what has worked best for me.

Peel as much as possible from the walls.

If you can't get to the backing, you may need to score the paper with a tool. Paper Tiger is easy to find. Light pressure because you don't want to damage the wall behind the paper.

Mix 50/50 fabric softener & warm water in a spray bottle or garden sprayer. I've read that vinegar is as effective but I have never used it.

Allow the mixture to sit for 5 - 10 minutes so that it saturates the adhesive on the paper backing.

Scrape with a plastic scraper.

Once it is all removed, wash the walls with warm soapy water (blue Dawn is my go-to) and a microfiber cloth.

You need to remove the paper. Soak the paper with Dif wallpaper remover with hot water and spray on with bug sprayer or spray bottle. Let sit, keep wet, then scrap off. You will have to do some repair to wall if it wasn't prepped correctly, but still less work that trying to smooth out your paper which could actually buckle up if you apply other products.

Removing the wallpaper and having it skimmed would be the best in my opinion.