First of all, make sure the Sheetrock is the mildew resistant grade that is made just for damp, wet areas. This is easily identified because it is green or purple in color on one side. Then, if you are talking about sealing the Sheetrock itself, there is a membrane you can buy that roll on like paint. If you are talking about sealing down where the tub meets the Sheetrock you would use silicone caulking. Make sure it is the type rated for ’tub and tile.’ This is the mildew resistant type.
We had the shower overspray onto the sheetrock at the far end of the tub/shower enclosure. What we did was get the mildew resistant sheetrock, put it up and seal with caulk. After that we went to Home Depot and bought decorative stones glued to mesh, arranged them on a flat surface and then put them up. Grouted and sealed, then put the grab bar back up. Beautiful.