My rose bush has a few yellow leaves with black spots how do I get rid

You have what is known as Black Spot. It's caused by water droplets left on the leaves after a rain or watering the plants. There is a spray you can use to prevent this. Watering just the base of the plant also helps. Pull off any yellow leaves and put them in the garbage.

I've heard spraying with buttermilk works. I just left mine be, they'd fall off and then re-leaf. Never water at night, leave time for leaves to dry before night.

This is called black spot and it is a fungus. Remove any affected leaves and use a product such as "Safers" brand fungicidal soap. When watering, do not water leaves or flowers directly and if possible water in the morning as evening watering tends to promote disease.