How can I keep cold air from going under my house in winter months?

If you have foundation vents you can install ones that close when it gets cold. (Mine close when it drops below 50.) Otherwise I'd concentrate more on insulation to keep the crawl space cold air from seeping into your house. And insulate your cold water pipes, of course to avoid freezing. (Insulating hot water pipes may also give you hotter water.)

I agree with Lizbeth, definitely consider encapsulating the space under your house if it's a crawl space. I remember growing up we would use bales of hay to block the vents, but as an adult I would worry about things making a nest. It worked though.

I have a large crawl space under my kitchen. The weatherization experts told me not to fill it with insulation as there are pipes and the possibility of problems with moisture. They suggested lining the walls with wrap insulation if possible. Pipe wrapping is a good idea in particular any exposed to the weather or in an uninsulated areas. We ended up insulating any walls, kitchen ceilings and we put insulation around the walls of the crawl space.