My veg Garden will only grow small vegetables.

I live where its clay also. Have you thought of having raised gardens filled with fresh dirt?

Same problem, so I started the seeds indoors to extend the growing season.

This allowed me to put in bigger plants when everyone else was planting the cheapie small plants from the home centers. I also put in drip irrigation-- a beaten-up old hose that I stuck some holes in. Both gave me much better results than in the past.

If you've got an extension service, see if they'll do a soil test for you to give you some idea of what might be missing from your soil.

Your soil is too heavy. Try mixing 50% potting soil in the garden soil. Wait until the season is over and buy potting soil on clearance. You might try moving to raised beds and using commercial garden soil/potting soil mix with about 30% your soil.

One of the things I've learned over time is that if there is too much nitrogen in the fertilizer mix it will cause beautiful plants with very small vegetables or even flowers. Many times it's missing enough phosphorous so even if you're fertilizing it might be the wrong mix. Read this article about growing vegetables and see if that helps:

http://homeguides.sfgate.com/causes-large-vegetable-plants-but-small-vegetables-56696.html

I also like Lois Vrns idea of raised gardens but regardless of how you do it, even with fresh dirt, you have to find the right balance of nutrients for what you're growing. Some farmers will actually let their fields grow with cover crops to make the growing environment better. My daughter and son-in-law have such a farm. They started off on their new farm adventure fine with the mission to learn to feed themselves and share with their community but they found that the dirt needed nutrients so they took the time to grow cover crops to build their soil. While their gardens may be slightly larger than yours, it's not like they are huge commercial farmers. But even commercial farmers have to rotate their crops if they want a good yield. It's really a science and way more than you should need for a home garden, just something to be aware of.

Here's an article about building better soil, if it interests you at all. It's a start.

http://www.sare.org/Learning-Center/Books/Managing-Cover-Crops-Profitably-3rd-Edition/Text-Version/Building-Soil-Fertility

I would hire someone to come in and heavily till your garden adding in as much good compost as possible and then see what happens.