You may have to go to the trouble of getting (or making) a sieve to remove some of the gravel from the soil. Once that is done, you will have to test the soil to see what deficiencies it has. You can get soil test kits at nurseries or home centers, or you can find a lab in the phone directory. Nursery staff can help you improve your soil based on the results. Maybe you need to add gypsum to loosen the soil. Maybe you need organic matter. An alternative is to make raised beds and import new soil.