I know this looks a little silly, but stay with me on this! The first thing you need to do is figure out how big you want the wall decor space to be. I would outline the area with brackets using chalk. This will give you a good basis.
Then I would start imaging where the larger books you plan to use will go. By placing these ahead of time, you will save yourself a lot of headache! Just draw imaginary square books on your wall with chalk, and if you don't like the layout you can easily erase it and try again!