Grouping by size or color is one option. Another way is to stack a few books flat to create bookends for the standing books. Alternating the standing and flat books looks visually appealing while allowing for more books to be shelved
Here are some cool styling ideas that you could apply to your own books: https://www.buzzfeed.com/chelseypippin/bookshelf-porn
You can stack the books flat under some plants, use two plants as bookends and even display a book here and there with an easel.
Stack some vertically to mix it up!
The books could BE the plant shelf.....
'Lots of ideas for creative bookends on pinterest: