How do I keep solar lights upright?

If the stakes are similarly sized, insert them into bricks that have circular holes which you have mounted flush to the ground, or place heavy rocks on at least two sides of the light stakes.

Another idea is to buy PVC pipe with a diameter that will fit the stakes. Cut the PVC to a height you want (at least 18" is recommended for frost freeze areas), drive them into the ground, then insert your light stakes into the PVC opening and glue in place, if desired. This way, your lights could be further off the ground.

I like the brick idea from Oliva. If the holes are too large, put some floral foam in the hole like is used for funeral floral arrangements since they are designed for outdoors.