Either attach them to something you can attach to the wall (like a board) by drilling small holes into the mortar and adding wall anchors or use an adhesive like liquid nails. I would personally do the board so that if your style changes you can change the number style without so much trouble.
How I did mine
Ideas from other Hometalkers
Agree with Twyla. You can use concrete anchors in the morter.
Use missionary screws screw into cement not brick