From Nookipedia, the Animal Crossing wiki
Buying seeds, saplings, signs, paint, candy, tools and stationary will not give you tickets. Once you have 5 tickets, you will have one chance. The more tickets, the more chances you have. Usually, Nook will offer hard to find things, or more often, an NES game. After you give him five tickets from that month, he will spin the balls in the container. Depending on the ball's color, you will either get a specific prize (gold ball for first place, red ball for second place, blue ball for third place), or no prize at all (white ball). Tickets cannot be sold for anything, and if it is not the month to use them, wait till that month comes again, or simply, throw them in the dump.
The Lottery does not appear in later games - instead, it has been replaced with the Spotlight item.