From Nookipedia, the Animal Crossing wiki
Each entry is rated in points according to size (small, average, big), rarity and luster (dull, average, impressive) - which makes it random to determine what insects will be rated highly, but more common insects tend to be overall lower rated than rarer ones. Therefore, handing over an insect species twice may not produce the same results, meaning the same insect may be able to overwrite an already existing entry.
During the day of the competition, each time a player gives a bug that achieves a new high score, they receive a gift from Nat. At the end of the competition, the winner of the Bug-Off receives a mention on the Bulletin Board as well as a trophy through the mail. There are no trophies for second place.
The Bug-Off is the counterpart to the Fishing Tourney, which runs during the other eight months of the year. The Bug-Off is different to the Fishing Tourney, since there is only one type of competition - bugs are caught according the color, rarity, luster and size only, unlike the Tourney which has two competitions - size and type.
 Names in other languages