In on-site horse race viewing, spectators identify racehorses based on the silks worn by jockeys. However, many silks are visually similar, and the target silks change for each race. Spectators must temporarily memorize the current target silks and then forget them to avoid memory interference in subsequent races. To address this challenge, we propose a puzzle-based method where users reconstruct silk patterns from divided parts to promote active engagement in memorization while facilitating later forgetting. We evaluated this method through two independent experiments consisting of an image recognition test and a video recognition test, using a conventional coloring method as a comparison baseline. The results showed that while both methods achieved high accuracy in the image test, the puzzle method demonstrated superior identification accuracy in the video test, particularly for distinguishing similar silks and maintaining performance across different camera angles. Furthermore, across both experiments, the coloring method showed a higher tendency for memory interference, whereas the puzzle method appeared to facilitate forgetting more consistently. Subjective evaluations indicated that the puzzle method helped users notice subtle differences between similar patterns and was preferred by many participants for long-term use due to its lower operational burden.