Takegawa.Lab

Piano performance needs various kinds of techniques such as correct rhythmic keying and rhythm, effective fingering, and so on. However, beginners who try to play the piano almost always give it up due to the difficulty of acquiring those techniques. To reduce learners' labor, we have developed a real-time fingering detection system for piano performance and a learning support system for using it.

Fingering Recognition Technology

The fingering recognition system detects fingering by a simple image processing of color markers attached to the finger nails which do not interfere with performance, furthermore it is able to conduct the fingering recognition in real-time. Our system corrects fingering detected by the image processing by applying rules based on a natural manner when playing music. These rules are defined by the features of piano performance, the physical characteristics of a keyboard, and common fingering functions. High-accuracy fingering detection is difficult not only in the image processing, but due to the fact that fingers frequently overlap each other; for example, the thumb is easily hidden by the other fingers in piano performance.

We have defined the 4 rules. As one of rules, there is ``The horizontal order of fingers 2 to 5 does not change.'' This rule can detect the non-detection of a marker and correct recognition errors of markers. We demonstrate how to apply this rule with the figure on the right. When the system was not able to detect the marker on the purple colored key, and then a user presses the key, this marker number was determined to be number five. Additionally, the system is able to correct miss color detection such as multiple markers and unsuitable marker order.

We conducted an evaluative experiment using three pianists and three trial pieces to look into the effectiveness of the prototype and the recognition rate. We compared the recognition rates while applying the rules verses not applying the rules. The average rate in the case of the rules is 95%, however the average rate in the case of not applying the rules is only 74%. Using musical rules, the system can recognize the fingering during the crossing of fingers, the simultaneous detection of two or more fingerings, fast tempo, and complex execution. Our system uses the simple image-processing method and corrects fingering with the above rules. Therefore, the proposed system can recognize the fingering in real time. The average processing time per frame is 20 msec, and the average frame rate of the camera that the prototype uses is 30 frames per second. The prototype system can complete the image processing within this frame rate.

This mechanism is useful for various interactive applications such as remote music lessons where teachers cannot check the details of fingering, a self-study support system which indicates fingerings, a performance education system which allows users to simultaneously learn the execution of playing with the fingering of a professional pianist.

 

More information

Takegawa, Y., Terada, T., and Nishio, S. ``Design and Implementation of a Real-Time Fingering Detection System for Piano Performances,'' Proceeding of International Computer Music Conference (ICMC2006), pp. 67--74 (Nov. 2006).PDF

 

Conventional piano practice support systems do not have a mechanism for evaluated fingering, and they do not enable learners to effectively learn fingering. In response to this problem, the goal of our study was to construct a piano practice support system that utilizes a fingering monitoring function using the above fingering recognition technique. We have proposed several methods to acquire information on piano performance such as fingering and keying information. The proposed system has a camera to detect fingering and a projector to present performance support information along the entire keyboard.

The learning support methods of this proposed system is explained in the above figure. The numerical values in the figure correspond to the following list numbers and the parenthetical numbers.

1. Each outline on the NextKeys (NextKey refers to the successive key that is pressed in a piece of music) provides keying information. The NextKeys are pointed out by arrow (1) in the figure.

2. The color of the outlines of NextKeys and the number presented on the NextKeys show fingering information. Additionally, the colors of the outline are the same as the colors of the markers attached to the finger nails.

3. When the correct finger is set on the NextKey, the whole area of the NextKey is filled in the correspoinding fingering color. The left NextKey is colored as shown in the above figure, because the correct fingering has been set. On the other hand, when the Next Key is pressed with incorrect fingering, the key is colored red. When NextKeys are not pressed, these keys are also colored red. Learners are able to not only understand the positions of NextKeys, fingering technique are learned easily and intuitively, but also rectify thire mistakes.

4. The outline of the CandidateKey, the second or third key pressed down, is shown in the fingering color assigned to the key, when the correct fingering is set. The CandidateKeys are pointed out by arrow (4), the outline of the leftmost CandidateKey is followed, because the correct fingering has been set. Learners remark that when the outline shows up on a key, the finger on the key does not need to move because of it being in proper position.

5. The keying sequence numbers are shown on the distal end of the NextKey and the CandidateKey. Additionally, the background color of the keying sequence number corresponds to the fingering color. The keys pressed simultaneously are connected with a horizontal line. Learners understand the flow of the keying positions and remark that they are able to simultaneously key widespread diapasons by this support.

6. The system presents a base line between the execution area of the right hand and that of the left hand.

7. The neighboring score based on the current execution position is shown above the keyboard. Each musical note is connected to the corresponding key with a line. Furthermore, since a NextKey line is bolder than that of a CandidateKey, the NextKey line is emphasized. This support enables learners to read a score easily, because the learner understands the relationship between the musical notes and the key positions intuitively.

8. The bars indicating the current execution position in the neighboring score are shown. This support helps learners read the keying timing of both hands easily from the score.

9. It is important for beginners, who try to learn a new score, to practice with one hand. The system has the scores for both hands or one hand, and the learner uses them selectively. Additionally, since it is important for the learner to be conscious of the relationship between the execution of each hand, even if the learner practices in one-handed mode, the musical notations for the other hand are also shown in a lighter shade. The right-handed score is shown in (9).

10. Pianists frequently cross the thumb and other fingers to achieve smooth execution. Since beginners do not understand either timing or method, the system announces the timing, and then shows the proper method.

We conducted an evaluative experiment with eight examinees to investigate the effectiveness of the proposed method. In this evaluation, we compared our method with the method Lighted Keyboard using a Lighted Keyboard from the viewpoints of ease of use and learn-ability on the basis of the number of keying error and fingering error. We presented the above execution support information in the proposed method, and presented only the red outline of NextKeys in the Lighted Keyboard method. We also presented the score that was added with fingering on all musical notations in the score.
Four examinees used the proposed method, and four examinees used the Lighted Keyboard method. All examinees belong to a graduate school of Information Science, and have no formal piano training, and are not able to read a score. The examinees practice ``Piano Sonata No. 11 in A major, K. 331: III (W. A. Mozart)'' whose range is from the beginning to bar 18 as the trial piece for one-handed playing. This examination comprised of two phases: practice and testing. During the practice phase, they practiced the trial piece for 15 minutes. They play the trial piece from the beginning to the end in the test phase after the practice phase. The system logged the number of keying errors and fingering errors during the test phase. In test phase, we presented only the score that was added fingering on all musical notations of the score.
The average number of keying/fingering errors in the case of applying the proposed method was 2.8/0.5, while the average number of keying/fingering errors in the case of the Lighted Keyboard method was 32.0/36.8. The results of the proposed method provided a significant enhancement in the learnability compared with the Lighted Keyboard method.
The examinees using the proposed method were able to comprehend the keying position and fingering, and able to read the score. On the other hand, because the examinees with the Lighted Keyboard method merely pursued the red colored outline displayed the key, they weren't able to read the score, and they didn't memorize the key positions. As a result, they made a lot of mistakes during the test phase. The reason that the Lighted Keyboard method has many fingering errors is that examinees were unable to check fingering and did not notice the mistakes.

We construct some applications to support keyboard execution using the characteristics that the proposed system is able to present various kinds of rich contents on the neighboring keyboard with a projector.

The icons which operate the displayed information of the proposed learning support system and the following application are displayed. The icons enable users to understand the function assigned each key easily.

The setting of hundreds of timbres is difficult and complicated in conventional musical keyboards, because the equipped buttons on musical keyboards have limitations. By integrating the keys of a musical keyboard and the visual expressive power of a projector, we achieve intuitive timbre setting. As initial step, we shows the timbre name on each key and users select a timbre by pressing a key displayed required timbre. Future work includes displaying not only timbre names but also the instruments icons to be able to bring up the image of instruments even if children.

This application controls the speed and the number of beat of the metronome by keying. Current beat is displayed at the distal end of the keys.

This application controls the color of the black keys.

This application presents sparkling bolls in synchronization with execution. The note name of pressed key is assigned to the color of the boll, and the velocity is assigned to the number of the ball.