Automated Guitar Scale
Performance Feedback

Raw guitar audio is converted into a Mel spectrogram — a 2D image where the X-axis represents time, the Y-axis represents frequency (warped to human pitch perception), and pixel brightness represents energy.

This single representation encodes all three performance features simultaneously: timing (horizontal position of onsets), tuning (vertical frequency alignment), and timbre (spectral texture and harmonic clarity).

Library: librosa · Format: .npy array

Each scale recording contains 8 consecutive notes. Segmentation detects where each note starts and ends by identifying sudden bursts of spectral energy — note onsets — in the spectrogram.

A two-phase approach is used: rule-based onset detection generates initial labels automatically, then a lightweight CNN onset detector is trained on those labels for robustness.

Output: 8 spectrogram patches — one per note — passed to the classifier.

Each of the 8 note patches is independently classified across three features using separate CNN models — one per feature. Each CNN takes a spectrogram patch as input and outputs a binary classification.

Keeping classifiers separate allows each CNN to specialize in its own visual pattern, enables independent debugging, and makes the system extensible.

Auto-labeling: tuning via librosa.pyin (cents deviation), timing via onset offset from metronome grid, timbre via spectral noisiness measure.

After classification, each note produces a feature vector [timing, tuning, timbre]. These 8 vectors are fed sequentially into an LSTM that maintains memory across the full scale — detecting patterns that are invisible at the note level.

Simple weight multiplication was explicitly rejected: two students with identical average scores can have completely different error patterns requiring different interventions. The LSTM learns to distinguish them.

Dataset strategy: 150 recordings split across 5 intentional error categories for balanced LSTM training.