Jordan Aiko Deja

Jordan Aiko Deja

I use the mouse with the left hand.

Piano Learning and Improvisation Through Digital Augmentation

Overview

Over the centuries, the piano has undergone various transformations. Still, several innovations have not prioritized the user, resulting in limited interactivity and frustration during the learning process. Digital augmentation, particularly Augmented Reality (AR), has emerged as a solution to enhance learning and user experience. In this study, I explore piano learning through an AR lens, with particular emphasis on improvisation, which holds potential for enrichment and application in related domains. The contributions of this thesis include 1) a survey of existing augmentations and recommendations for future piano interface design, 2) a proposed design of the augmented piano learning space informed by our findings in our survey, 3) the development of an AR piano training system for improvisation through a co-design process with experts, and 4) empirical findings evaluating our AR piano training system and the interaction metaphors we designed. Furthermore, this work aims to generalize insights gained from our contributions and a speculative workshops to inform the use of AR on improvisation in broader applications such as dance or guitar. Overall, this thesis demonstrates how AR interfaces can enable interactive and engaging learning experiences with musical instruments.

Goals and Targets

In this PhD we have formulated the following sub research questions to guide our method (UMAP DC 2021, MobileHCI DC 2021):

  • RQ0: What does the current state-of-the-art of augmented pianos teach us about piano learning? ISS 2022 PACM HCI
  • RQ1: How do we inform the design of future augmented piano training systems? IMI 2022
  • RQ2: How do we build an AR Piano Improvisation Training System using Co-Design? HCI SI 2023.
  • RQ3: Do augmented visualisations help piano learners improvise? ONGOING
  • RQ4: How do we generalise improvisation to other similar domains? ONGOING

#### Contribution 1: Survey of the Augmented Piano Space

#### Contribution 2: Proposed Design Space for Augmented Piano Training Systems

To know more details in this dissertation, we invite you to read our existing publications (see below, PDF-available):

  • Adaptive Visualisations Using Spatiotemporal and Heuristic Models to Support Piano Learning. UMAP DC 2021. PDF
  • Encouraging Improvisation in Piano LearningUsing Adaptive Visualisations and Spatiotemporal Models. MobileHCI DC 2021. PDF.
  • The Vision of a Human-Centered Piano. IMI 2022. PDF.
  • A Survey of Augmented Piano Prototypes: Has Augmentation Improved Learning Experiences?. (ISS 22). PDF.
  • Piano Learning and Improvisation Through Adaptive Visualisation and Digital Augmentation. (ISS DC 22). PDF.
  • Teach Me How to ImproVISe: Co-Designing an Augmented Piano Training System for Improvisation. HCI-SI 2023. PDF.