14 hours ago
3D Sensing & AI Give Operatic Voice To Non-Verbal Performers
MELBOURNE, AUSTRALIA - JULY 26: Anthony Warlow and Ana Marina, cast members of The Phantom of the ... More Opera perform on stage at the photo call for the new production of Broadway's longest running musical "The Phantom of the Opera" at the Princess Theatre on July 25, 2007 in Melbourne, Australia. (Photo by)
Ever wonder what it takes to deliver an opera performance that is historic in terms of longevity, quality and audience numbers? Talented performers, money, advertising, writing, music, etc. It is a process that has been followed for more than a century, and is fairly predictable and linear.
Now imaging what it takes to deliver an opera performance about a mother and her non-verbal, non-ambulatory son? With a cast composed of predominantly disabled performers, many with impaired or limited speech? Dedication, time, focus, persistence, creativity, technology, philanthropic support, vision AND innovation. Over a 7 year period, librettist Brenda Shaughnessy and composer Paola Prestini, directed by Jerron Herman and Jay Scheib, created Sensorium Ex, culminating in a premier performance in Omaha, Nebraska in May 2025. It fuses 3D sensing, artificial intelligence, disability solutions, operatic talent and artistic creativity to deliver a groundbreaking performance.
Dr. Luke Dubois is the Associate Professor of Integrated Design & Media (IDM) and Co-Director of the Technology, Culture and Society Department at New York University's Tandon School of Engineering. This is the only department in the country that is part of an engineering school AND accredited by the National Academy of Design. He is also the Co-Director of the Ability Project, a joint research initiative with the Occupational Therapy Department (NYU Steinhardt) and the Interactive Telecommunications Program (NYU Tisch). It supports research in assistive technology and adaptive design for people with disabilities. Dr. Dubois has a Ph.D. in music and a strong background in electronics, speech synthesizers and solutions that fuse art, engineering and design. His laboratory/office at NYU Tandon is reminiscent of a music studio where you can easily imagine legendary bands like Pink Floyd, Led Zeppelin, Jethro Tull and the Boss recording (Disclosure: I am a fan).
Figure 1: Luke Dubois in Lab, DIY (Do It Yourself) Kit Built Synthesizers, Historic Guitars
Speech synthesizers as human aids (also called AAC or Augmentative & Alternative Communication devices) were originally invented by Ray Kurzweil in 1975. They used OCR (Optical Character Recognition) to assist the blind or visually impaired to 'Listen' to written text. The next evolution in this technology occurred with Intel and Stephen Hawking who used keyboard text input input and eye/gaze movement to communicate via a speech synthesizer. It worked and helped at the time, but the output was, well, robotic. This is a problem for something like the opera where nuance, emotion and artist personalization are key attributes (based on a survey of ~40 such artists). Dr. Dubois' collaboration with Sensorium Ex started 5 years ago, with a request from Paola Prestini to develop the technology to enable speech challenged artists to perform in the opera.
Solution Strategy: Dr. Dubois and his team came up with a two-step process:
1. Personalizing Speech Using AI: some people with cerebral palsy have a condition called apraxia, in which the person mentally feels that they are speaking normally, although it vocalizes as jumbled speech. Using a combination of speech editing software (ProMoNet) and a neural vocoder, these sounds can be transformed into coherent speech (but still robotic). NJIT (New Jersey Institute of Technology) Professor Mark Cartwright uses these tools in conjunction with machine learning and AI to inject the individual's sound and speech characteristics. This humanizes and personalizes the robotic speech created by the ProMoNet and the neural vocoder tools.
2. Injecting Individual Creativity and Emotion into the Opera Performance: the challenge is to enable the performer to insert emotion, tempo, emphasis and vocal stretch. People affected by cerebral palsy have control of certain types of hand movement (some macro and others micro). Using their hands to control the different nuances of their operatic speech in real time is a challenge. Enter Eric Singer, a Brooklyn, NY based musician/engineer/entrepreneur, self proclaimed mad scientist and owner of Singertronics. He designed and built the Ther'minator (Figure 2), which transforms LiDAR (Light Detection and Ranging) sensing signals into modular voltages (between 0-5 volts), and transmits it using WiFi. The name is a spin on the Theremin, an electronic musical instrument controlled without physical contact by the performer.
Figure 2: Therm'nator Time of Flight Sensor (Center area is the STMIcro VL53 LiDAR Chip)
Ther'minator uses very low cost (< $10) Model VL 53 Time of Flight (ToF) sensors from STMicro . These sensors are composed of 940 nm SPADs (Single Photon Avalanche Detectors) and VCSELs (Vertical Cavity Surface Emitting Laser), integrated with optical noise filters, short pulse laser drivers and readout electronics. Their performance is suitable for short range 3D sensing (< 5 m), and typically used in automobile airbags to sense human presence. Higher power versions are used in smart phones.
Lets Omaha: Using a combination of three Ther'minator units, with other electronics for power and signal processing converts the performer's hand movements into output electrical signals (see Figure 3). This board was designed by Andrea Lauer, the production designer for Sensorium Ex.
Figure 3: Sensor and Electronics Board for Sensorium AI Voice Control Showing 3 Ther'minators
The signals are fed into a computer that integrates the LiDAR output voltages with the performer's personalized speech (through the AI trained algorithm discussed earlier). A calibration is performed for each individual since arm movements and their correlation with the performer's vocal intent vary.
Figures 4 shows Jakob Jordan (playing the lead role of the son, Kitsune, in Sensorium Ex) practicing with the sensor board and in dress rehearsals.
Figure 4: LEFT: Performer Jakob Jordan Practicing with the Ther'minator Board at the Sensorium Ex ... More Production Residency, April 2025; RIGHT: Sensorium Ex dress rehearsal at Sapp Fieldhouse 5-20-25
The Omaha premier held 4 shows with about 150-200 attendees in each. Seating was configured to accommodate a spectrum of disability conditions. Paola Prestini, the composer of Sensorium Ex, also leads National Sawdust and Vision Into Art, both non-profit organizations dedicated to promoting and supporting art across various genres of performers (Vision Into Art sponsored Sensorium Ex). I spoke to her about the 7 year journey for Sensorum Ex.
Future Directions
This effort is a monumental disruption in the fusion of art and technology. How will it impact speech-challenged artists in the future? Will mainstream productions like the Phantom of the Opera or Hamilton incorporate such talent in their productions, with higher levels of assistive performance enhancements? Can the technology demonstrated here be extended to other quality of life enhancements like physical therapy and rehabilitation? A few thoughts:
Using technology for assisting physically challenged individuals is an accelerating trend. Robotics, 3D sensing, AI, speech enhancement and movement automation technologies can dramatically elevate quality of life for all humanity.
