Using different senses to read data

An interesting article appeared in The Guardian that talks about how to help visually impaired people read data, The technique is called sonification, using sound to represent data:

It reminded me of this video which shows data from the landing of the Huygen’s probe on Titan in 2005:

You can find out more about the probe here: https://solarsystem.nasa.gov/missions/cassini/mission/spacecraft/huygens-probe/

The importance of the senses

We use sound to communicate directly with each other (speech, clapping, laughing), learn about our environment (kettle whistling, closed/open space, weather), and to create and listen to music.

Sight is also used to to convey the same and different kinds of information that can supplement sound (body language, looking at a kettle or out of a window, reading text and musical notation) but, as with sound, there are some things that exist exclusively in that domain that doesn’t naturally translate to sound, or vice versa except, perhaps, in our imagination. Here’s a guy who was only able to see grayscale colours but can now “see” colour using sound: https://en.wikipedia.org/wiki/Neil_Harbisson

The same goes for pressure, and all the other senses that combine to give us such a rich experience of the world we live in.

The great thing is that we have technology that can translate these effects to different sensory domains to create an even richer experience, even for things that are invisible. One example of the latter is the clicking of a geiger counter. Another is x-ray photography. The video above shows how we can assimilate different pieces of data at the same time; by translating sensory data from the probe to numbers streamed to receivers here on earth to a visual display of the data which has been sonified. How cool is that?

I am particularly interested in sound and I want to briefly explore the reverse of sonification – representing sound using our other senses – in the context of speech which can be a problem for people like me with hearing impairment.

Speech

With only a high-frequency hearing loss I am very lucky to be able to understand speech in most situations. Areas of difficulties are movies, tv shows, podcasts and radio broadcasts that have background noise (I include music here). The same goes at concerts and other public places, like pubs (see what I did there?).

This also affects how I speak. Take the words “shirt” and “church”. To me, in a natural environment and not looking at the speaker, those words sound identical. I can hear vowels (a, e, i, o, u and variants thereof) very well, it’s the consonants I have difficulty with. The fact that “i” and “u” are pronounced exactly the same way in this example compounds that difficulty. This is what the words sound like to me:

undefined sound + ur + undefined sound

Of course this meant I talked like that, even with a hearing aid, so I had speech therapy. Speech therapy consisted of using my hands to sense what my vocal chords and mouth were doing and lots of diagrams showing a cutaway of a head from the side so I could see the positions my tongue and mouth needed to form to properly pronounce letters and words That helped a lot. My speech makes me sound a bit drunk sometimes, mainly in the mornings or when I’m tired. This has been exacerbated by surgery on my tongue but people understand me when I’m talking which is important for confidence as well. Oh, and even in Spanish (216 days of continuous learning and 48115 XP) and despite learning Latin-American Spanish and not European Spanish. I digress.

Context is important. If someone says “I went to church on Sunday” I’m unlikely to mistake it with “I went to shirt on Sunday”. I don’t think anyone “put on a church” either.

I’m also pretty useless at lip-reading but it does help a lot to see someone with their face uncovered speaking directly in front of me. I can’t read lips like HAL.

I can’t read lips like HAL

Other aspects of body language also adds to the auditory experience; posture, eyes, hands and arms, even clothing. I will pay more attention to a police officer in uniform telling me to move on than someone in casual wear.

But what about the sounds I can’t hear? The average human being will hear frequencies between 20Hz and 20kHz. Here’s a nice graph showing the limits of sound perception and threshold:

hearing limits

The graph is showing that we can hear frequencies around 1-2kHz very well with perception tapering towards the upper and lower frequencies. Very low and very high frequencies need to be amplified to be on par with the middle frequencies for the purposes of music performance and recording, for example.

Sounds below 20Hz are classified as infrasound, which mole-rats and elephants can hear very well. Sounds above 20kHz are classified as ultrasound. Cats and dogs can hear up to 40kHz, bats and dolphins can hear up to 160kHz (I know, right?).

Humans can sense infrasound but with other senses, such as balance, pressure, and so on. Ultrasound can also be perceived through bone conduction.

So, how do I get to more information from speech? Here’s a brief summary of some techniques.

Subtitles

This is a very obvious one, and a technology that many people become increasingly dependent on as they get older. Real time captions (another word to describe it) are getting better too. Useful for learning languages too.

Lowering frequencies

One technique is to lower frequencies so that sounds at 20kHz can be heard at 10kHz for example, but there are problems with this.

Haptics

This is something I would like to try. A technology that uses the human skin as a receptor by using vibration and pressure to represent sound. I can imagine this would help me with my own pronunciation and enunciation. Here’s a guy who climbed Everest using BrainPort – haptic technology that utilises the tongue.

There are many other applications for haptics; prostheses, sensory augmentation, gaming, even sex!

That’s it for today. Adios!