From eyes that see over a million different colours to ears that detect a staggering range of frequencies, the human senses are incredible. But none can hold a candle to the most powerful part of our sensory system: the brain.
The human brain does the most impressive feat of data analysis imaginable – decoding electrical impulses from all over the body and combining them to create the rich, multi-modal experience we call reality. But what if a connection fails, and a sense is weakened or lost? That’s where sensory substitution comes in.
Watch my short video to find out more…
Sense and switchability
What does yellow taste like? On the face of it this question seems illogical. But, for some people, tasting in colour is a real phenomenon.
Synaesthesia is a neurological condition where one sense triggers another. Sounds elicit shapes and colours; words can trigger smells. Legendary physicist Richard Feynman once described how he saw equations in yellows, blues and browns. Singers and composers including Beyoncé, Billy Joel, Thom Yorke and Hans Zimmer don’t just hear music; they experience it in shapes and colours too.
The cause of synaesthesia is relatively unknown, and there isn’t even a consensus on how many people experience it.
Some experts believe that all of us are synesthetic to some degree. Take the famous Bouba-Kiki effect. If I told you that one of the shapes below was called Bouba and the other Kiki, which name would you say goes with which shape?
This question makes about as much sense as asking what yellow tastes like. Any association between abstract shapes and made-up words is purely arbitrary, right?
But if you played along anyway and said that Kiki was the angular shape on the left, then you’re in good company.
This experiment was devised almost a century ago and it has been tried out on people all over the world. It turns out that, regardless of culture, language or writing system, the overwhelming majority of people answer the same way. This suggests that most of us have at least some inherent tendencies to associate speech sounds with certain shapes.
Beyond the curious phenomenon of synaesthesia, the concept of deliberately swapping one sense for another has a long history. The origins of sign language go at least as far back as ancient Greece, while braille has been helping blind people to read since the early 1800s.
These early forms of sensory substitution utilise the brain’s impressive ability to adapt and switch inputs when required. Today, this idea is being supercharged by technology. From using fingers to read and hands to speak and listen, we now have the potential to open up whole new ways of sensing reality.
Seeing in tongues
“Just give the brain the information and it will figure it out.” – Paul Bach-y-Rita, inventor of the BrainPort
Art student Emilie Gossiaux was hit by a truck while cycling in New York City in 2010. She suffered a traumatic brain injury and never recovered her sight.
During a two-year recovery period, Emilie was one of the first people enrolled into a study exploring the use of a revolutionary new device. The BrainPort converts images picked up by a head-mounted camera into a series of small vibrations which are relayed to a piece of metal that sits on the tongue.
Its inventor, Paul Bach-y-Rita, argued that we see with our brain, not our eyes. He believed that if we could just find another way to get visual information to the brain, it would work out what to do with it. Successful trials of the BrainPort proved that he was correct – the brain really is capable of decoding visual data from another source.
In the clip below, Emilie describes how this technological breakthrough helped her to draw again.
Blind mountaineer Erik Weihenmayer, another early user of the BrainPort, described the sensation as “pictures being painted with tiny bubbles”. He has used the device on expeditions and has movingly described how it enabled him to see his son smile for the first time.
Another device, called the vOICe, converts images into sounds, using pitch to illustrate the elevation of objects and volume to convey brightness. Unlike the BrainPort, this method is less reliant on expensive specialist equipment. A version of vOICe is freely available as an app that can be used with most smartphones.
Wristbands like the Neosensory Buzz and WearWorks Wayband are enabling people to feel sounds and navigate directions through discreet vibrations. The latter device was worn by ultramarathon runner Simon Wheatcroft when he attempted to become the first first blind person to run the New York marathon unassisted.
The sixth sense?
As great as our biologically-evolved human senses are, there’s a lot that they miss. Everything that we can see, hear, smell, taste and touch amounts to only a tiny fraction of the world around us.
We only need to look to the animal kingdom to get a sense of just how much is happening beyond the confines of what human senses are able to detect.
In his book An Immense World, Ed Yong describes how every species is trapped in their own ‘sensory bubble’, able to detect only a small portion of their surroundings.
Birds and sea turtles can sense Earth’s magnetic field; octopuses can see the polarisation of light waves; bees and reindeer can see ultraviolet light, and some snakes can detect infrared. Sharks and platypuses can detect electricity, while cows have 360-degree vision.
Our human senses only give us a glimpse of reality. What if sensory substitution technology could help us to experience the rest of it? Gadgets like radio receivers, GPS and LiDAR could be used to give us new sensory superpowers. Maybe we’ll one day be able to ‘see’ through walls using infrared, and ‘hear’ the location of our friends and family using GPS data.
Neuroscientist David Eagleman believes this is exactly what the future has in store. Our perception of reality need no longer be ‘constrained by our biology’. We may soon be able to change the experience of being human.
So, how might this work?
Speaking to New Scientist in 2015, Eagleman said:
“…the brain is really flexible about what it can incorporate into its reality. It receives information in the form of electrochemical signals from our eyes, our nose, our skin, and works out meaning from them. Crucially, it doesn’t care where these signals are coming from; it just figures out how to use them.“I think of the brain as a general-purpose computer. Our senses are just plug-and-play devices that we have inherited through evolution. And if that’s the case, we should be able to interface any data stream into the brain and it will figure out how to deal with it.” =
In the TED Talk above, Eagleman demonstrates his company Neosensory’s own sensory substitution device – the Versatile Extra-Sensory Transducer (or VEST for short). This waistcoat is fitted with 32 vibrating motors which, using a smartphone app, convert sounds into vibrations. It was designed to help deaf people to hear, but Eagleman doesn’t want to stop there. If the brain can decode these tactile sensations back into speech and other sounds, then why not other types of data too? Might brokers make better decisions if they could ‘feel’ stock market fluctuations using a live feed from the internet? Could meteorologists ‘feel’ patterns in the weather to make more accurate forecasts?
Senses working overtime?
The BrainPort has been revolutionary in how it has reshaped our understanding of the human brain, but in practical terms it is still just a novel – and very expensive – curiosity. A future where we power ourselves up every morning with a bespoke selection of new super-senses is probably still a long way off.
Erik Weihenmayer, who we met earlier, was the first blind person to climb Mount Everest. Following this extraordinary feat, he has subsequently led expeditions of blind Tibetan teenagers and then injured veterans part way up the world’s highest mountain (these Himalayan adventures have been documented in the award-winning films Blindsight and High Ground). It’s worth pointing out that the BrainPort wasn’t used on either.
Thirteen years after her life-changing accident, Emilie Gossiaux is today a successful and award-winning artist. She no longer uses the BrainPort though. Describing her experience with the device, she said:
“The images I saw were really grainy, and I couldn’t see details, but I could see high-contrast black and white. So I was drawing with a thick black marker on plain white paper. That got me back into the feeling of using a pen in my hand and seeing marks. It helped me build up my muscle memory. But it ended up being really exhausting, so I decided to stop using my vision when I draw.”
Sensory substitution may one day have the capacity to extend the horizons of human perception. But before then, there’s still a lot more we can do to help the greatest number of people overcome obstacles to socialising, working, and getting around in a world that all too often doesn’t take account of their needs.
Recommended links and further reading
Sensory Substitution (Eagleman.com) Links to David Eagleman’s TED Talks, books and research.
Paul Bach-y-Rita and Neuroplasticity (YouTube) A short 10-minute documentary which explains the workings of the BrainPort and includes an interview with Bach-y-Rita, the ‘father of sensory substitution’.
Seeing with Your Tongue (The New Yorker) This 2017 article includes an interview with mountaineer Erik Weihenmayer and the story of how the BrainPort was invented.
An Immense World - Ed Yong The Pulitzer Prize-winning author’s 2022 book on the extraordinary senses of animals.
Comments