Sensory substitution

Sensory substitution is the principle to transform characteristics of one sensory modality into stimuli of another sensory modality. It is hoped that systems that base on sensory substitution can help handicapped people to restore the ability to perceive a certain sensory modality.

A sensory substitution system consists of three parts: a sensor, a coupling system, and a stimulator. The sensor records stimuli from one sensory modality and gives them to a coupling system which interpretes these signals and transmits them to a stimulator.

The research on sensory substitution raises many questions concerning human perception and the plasticity of the human brain.

Applications

Applications are not restricted to handicapped persons, but also include artistic presentations, games, and augmented reality. Some examples are substitution of visual stimuli to audio or tactile, audio to tactile. Most popular are probably Paul Bach-y-Rita's Tactile Vision Sensory Substitution (TVSS), developed with Dr. Carter Collins at Smith-Kettlewell Institute and Peter Meijer's Seeing with Sound approach (The vOICe).

Technical developments, such as miniaturization and electrical stimulation help the advance of sensory substitution devices.

Tactile Vision Sensory Substitution

The TVSS converts the image from a video camera into a so-called tactile image. The tactile image is produced by four hundred activators placed either on the back, on the chest, or on the brow. The activators are solenoids of one millimeter diameter.

In experiments, blind (or blindfolded) subjects equipped with the TVSS can learn to detect shapes and to orient themselves. In the case of simple geometric shapes, it took around 50 trials to achieve 100 percent correct recognition. To identify objects in different orientations requires several hours of learning.

"Seeing with sound" sensory substitution

The vOICe vision technology is one of several approaches towards sensory substitution (vision substitution) for the blind that aims to provide synthetic vision to the by means of a non-invasive visual prosthesis. The vOICe converts live camera views from a video camera into a soundscape. Views are typically refreshed about once per second while associating height with pitch and brightness with loudness in a left-to-right stereo scan of every camera snapshot. The effective image resolution in sound is up to several thousand pixels as can be proven by spectrographic analysis.

The ultimate goal is to provide synthetic vision with truly visual sensations by exploiting the neural plasticity of the human brain. Neuroscience research has shown that the visual cortex of even adult blind people can become responsive to sound, and seeingwithsound might reinforce this in a visual sense with live video from a head-mounted camera encoded in sound. The extent to which cortical plasticity indeed allows for functionally relevant rewiring or remapping of the human brain is still largely unknown and is being investigated in an open collaboration with research partners around the world.

The core behind this vision substitution system is a piece of software that converts the vertical position and brightness within a scanned source image into a sound wave that has a corresponding modulated frequency and amplitude. Other approaches to the substitution of hearing for vision use binaural directional cues, much as natural human echolocation does. An example of the latter approach is the "SeeHear" chip from Caltech.

Criticism

It has been argued that the term "substitution" is misleading, as it is merely an "addition" or "supplementation" not a substitution of a sensory modality.

See also

• Biological Neural Networks
• Brain implant
• Human echolocation, blind people navigating by listening to the echo of sounds

References

Additional sources

• Sensory Substitution- Limits and Perspectives

External links

• Tongue display for sensory substitution
• The vOICe auditory display for sensory substitution.
• Webpage of Laurent Renier
• Artificial Retinas