Sensory Integration

Every time Tim sees the letter “E,” he also sees the colour red. He describes the colour change as if suddenly forced to look at the world through red-tinted glasses. When Tim looks away from the letter “E,” his world returns to normal, until he encounters the letter “O.” Then the world turns blue. For Tim, reading a book is like living in a disco... 

For a long time, Tim thought this happened to everyone. When he discovered this happened to no one - at least not in his immediate circle - he began to suspect he was crazy. Neither impression was correct, of course. Tim is suffering - if that’s the right word - from a brain condition called synaesthesia. Though experienced by as many as 1 in 2,000 people (some think 1 in 200), it is a behaviour about which scientists know next to nothing. At first blush, there appears to be a short-circuiting between the processing of various sensory inputs. If scientists can nail down what happens when sensory processing goes wrong, they may gain more understanding about what happens when it goes right. So, synaesthesia intrigues scientists interested in how the brain processes the world’s many senses. The effect that this has on learning forms the heart of a key Brain Rule: Stimulate more of the senses at the same time.

Saturday night fever

That you can detect anything has always seemed like a minor miracle to me. On one hand, the inside of your head is a darkened, silent place, lonely as a cave. On the other hand, your head crackles with the perceptions of the whole world, sight, sound, taste, smell, touch, energetic as a frat party. How could this be? For a long time, nobody could figure it out. The Greeks didn’t think the brain did much of anything. It just sat there like an inert pile of clay (indeed, it does not generate enough electricity to prick your finger). Aristotle thought the heart held all the action, pumping out rich, red blood 24 hours a day. The heart, he reasoned, harboured the “vital flame of life,” a fire producing enough heat to give the brain a job description: to act as a cooling device (he thought the lungs helped out, too). Perhaps taking a cue from our Macedonian mentor, we still use the word “heart” to describe many aspects of mental life.

How does the brain, brooding in its isolated bony chambers, perceive the world? Consider this example: It is Friday night at a New York club. The dance beat dominates, both annoying and hypnotic, felt more than heard. Laser lights shoot across the room. Bodies move. The smells of alcohol, fried food, and illegal smoking mix in the atmosphere like a second sound track. In the corner, a jilted lover is crying. There is so much information in the room, you are beginning to get a headache, so you step out for a breath of fresh air. The jilted lover follows you.

Snapshots like these illustrate the incredible amount of sensory information your brain must process simultaneously. External physical inputs and internal emotional inputs all are presented to your brain in a never-ending fire hose of sensations. Dance clubs may seem the extreme. Yet there may be no more information there than what you’d normally experience the next morning on the streets of Manhattan. Faithfully, your brain perceives the screech of the taxis, the pretzels for sale, the crosswalk signal, and the people brushing past, just as it could hear the pounding beat and smell the cigarettes last night. You are a wonder. And we in brain-science land are only beginning to figure out how you do it.

Scientists often point to an experience called the McGurk effect to illustrate sensory integration. Suppose researchers showed you a video of a person saying the surprisingly ugly syllable “ga.” Unbeknownst to you, the scientists had turned off the sound of the original video and had dubbed the sound “ba” onto it. When the scientist asks you to listen to the video with your eyes closed, you hear “ba” just fine. But if you open your eyes, your brain suddenly encounters the shape of the lips saying “ga” while your ears are still hearing “ba.” The brain has no idea what to do with this contradiction. So it makes something up. If you are like most people, what you actually will hear when your eyes open is the syllable “da.” This is the brain’s compromise between what you hear and what you see—its need to attempt integration.

But you don’t have to be in a laboratory to show this. You can just go to a movie. The actors you see speaking to each other on screen are not really speaking to each other at all. Their voices emanate from speakers cleverly placed around the room: some behind you, some beside you; none centered on the actors’ mouths. Even so, you believe the voices are coming from those mouths. Your eye observes lips moving in tandem with the words your ears are hearing, and the brain combines the experience to trick you into believing the dialogue comes from the screen. Together, these senses create the perception of someone speaking in front of you, when actually nobody is speaking in front of you.

How the senses integrate

Analyses like these have led scientists to propose a series of theories about how the senses integrate. On one end of this large continuum are ideas that remind me of the British armies during the Revolutionary War. On the other end are ideas that remind me of how the Americans fought them. The British, steeped in the traditions of large European land wars, had lots of central planning. The field office gathered information from leaders on the battleground and then issued its commands. The Americans, steeped in the traditions of nothing, used guerrilla tactics: on-the-ground analysis and decision making prior to consultation with a central command.

Take the sound of a single gunshot over a green field during that war. In the British model of this experience, our senses function separately, sending their information into the brain’s central command, its sophisticated perception centers. Only in these centers does the brain combine the sensory inputs into a cohesive perception of the environment. The ears hear the rifle and generate a complete auditory report of what just occurred. The eyes see the smoke from the gun arising from the turf and process the information separately, generating a visual report of the event. The nose, smelling gunpowder, does the same thing. They each send their data to central command. There, the inputs are bound together, a cohesive perception is created, and the brain lets the soldier in on what he just experienced. The processes can be divided into three steps:

Step 1: sensation

This is where we capture the energies from our environment pushing themselves into our orifices and rubbing against our skin. The effort involves converting this external information into a brain-friendly electrical language.

Step 2: routing

Once the information is successfully translated into head-speak, it is sent off to appropriate regions of the brain for further processing. The signals for vision, hearing, touch, taste and smell all have separate, specialized places where this processing occurs. A region called the thalamus, that well-connected, egg-shaped structure in the middle of your “second brain,” helps supervise most of this shuttling.

Step 3: perception

The various senses start merging their information. These integrated signals are sent to increasingly complex areas of the brain (actually called higher regions), and we begin to perceive what our senses have given us. As we shall see shortly, this final step has both bottom-up and top-down features.

The American model puts things very differently. Here the senses work together from the very beginning, consulting and influencing one another quite early in the process. As the ear and eye simultaneously pick up gunshot and smoke, the two impressions immediately confer with each other. They perceive that the events are occurring in tandem, without conferencing with any higher authority. The picture of a rifle firing over an open field emerges in the observer’s brain. The steps are still sensation, routing, and perception. But at each step, add “the signals begin immediately communicating, influencing subsequent rounds of signal processing.” The last stage, perception, is not where the integration begins. The last step is where the integration culminates.

Which model is correct? The data are edging in the direction of the second model, but the truth is that nobody knows how it works. There are tantalising suggestions that the senses actually help one another, and in a precisely coordinated fashion...

(This article has been adapted from John's book, Brain Rules)