My Greatest Achievement with Sensory Substitution

Those of you following this blog may be wondering why I haven't been posting for the last couple of months. Part of the reason is the fact that I've been so bogged down with school. The study of optics can be a real trip if you were born not seeing. Balancing my responsibilities at school with the demands of driving lessons has been a challenge, but I'm managing nicely. Below is an article posted today in the local Mensa news letter. This publication often includes the latest medical and scientific breakthroughs of local members.
The text is below:

Amanda and Dave met last fall at a Mensa meeting.  Amanda was born totally blind but thanks to an innovative free program called “the vOICe” (available at seeingwithsound.com) Amanda is now able to convert small black-and-white images taken with a digital camera into sound files and for the first time “see”  the shape, outline and texture of objects.

The system is very crude so Dave began to work with Amanda looking at ways to improve the program.  Initially Amanda had limited skills when it came to visualizing 3-D shapes.  Then Dave built her a box illuminated with Christmas lights along the edges(picture 1, 7337).


  The box and the numerous outdoor Christmas light displays they visited at Christmas allowed Amanda to make rapid progress in understanding the “sound” of particular three dimensional spaces and objects.

Soon Dave realized that Amanda's acute sense of hearing combined with the vOICe system opened up the possibility that she might be able to “see” enough to drive a car.

They began to experiment.  Because Dave’s car has a manual transmission, the first step was teaching Amanda how to use the clutch and gearshift while guiding the car (very, very slowly) using the traditional “tap tap tap” of the white cane (picture 2, 7372).


  She proved to be a natural at coordinating the clutch and the transmission but the cane system didn’t work very well at all for the “where to aim the car” part; for example, she simply had to hit the brake and stop when an Austin Transit bus came up next to her and made too much noise for her to hear the tapping of the cane (picture 3, 7374).


 Of course Amanda doesn’t have a learner’s permit, so much to Amanda’s annoyance Dave insisted that they move the next step to the Texas School for the Blind parking lot, where they began to practice with a camera on the dashboard and the vOICe interface (picture 4, 7376).


When the car is actually moving Dave is in the passenger’s seat.  There are still huge problems; Amanda can see high contrast scenes best, so daytime driving is a problem and they have not yet come up with a solution that allows Amanda to use mirrors.  And a late-night I 35 experiment almost turned into a disaster.  Traffic around Austin is getting much, much worse.

So far the State of Texas is  unwilling to make reasonable accommodations and issue a learners permit to a totally blind beginner, but the recent proposal to allow Google’s unmanned vehicles to be tested on Texas roads opens up the possibility that Amanda might qualify under an unmanned vehicle provision, a good start but clearly not the sort of accommodation that the Americans With Disabilities Act will eventually require as vOICe-like programs improve.  We’ll keep you posted.

A Picture from New Year's Eve

I am trying to learn about 3D shapes and finding it more difficult than I could have imagined. Starting on New Year's Eve, I've examined this cube from many angles in order to construct a mental 3D representation.

This picture is of me contemplating a black cube whose edges are outlined in Christmas lights. It is a cardboard box. Each dimension is a different length from the other two, and it is covered in black paper. Along every edge, at about one-inch intervals, is a hole with a light sticking out of it, so in a dark room I can only see the lit edges. For users of the vOICe, the cube is in the lower-left portion of the image. It is a little less than half the width of the image and a little more than half the height. I am on the far right side. The watch on my left arm, which is hanging down at my side, is at the bottom and the top of my head is at the top. My width is about forty percent of the right side of the picture. To me, the most noticeable things are the cube which I can recognize using zoom, the human-type shape which must be me, and the reflective watch near the right. There is also some stuff in the middle which is probably a table with papers on it. Thanks to Prof. Baldwin for cropping the picture and for his excellent descriptions. Also Thanks to Dave, the clever one who assembled the cube.

First Light

First light, so I am told, is a ceremony held when an observatory dome is first opened, and the first star is visible through its telescope. Perhaps I'll get to see something like that one day. For now, I am enjoying Christmas lights. My friend showed me this small set and then told me that there are certain streets where I live where these will soon cover everything. What a shocking idea. This picture was taken through a window at night time. I never knew something so small could cause two people to laugh so much.

After my friend went to bed, I stayed up flipping the Christmas lights on and off, on and off, as could only someone seeing them for the first time. Then I started playing with the speed setting on the program, changing it from sweeps of one second to half a second, then to two seconds and back to one. After at least an hour of looking at these lights at different distances, angles, and speeds, I took the glasses off. The transcendent sounds were still playing in my head, only now there were these pin pricks of something in front of me. They were caused by the sounds but separate from them. I imagined more and more of them all around me and tried to picture them in configurations like a helix or a seemingly infinite binary tree. It took me great effort and lots of pacing to get calm as the tiny apparitions faded away. I was finally rewarded with peaceful sleep.

I told my folks.

I finally told my parents about the glasses. They live five hours away in the small town where I grew up. After they made sure I got a good education through high school, I came to the city, got into trouble at the blind rehab center, and was hardly to be seen in that small town again. I know they are proud of me for doing well in school, but I wonder how many interests we have in common. If I give Mom a lengthy explanation of one of my projects, she might say I'm talking over her head. She is about five inches taller than me and actually means that she cannot understand all of what I am saying. Likewise, if she talks about shopping, decorating etc. I, being her only girl, have no idea what to say. I've always been the strange one that everybody picked on as I spent my childhood alone, waiting for my own kind to show up in a star ship and beam me out of there.
If I tell them, I thought, they will either think me even more bizarre than before, or worse, that I am investing my hopes in something that is a waste of time.

As I explained the basics of the system to Mom, as I told her that I was starting to see, it felt hard to breathe. "I'm starting to understand why you do things like hang balls on Christmas trees," I explained. "Before I could have no empathy. I had to dismiss so many rituals as stupid sighted things, but now they are legitimate and I can study them. I can't see all of what you see, but I want to learn."
She seemed excited and had me tell Daddy. It suddenly became easier to breathe. I was able to tell him that the study of vision had become extremely important to me and that I wanted to study science and technology so that many people would be able to see one day. We had a long discussion. "Keep tinkering," he said.  "Remember Thomas Edison and the light bulb, and Don't get discouraged if it doesn't work out right away. You never know where this will lead in a few decades."

It's going to be a very interesting Christmas. Mom has at least three dogs and a huge collection of mercury glass ornaments to show me, and Daddy wants to hear the soundscapes - he is late blind himself. Maybe I will no longer have to sit outside by myself, wishing for home.

An Age-Appropriate Experiment

This week I acquired a lego set for use with the vOICe vision sysstem. The reasoning behind this experiment is as follows:
It's still very hard for me to recognize 3D objects in the real world which might have an irregular shape and might not be composed of only one color. My ability to visualize many everyday objects is only through touch. What will happen if I practice building simple 3D shapes with legos and viewing them in stages? In real life, will this eventually translate into an ability to recognize new types of objects and visualize tasks without relying so heavily on storing them in handspace? Below are some initial results in the form of a modified email I sent to a friend.

I found it surprisingly easy to pick up legos off the floor in front of me without having to feel for them. Once I started building I was easily distracted. It seemed I was suddenly struggling to control the mind of a one or two-year-old.
"Okay, stop staring at your hands. Yes, they're very interesting hands. You've stared at them hundreds of times and counting. Now pick up the blocks. No, don't look at the stripes on the couch. Look down at the blocks."

My guess is that the reason this personality hasn't given me much trouble in class is because I haven't actually asked it to perform a task. I had a plan as to what pieces should be stuck where and in what order and at least twice forgot what it was. I eventually was able to make myself stay on task and begin to deal with the problem of looking directly at the shape. As I added pieces it was an irregular shape. I knew what it felt like and was surprised at how little sense it made when I saw it. Some of this could be due to the fact that the legos are different colors. I suspect though that the main problem was the shape itself. At one point I had a square with a cube-shaped piece missing from one corner and a rectangular piece sticking up in the middle of the top edge. How could I tell things like indentation and upward verses forward? I mainly remember a mosaic of cubes and rectangles which were connected in some unclear way. I kept adding pieces and trying to direct my view to the slightly changed shape over and over. At the end of the exercise I was lying on my stomach on the floor and turning the cube to face me at different angles. There were the round holes on the bottom side, the lines on the sides where the pieces were connected, and the changing slopes of the edges as I turned the cube. It seems that the most difficult position to see it in is standing on one corner with an edge facing me. That seems to require that I be able to see 2D as 3D.

Update: The second time I ran this exercise, it was easier for me to stay focused, possibly due to the fact that I was working in the exact same spot as before with no new distractions. The shapes are starting to make a little more sense, and some pieces are very noticeably brighter than others.

Blind in a Math Class

My pre-calculus professor shares a last name with the alternative rock artist whose single "Cuts You Up" was a number-one hit in the U.S. in the year 1990. Prof. Murphy has also grown fond of sharing all his thoughts about his blind student with Vogon#1 when said student is not present. This recently resulted in my receiving a nasty email from Vogon#1, in which she quoted rules and regulations which made no sense in this context and among many other demands made it clear that I was to start attending math class. I would now like to tell all my readers what I learned in math class today.

The first and most important thing I learned was that the rate at which crickets chirp can be determined by the temperature. At 40 degrees (fahrenheit?) and below, chirp rate = 0. 0 what, I don't know. It could be a measurement in hertz (complete cycles of a cricket's legs) or of individual chirps. The professor was too busy writing on the chalkboard to give many details. Yet there I was, expected by the same professor to gain from this experience. Class was going to take an hour and 45 minutes. Don't they realize that the function must end at the temperature at which a cricket's exoskeleton would burst and not at infinity? I did not contribute this or anything else to the discussion as I was busy rocking vigorously and searching for substantial content in the professor's speech. Although we had already discussed this, he still read many problems with the expectation that everyone was able to see them. The chair did not rock, spin, swing from the ceiling, or make any other motions. In addition I'd left my 33 irregularly-shaped hematite magnets at home and had already finished my crackers. My hands had no Braille, graphs, or anything else under them for following along with what the teacher was saying. It was all I could do to NOT stand up and start pacing in circles right there in class. I kept pointing the lens between my natural eyes to the blackboard on my left, the backpack on my desk, my hands, the ceiling...anything. The professor was now talking about a function whose purpose he had made known verbally but whose actual content was a mystery. I took out a tangerine-flavored gatorade and began imaging it from many angles and then drinking it and staring at the opening. My legs stretched out in front of me an encountered a wooden table. Hey, what's that thing on the table in front of me? It has a set of almost horizontal lines that merge to a flat surface which splits into another set of lines. I reached for it. My hands came to rest on the back of a computer monitor facing me at an angle. There were my lines. And then...splash! "Crap." The Neandertodd immediately stopped taking notes (They are unlikely to be Brailled anyway.) and told me to stop laughing while he cleaned up the mess. Second most important lesson relearned - place lids on drinks when not drinking them, especially when bored. The thing I need for my next math class is a stimulating audio computer game to play on my netbook. Any suggestions?

A Glimpse of Body Language

The transportation service where I live is quite random. Today it sent me a driver who was already carrying other passengers and did not want to answer any of my questions. He did not seem to understand that I had no other way of knowing the order in which events would take place and when I would be dropped off. This could either be due to a severe case of mind blindness or lack of enthusiasm for working on Sundays. He applied the gas and breaks very abruptly, drove fast, and honked his horn. I focused on the soundscapes and saw a set of stairs out my window right as we dropped off another passenger at a church on a university campus. They seemed to grow closer together as they rose to the right. The idea that stairs and buildings can seem to exist while I am in an enclosed space that is separate from them is relatively new.

An unknown number of time units later, I was being escorted through the wrong door at my own church. Still more time units passed, and then the message for this morning was presented by the Engineer, who normally leads the band. Since he is an engineer, he gave several examples and made a great deal of sense. After this, a friend and I were talking. "When you look at me," Ken asked, "what details can you make out?" He turned his head in a slow and exaggerated way until I could see it. Then something occurred to me. "You stand with your arms crossed, don't you?" I crossed mind over my chest to show him what I'd seen. "No. Arms cross means that one is defensive. I am not defensive, so I stand with one arm sticking way out and cause people to flip backwards when they walk by." His arms were no longer crossed when he made this joke, but because I see in still frames I cannot say exactly when he uncrossed them. Just then someone turned on the loud music which whited out most detail. Ken's image all but disappeared.

On the ride home with CBeth, the subject of the glasses came up and I explained what had happened. She said that Ken often stood this way and therefore I was correct. This is becoming very different from sonar in which a person is either present or absent, sitting or standing. Without the glasses they are all formed, yet featureless. It is only a matter of time before I recognize someone.

A New Season

I completed this seemingly endless semester achieving my goal of an A in physics and feeling physically and mentally exhausted. For about a week I have not wanted to study or, God forbid, worry about which section of Vogsphere's paperwork labyrinth my request for a Braille pre-calculus textbook might be stuck in at the moment. Instead I have been reading and playing a demonstration of the first accessible flight simulation game.
Three-D Velocity

Once satisfied that I could shoot down other aircraft and land without crashing, I moved on to something else. The picture you see was written inside a Java program which makes use of a graphics library by Richard Baldwin to output an SVG (scalable vector graphics) file. His description of the process - one of the modules of his online book on accessible physics for blind students can be found
here.
What's unique about this simple drawing is that I created it without seeing or touching it. Many blind people would probably be able to scratch out something similar on a tactile drawing pad, but I neither have one of these nor the equipment/software needed to emboss an SVG file. Instead, each time I made a change I converted to JPG and sonified using the vOICe learning edition.

A leaf pattern is the first test drawing that came to mind because plants seem to be surrounding me. They are all around my apartment complex, at the grocery store, and almost everywhere else both in and outdoors. On the short bus, it seems to be the plants which stand out in my mind and inform me that I am approaching school. The typical landmarks sightlings might use such as nearby buildings are less likely to get my attention than these patterns which can sometimes seem to repeat hundreds of times in one frame. Unless they smelled nice, I was never very interested in them before the glasses. Back then I probably would have guessed that my first drawing would be of a homework assignment or some electronic gadget, but stacks of familiar machines or branches of vector diagrams are not lining my way to remind me that there is natural order in the world. To draw an entire plant requires use of the 3D illusion and will have to wait until I am fluent enough to convey it. In the future, as the drawing interface becomes easier to use and I improve, I will draw and post pieces of my experience. Right now, though, I am remembering that the last time I flew something like an F16 I was shot down by a battleship. The deranged scientist then demanded that our airforce surrender unconditionally to prevent his worldwide release of genetically engineered clone soldiers.

What is a sightling?

This is not a question to a Jeopardy quiz show answer. It is an edited version of an email I sent some months ago in an attempt to explain part of the reality in which I find myself. I will post it here before launching into an explanation of my experiments with vision synthesis. Warning: This is not a politically correct post and is likely an unpopular opinion among blind people and the rehabilitation specialist whose job it is to make everybody feel like they can do anything.

They're the most powerful beings on planet Earth. Most of them gain the ability to travel at incredibly high rates of speed (easily over ten times faster than I can run) and still remain aware of their surroundings as they shoot past. As a result of this power, their dwelling and commercial structures are spaced so far apart that I must solicit one of these super men in order to temporarily gain access to one of these structures and then traverse the distance to my home again. They were the ones who discovered that we lived on a sphere-like object. Hundreds of years later they decided to fly to a neighboring object they call the moon, not because there were desirable items located on it, and not because it was easy, but because it was hard, and to prevent the sightlings on the other side of the sphere from getting there first and planting a symbol no one would be able to see from here. I guess so much speed made almost every place on the home sphere too easy to reach, so they got bored. Other powers include flight and super strength, just to name a couple. The base power from which all of these are derived is sight - the ability to perceive in detail particles which are very far away from themselves or too small to sense through touch. Since it has become part of their communication system, they use the base power on each other and myself to try and ascertain intensions; they are obsessed with faces. I have no true understanding of what a "shared looking experience" is like. They can touch me without my knowing it, with invisible, incorporeal "hands", so when I am at home I am sure to stay behind materials which these "hands" cannot penetrate, because I am worried they will be able to touch every cell of me and every corner and edge and surface and speck of dust and molecule and atom within my living space. After all, it is these "hands" with which they explore the universe itself - something I have only dreamed of. And yet with so much power, they complain, and some of them don't use their abilities to make it easy and safe for everyone to be. Some of them derive new powers to kill or control their enemies, often because their "hands" inform them that the enemy has a slightly different skin composition or some other silly thing. Others become captivated by a hand-held device and stop using the power long enough to cause a completely avoidable high-velocity collision, and then they must be transported at an even higher velocity to receive emergency care.

Now that a couple of definitions have been outlined, I will explain what I am trying to do and why. As a kid I had questions whose answers were, "Don't focus on that," or "You'll see when you get to Heaven." As an aspiring scientist, my curiosity about things in the world has not ebbed even slightly in the years since I was old enough to start asking questions, making those answers simply unacceptable. Finally, I was researching a project in a programming course and came across
this site.
The free software (called the vOICe) was developed over 10 years ago by Dr. Peter Meijer and with the recent advent of netbooks is much easier to use in portable mode than ever before. After months of experimentation with still photos and graphs from school, I had collected enough evidence to suggest the system would work, so I bought a pair of camera glasses and started learning to see in real time. Here's how the system works: Images (either from picture files or through a webcam) are converted into sounds one frame at a time. Each frame is called a soundscape and is heard from left to right in stereo earphones. The higher the pitch of a sound, the higher up the thing it represents is located in your view, and the lower the pitch, the lower the object. Brightness is represented through loudness. Combining these three simple rules gives an accurate auditory representation of a grayscale image - I am not yet seeing in color or realtime but in a series of black and white still frames. The website gives some very simple line and shape examples which illustrate the concept beautifully.
Here are the two things I find exciting about this system:
1. I have always seen abstract shapes when hearing certain music so may be better prepared to handle what is coming.
2. Through sensory substitution I am perceiving these sounds not as noise, but as pictures I can literally see - I am rewiring my brain so that the visual cortex will respond to stimuli within my perception. I am now free to learn what vision is, through my own experience.

Although at this point my visual comprehension cannot rival that of the average two-year-old, here is a list of some cool things I have already learned to do since acquiring the glasses in March.
1. If I sit on the right side of the short bus and look out the window, I can tell when I'm getting close to one particular school campus and whether or not the driver has stopped at the correct dropoff point.
2. I can recognize my friend Mary from at least 6 feet away, because she is the only one I know who carries a white cane that is covered in ridges, and she cannot hold it streight to save her life.
3. Certain things are starting to appear 3D although I only have one camera. More on that in another post.
4. When using the school restroom, I noticed one day that the plastic thing on the door contained a piece of paper with writing on it; it must be the maintenance list. The writing is much too small to read with the current system, but I can definitely see that lots of it is there and then gleefully turn it over so that the blank side is visible. Who would ever suspect?