Dawn of the New Everything

Preface: Virtual Reality’s Moment

It was the late 1980s, and a large envelope with a formidable DO NOT X-RAY sticker had just been dropped through the slot of the front door of a tech startup in Redwood City, California. The envelope contained a floppy disk that held the first digital model of a whole city. We had been waiting all morning. “Jaron, it’s here, get to the lab!” One of the engineers rushed to snatch the envelope before anyone else could get at it, sliced it open, ran to the lab, and slid the disk into a slot in a computer.

It was time for me to enter a brand-new virtual world.

I squinted up at my hand against a perfectly clear blue sky. My gargantuan hand, soaring above downtown Seattle. It might have been a thousand feet from wrist to fingertip.

There was a bug, obviously. A hand should be about the right size to pick up an apple or a baseball, not bigger than a skyscraper. You shouldn’t have to measure a hand in feet, much less thousands of them.

The city was abstract. This was in the early days of VR, so plasticine blocks stood in for most buildings, in a jumble of too-cheerful-for-Seattle colors. The fog was preternaturally uniform and milky.fn1

My first thought was to stop and fix the bug, but instead I took a moment to experiment. I flew down and tried to nudge a ferry on the sparkling Puget Sound. It worked! I had control. Not what I expected. That meant that I could still inhabit my hand when it was preposterously huge.

Once in a while, a bug in VR exposes a fresh way that people can connect to the world and each other. Those are the best moments. I always stop and linger when it happens, to hang on to the sensation.

After a few experiences of VR bugs, you have to ask yourself, “Who is it who is suspended in nothing, experiencing these events?” It is you, but not exactly. What is left of you when you can change virtually everything about your body and the world?

A bundle of cables connected my EyePhone, through a loop hanging from the ceiling, to a line of refrigerator-size computers that roared to keep cool. I wore a DataGlove on my hand; slick black mesh woven through with fiber optic sensors, and yet more thick cables from the wrist arcing up to ceiling rings. Blinking lights, flickering screens. The EyePhone’s rubber rings left moist red indentations around my eyes.

I wondered at the strangeness of the world I found myself in, now back in the lab. Buildings in Silicon Valley used to have carpeted walls and cheap Space Age desks with fake wood grain. A faint smell of aluminum and dirty water.

A gang of eccentric technical geniuses converged, impatient to try. Chuck in his wheelchair; a robust, bearded lumberjack. Tom acting all professional and analytical, even though only a few minutes earlier he’d been telling me about his crazy adventures exploring San Francisco overnight. Ann seemed to be wondering why she was yet again cast in the role of the only adult in the room.

“Did it feel like being in Seattle?”

“Kind of,” I said. “It’s, it’s … marvelous.” Everyone shoved toward the gear. Every little iteration of our project got better. “There’s a bug. The avatar hand is huge—by magnitudes.”

I never got tired of the simple act of using my hand inside VR. When you could bring your body in there, you were not just an observer, but a native. But every tiny detail of functionality, of figuring out how a virtual hand could hold virtual things, turned out to be a struggle.

Fix a problem with how virtual fingertips mistakenly penetrate objects they are trying to pick up, and you might accidentally make the hand gargantuan. Everything connects with everything. Every tweak of the rules of a new world is a potential setting for a startling, surrealistic bug.

Bugs were the dreams within virtual reality. They transformed you.

A moment with a giant hand changed not only how virtual reality felt to me, but how physical reality felt. My friends in the room now looked like pulsing beings, translucent. Their transparent eyes were filled with meaning. This was not hallucination, but improved perception.

Physicality revealed in fresh light.

Introduction

What Is It?

VR is those big headsets that make people look ridiculous from the outside; those who wear them radiate startled delight at what they’re experiencing from the inside. It’s one of the dominant clichés of science fiction. It’s where war veterans overcome PTSD. The very thought of VR is the fuel for millions of late night reveries about consciousness and reality. It’s one of the only ways, for the moment, to raise billions of dollars fleetly in Silicon Valley without necessarily promising to spy on everybody.

VR is one of the scientific, philosophical, and technological frontiers of our era. It is a means for creating comprehensive illusions that you’re in a different place, perhaps a fantastical, alien environment, perhaps with a body that is far from human. And yet it’s also the farthest-reaching apparatus for researching what a human being is in the terms of cognition and perception.

Never has a medium been so potent for beauty and so vulnerable to creepiness. Virtual reality will test us. It will amplify our character more than other media ever have.

Virtual reality is all these things and more.

My friends and I founded the first VR startup, VPL Research, Inc., in 1984. This book tells our story, and explores what VR might mean to the human future.

Recent VR enthusiasts might exclaim, “1984, no way!” But it’s true.

You might have heard that VR failed for decades, but that was true only for the attempts to bring out a low-cost, blockbuster popular entertainment version. Just about every vehicle you’ve occupied in the last two decades, whether it rolls, floats, or flies, was prototyped in VR. VR for surgical training has become so widespread that concerns have been expressed that it’s overused. (No one would suggest that it shouldn’t be used at all; it’s been a success!)

What Can a Book Do That VR Can’t, at Least as Yet?

The romantic ideal of virtual reality thrives as ever. VR the ideal, as opposed to the real, technology weds the nerdy thing with the hippie mystic thing; it’s high-tech and like a dream or an elixir of unbounded experience all at the same time.

I wish I could fully convey what it was like in the early days. There was a feeling of opening up a new plane of experience. Inhabiting the first immersive avatars, seeing others as avatars, experiencing one’s body for the first time as a nonrealistic avatar; these things transfixed us. Everything else in the tech world was dull in comparison.

I cannot use VR to share what that experience was like with you, at least not yet. VR, for all it can do, is not yet a medium of internal states. There is less and less need for me to make this point as VR becomes more familiar, but it’s a clarification that I have been called upon to give many times.

There’s occasional talk about VR as if it is on the verge of evolving into telepathic conjuring of arbitrary reality along with a conjoining of brains. It can be difficult to explain that VR is wonderful for what it is, precisely because it isn’t really everything.

Eventually a new culture, a massive tradition of clichés and tricks of the VR trade, might arise, and that culture might allow me to convey to you how early VR felt, using VR-borne technique. I have spent many hours daydreaming about what a mature culture of expression would be like in VR. A cross between cinema, jazz, and programming, I used to say.

First VR Definition: A twenty-first-century art form that will weave together the three great twentieth-century arts: cinema, jazz, and programming.fn1

Even though no one knows how expressive VR might eventually become, there is always that little core of thrill in the idea of VR. Arbitrary experience, shared with other people, conversationally, under our control. An approach to a holistic form of expression. Shared lucid dreaming. A way out of the dull persistence of physicality. This thing we seek, it’s a way of being that isn’t tied just to our given circumstances in this world.

If I tried to tell the story of VR dispassionately, I’d be lying. What makes VR worthwhile to me is that it’s about people. I can only tell you what VR means to me by telling my story.

How to Read This Book

Most of the chapters tell a story that begins in the midsixties, when I was a boy, and ends in 1992, when I left VPL.

There are also chapters interspersed throughout that explain or comment on aspects of VR, such as a chapter on VR headsets. These “about” chapters include a dusting of basic introductory material, a hearty portion of sharp opinions, and more than a few out-of-sequence anecdotes. You have my permission to skip through them if you prefer storytelling to science or commentary. Or, if you don’t like storytelling and just want to read my thoughts on VR tech, then race right to those chapters.

Some of my stories and observations are found in long footnotes. I bet you’ll be glad if you find the time to read them, but you can leave that for later. There are also three appendices that expand on my ideas from the period, but are ultimately more concerned with the future than the past. Read them if you want to know what it feels like to have an informed worldview that doesn’t include AI destroying humanity any minute.

In keeping with the time period of the narrative, I’ll talk more about classical VR than mixed reality,fn2 even though that’s what I’ve worked on more lately. (Mixed reality means the real world is not hidden entirely by the virtual one; you see virtual stuff placed within the real world, as experienced lately in a HoloLens.)

Meeting My Younger Self

Never thought I’d see you again.

You are so wrong. It would be easier not to deal with you. I’ve been feeling more comfortable with myself than ever before. Dealing with you brings up crummy old patterns. I get insecure and depressed. You’re recidivism bait. I’m only doing this because I think it would be useful for other people to know about you.

Yeah, most people call it VR now.

No one remembers or cares about that war. It’s just words.

Well, we’re about to find out. It looks like this book might come out at about the same time that VR gets commonplace.

Yeah, who knows … You know how hard it is to do VR well.

Oh, you’d miss them. You won’t believe it, but singularity freaks cross-bred with libertarians, and their fanatical offspring are the main drivers of tech culture these days.

I feel embarrassed that you were expecting a perfect world.

Oh, c’mon, let’s not fight. There are plenty of people out there to fight with.

Well, usually not while they’re inside, but yeah, a lot of people will probably be able to make worlds.

Hey man, I’m not the VR police. I don’t run the show.

It’s actually great to watch the kids reinvent VR. There are all these cute VR startups and teams in the big companies. Some of them even remind me of you and VPL, though the fashion these days is a lot straighter.

Look, you can’t devote yourself to serving people if you think you’re better than them. VR will be kind of crummy but also kind of great and it will evolve and hopefully get really great. You have to relax about it. Enjoy the process. Respect the people.

Well, yeah, I guess … this book …

No, VPL is long gone. Microsoft brought out a self-contained mixed reality headset—doesn’t need a base station—goes anywhere. You’d be really impressed.

Um, my research post lately is in Microsoft’s labs.

Give it a rest. Classic VR gear is also shipping; not unlike what we used to sell. One of the social media companies bought this little company called Oculus for two billion dollars.

Oh, that’s a corporation people use to communicate with each other and keep personal remembrances, and there are algorithms that model the people so offers can be targeted; these companies can make people sadder or more likely to vote by tweaking the algorithms. They’re the center of a lot of people’s lives.

It’s both paradise and hell.

Weirdly, the new generation gap is—supposedly—that young people are more comfortable with corporations running digital society.

But that’s normal for the world. It’s what happens with time.

Maybe you do.

1. 1960s: Terrors of Eden

Border

My parents fled the big city right after I was born. They roamed for a while; eventually alighting in what was, at the time, an obscure and harsh place. The westernmost corner of Texas, outside El Paso, at the juncture of New Mexico and Mexico proper, was an outback, barely part of America. It was impoverished, relatively lawless, and of unsurpassed irrelevance to the rest of the country.

Why there? I never got a clear answer, but my parents were probably running. My Viennese mother had survived a concentration camp and my father’s family had been mostly wiped out in Ukrainian pogroms. I do remember hearing that we had to live as obscurely as possible, but it would be unacceptable to live too far from a good university. They came to rest in a place that split the difference, for there was a good university nearby in New Mexico.

I remember my mother saying that the Mexican schools were more like those in Europe, with a more advanced curriculum than was available in rural Texas at the time. Mexican kids were a couple of years ahead in math.

“But Europe wanted to kill all of us. What’s good about Europe?” She replied that there were beautiful things everywhere, even in Europe, and you have to learn how to not get shut down completely by the evil of the world. Besides, Mexico most definitely wasn’t Europe.

So I crossed the international border every morning to go to a Montessori school in Ciudad Juárez, Mexico. Sounds strange today, since the border has come to look like the world’s most advertised prison, but back then it was understated and relaxed; creaky little school buses crossed all the time.

My school was a world apart from the one I would have attended in Texas. Our schoolbooks were sheathed in fantastic images of Aztec mythology. Teachers dressed up for holidays; colorful fabrics, mod 1960s cuts, with large, living, iridescent beetles soldered to silver chains, free to wander on shoulders. Every hour or so, the beetles were offered brightly colored sugar water from eyedroppers.

Since it was a Montessori school, we were free to roam like beetles, and I made a discovery. Looking through a ragged old art book on a low shelf in our forlorn schoolhouse, I saw an image of Hieronymus Bosch’s triptych The Garden of Earthly Delights.

Window

I remember being scolded at my little school for not paying attention. Instead I stared out the window endlessly, as if hypnotized. But this was not spacing out. It was an intense contemplation.

¡Atención! Pay attention!

The Garden of Earthly Delights had left me thunderstruck. I imagined being inside it, petting the soft, giant birds of intimate velvet colors, crawling through playgrounds made of transparent fleshy spheres, plucking and blowing the mammoth musical instruments that pierced each other and would eventually pierce me. I imagined how that would feel. An intense tickling, a spreading warmth.

A few of Bosch’s figures look out from the canvas. What if I were one of them? When I was staring out that window, I was looking out at our supposedly normal world from inside the painting. No small chore; it took hours, infuriating the teachers.

¿Qué es lo que estás mirando? “What have you been staring at?”

I saw the occasional naked child alight on the little sandbox, then prance until caught, rather like in the painting. But I also saw beyond the yellow grass of the schoolyard, through the chain-link fence, to a dusty and chaotic city street.

Grizzled men in frayed straw hats inside the glass heads of mammoth trucks painted in carnival colors, blinding speeds, black noisy exhaust clouds; weathered pastel neighborhoods vanishing up into tortured striations of rock in the far desert mountainside; silver planes in the sky filled with people. Right across the street was a heroic two-story mural of Quetzalcoatl climbing a parking lot wall.

Estoy viendo maravillas. I see miracles.

Right up close, just behind the fence, I could make out more detail: furled growths on the chest of a beggar; the tottering motion of a polio survivor delivering stacks of fresh newspapers; dirt on the fringes of a teenage boy’s green shirt; a pyramid of shiny green cut cactus on the handlebars of his wobbling bike. I once saw gashes in the face of a sullen prisoner in the smoky rear chamber of a careening Mexican police car, viewable for the barest instant through blinding, sweeping beacons.

Trifecta

Was everyone else in my little school blind and deaf? Why were they so inert? Why wasn’t everyone else thunderstruck? I did not understand them.

I became obsessed with useless speculations. What if I had gone to a school across the river, in Texas? Things must be more orderly in Texas. If you took a copy of The Garden to Texas and the little naked people looked out, would they see a world that looked weird, or would they say, “Wow, we didn’t know anyplace could be that boring!”

Was it possible that every place in the whole universe was wondrous, but people just get worn out by the chore of perception? Is that why all the other kids just sat there, pretending that everything was normal?

Of course I couldn’t have articulated these words. I was tiny.

I stared and stared at the painting and then out the window and then back. I felt my interior color shift each time, like blood rushing in and out of my head. Why was the painting so luscious? What was so naughty about it that drew me in?

Even better was staring at the image while listening to Bach. The schoolroom had a battle-worn record player. One LP had Bach’s organ music played by E. Power Biggs and another offered Glenn Gould on piano.

My favorite thing was staring into The Garden while listening to the Toccata and Fugue in D minor, loud, and eating from a bowl of Mexican chocolates, tinged with cinnamon. Hardly ever allowed.

Mood

My earliest memories are of being consumed by an overpowering subjectivity. Everything was distinct, moody, filled with flavor; each little place and every moment was a fresh spice in an endless spice cabinet, a new word in an endless dictionary.

It continues to surprise me how difficult it can be to convey a state of mind to those who do not immediately recognize it. Imagine you are hiking in the light of the full moon at midnight on a high ridge in New Mexico, looking down on a valley dusted in new snow that appears to fluoresce. Now imagine an exchange between two fellow travelers, one a romantic and the other possessing a dry, analytical temperament. The romantic might say, “Isn’t this magical?” while his opposite might say, “Well, the visibility is unusually good and the moon is full.”

In my childhood I was hyperromantic, unable even to conceive of a pragmatic notion like “visibility,” because the experience of “magic” was completely overwhelming, to the near exclusion of everything else. My early experience was of the dominance of flavor over form, of qualia over explanation.

Over time I’ve learned to become more normal, or more boring. It used to be that I could hardly stand to fly from one place to another because the shift in mood and quality would be so overwhelming. I would always be stunned by what it felt like to land in San Francisco, coming from New York, even after doing it hundreds of times. The air was brisk, tinged with the smell of gasoline, but also the ocean; it was thinner, less pregnant. Just to take in the shift in feeling could take hours.

I worked at being able to suppress the overwhelming burden of subjective mood for many years, and started to make progress in my late thirties. These days, I fly from one place to another without difficulty. The airports are all finally starting to feel similar.

Spun

I called my parents by their first names. Lilly had been a prodigy concert pianist as a girl, born to a successful Jewish family in Vienna. Her father was a professor and a rabbi; an associate of Martin Buber’s. They lived in a nice house, had comfortable lives. My grandparents were determined to wait out the threatening politics of their day. They were convinced there was a limit to how low people would sink.

Lilly was a precocious and resourceful teenager, and while it would normally be the last thing you’d care about, it turned out to be crucial that she was very light-skinned and blond. She was able to talk her way out of a pop-up concentration camp by passing as Aryan, and then to forge paperwork to get her father released just before he would have been murdered.

Maneuvers like this were only possible in the earliest days of the Holocaust, before genocidal procedures were optimized. In the end, most of my mother’s family was murdered by the Nazis.

Some got out, eventually to New York City. At first, Lilly made a living as a seamstress; soon she had her own lingerie brand. She studied painting, and was still young enough to train as a dancer. She earned her own money to pursue these dreams. In photographs, she looks like a movie star.

We were so close that I had barely ever perceived her as a separate person. I remember playing Beethoven sonatas on the piano for her and her friends, and it felt as if we were playing them together from the same body. The interpretation was languorous and showy.

My parents had just transferred me to a Texas public elementary school. No art books to peruse, nothing interesting out the window. They were worried that I wasn’t learning what I would need in order to integrate into America.

Boy, was that true. In order to get to my new school, I had to walk through the territories of neighborhood bullies. These were kids with cowboy drawls and dirty boots. I was shocked when my parents decided a karate class would be prudent for me.

I loathed every little thing about karate, except that the costumes were kind of cool. When my mother came to the faux Texas dojo to see a demonstration of my training, I stood still and took it while another boy hit, kicked, and chopped at me. I don’t remember feeling scared or shy, but instead that fighting this other person would be stupid, wrong; just bad. Besides, the kid couldn’t really fight and nothing he did actually hurt. But my mother was horrified; she looked disappointed in me for the first time ever. I remember feeling the sky drop.

The next morning, as I walked across the hardened soil and stubby yellow grass in our yard on my way to school, I was surrounded by big bellicose bullies. I had a baritone horn with me, which is like a mini-tuba. But to a nine-year-old, it’s about as big as a tuba, and a strategy formed in my mind.

I started turning like a helicopter, the horn extended like a shield, though it acted like a battering ram. The bullies were not clear on the concept of momentum, and tried charging me head-on a couple of times, only to be knocked sideways to the ground. They couldn’t find it within themselves to stop for a moment to reconsider their approach. I think there were three of them, soon bruised and running away. I was dizzy, but music had saved me.

Suddenly my self-satisfaction was atomized by shrieking. Lilly was standing behind the front door, cracked open just a bit, wailing as if the Nazis had come for me. She was not dressed and did not come outside. It took me years to realize that she must have experienced a flashback to Vienna.

At the time, I was terrified by her reaction. My not fighting at the karate studio had displeased her, and yet here I fought, but that also freaked her out. Suddenly I felt a disconnection. The sensation was so disorienting and unpleasant to me that I didn’t know what to do. I ran away, to school. That was the last time I saw her.

Beyond Recall

A glum man with sharp features, in a perfectly pressed military uniform, knocked at the door of the classroom and asked for me. I was happy to get out of a droning lecture about the Alamo, but something felt terribly wrong.

Soon I saw that the principal was also there, and this man asked me in the most formal voice I had ever heard to follow them to her office, where I had never been. There was a flag, a framed photo of President Johnson. Was I in trouble for hitting the bullies with the horn?

Then these strangers told me that my mother was dead and my father was in the hospital.

It happened to be the day Lilly was set to go into town for her first-ever driver’s test. The DMV was about an hour away, near downtown El Paso. Ellery, my father, drove on the way there. She passed the test.

Lilly was driving back, on the big freeway, when their car spun out of control, flipped, and flew off a high overpass. Or so said a fresh newspaper clipping, which the principal gave me, as if that was helpful.

For years I worried that Lilly’s traumatic flashback that morning might have made her panic on the road. I was consumed by guilt. Had I been part of the problem?

Decades later, an engineer friend of mine read about a possible flaw in the model year of the car. It was a match for the events of the accident. By that time it was way too late to look into legal recourse, but I wondered, why did my parents even buy a car from Volkswagen? It wasn’t a “beetle,” the model designed by Hitler, but still.

The choice must have been part of my mother’s program to find the good in Europe, in everything.

It turned out that the military man was a distant relative the police had tracked down. He was named in my mother’s will, and was stationed at Fort Bliss, the military base that makes up much of El Paso. I had never heard of him.

I was taken to the hospital to see my father, whose body was blackened in between bandages, after he had become conscious. Both of us cried uncontrollably, so hard it felt like I’d choke to death.

This memory is a wall. I remember almost nothing else from before my mother’s death. My slate was shaken clean.

Sound

I was disconnected from the world for a long time after. Endured a desolate tour of life-threatening infectious diseases, barely aware of my circumstances. I was virtually immobile for a year in that same hospital.

Ellery was devoted. He slept on a cot next to my hospital bed. The seasons cycled, and I finally started to engage with the world again. I remember paying attention to my new surroundings for the first time.

The hospital was cramped, hot, and noisy. Cracked pea-green tiles running halfway up the wall, greasy windows embedded with chicken wire, splintered frames, peeling dark green paint. Smelled like medicine and urine. Big tough nurses with tiny crosses hanging from their crinkled necks; they moved like tanks, mostly ignored everyone.

I started to read. Books propped on crimped-up bedsheets.

Then, two moments of irreversible positivity sparked in me, just because I read sequences of words.

One was the Jewish admonition to “choose life” in a children’s book about Jewish culture. There was a logic to it, since death would come soon enough, no matter what, so choosing life was at least a reasonable bet. Like Pascal’s Wager, but for this life. (Not that I would have heard of Pascal or his wager as a kid.) But as I thought about it, I realized that “choose life” has even more going on.

It’s so obvious that you could miss it, but the phrase tells you that life is a choice. Furthermore, it suggests that once you notice that you’ve chosen to live, then you might notice that you can probably also make further choices. I needed to hear that, because it had not even occurred to me that I had any choice at all at the time. Before reading those words, all I could do was lie there, waiting for whatever might happen next.

But then there’s an even deeper level to the phrase. You choose even though you can’t ever know what that means. This physical world we inhabit; we’re only in it because of a crazy bet we make with the unknown. Maybe there’s peace and happiness to be found in uncertainty. There isn’t anywhere else to look.

I guess you, my reader, might wonder if I’m forcing adult thoughts into my recollections of a child’s brain, but I remember this phase pretty clearly. I was obsessed with what’s usually called philosophy, and it helped.

The second bit of reading was a biography of Sidney Bechet, one of the great early New Orleans wind players. According to the book, he overcame his childhood respiratory problems by playing the clarinet. Well, I had a nasty case of pneumonia that persisted for months, along with other respiratory distresses, so I asked Ellery for a clarinet. Not only was it a great way to annoy the nurses, but my lungs started to clear.

This is starting to sound like a familiar inspirational healing narrative, but there is something else you should know. My father and I never again spoke of my mother.

Silence is not forgetting in intimate settings. Just the opposite. We still lit Yahrzeit candles; we cried for years.

Decades later, I realized that both my parents had no choice but to put those who died out of mind much of the time. It was the only way to make space for life, for there were so many who had died so horribly.

Ellery had an aunt who was entirely mute, but she wasn’t born that way. As a girl she had survived by keeping absolutely silent while her big sister, who she clung to, was slain by sword where they hid under a bed during a pogrom.

To Ellery, Lilly’s death was one of many. Therapists, daytime TV hosts, and social networks all counsel that we should talk. It’s a luxury when you can afford to do it.

Outlasting Cruelty

When I emerged after my long dormancy, hosting one disease after another, I was fat, but didn’t know it at all. I was numb. When I finally went back to school and the kids laughed so cruelly, only then was I suddenly aware.

Normally, taunts from children would be traumatic, but there was more. Juvenile cowboy bullies bragged about how they drowned a tiny Chicano kid in the neighborhood swimming pool, an event that the adult world had officially registered as an accident, though everyone knew.

The bullies said I was next. They were credible. There were only a few Chicano kids, and they’d come in with casts and scars, and look away from everyone.

One of the teachers pointedly reminded us in class that the Jews killed Jesus and were still paying the price. She then said this ancient, cosmic crime probably had something to do with my mother’s accident. My mother had had it coming.

I realize now that this teacher was making her best attempt at kindness. It was her way of saying I couldn’t help that I was born Jewish. In a similar vein, she urged the white children to be understanding, since Mexicans couldn’t help the fact that they were less intelligent.

Thereafter, I was bombarded with demands that I convert. My memories of that school are of constant onslaught, racism, and violence; of adults who were no better than the children.

I was a few years younger and therefore smaller than the other kids in my grade, and an easy mark. A notorious cowboy kid confronted me, with a crowd urging him on. He was a dandy, in a kid’s black western shirt. I suddenly recalled lessons from the karate school, which by then had taken place so long ago, in the unreal antebellum. I built up tension in my whole body in order to aim a punch and knocked the guy flat on his back.

It would be nice to tell the Hollywood story here. Wouldn’t I suddenly be an alpha, raised on shoulders, loved? No, I found myself more isolated than ever. I was regularly ambushed and beaten.

The thought of connecting with other people—of having friends—was terrifying, and strangers were dangerous. It was impossible to say how much of my dread was due to my circumstances and how much was inherited from my parents.

Reality is ever changing. The odd demographic churnings of the region eventually brought me into contact with a variety of people, and I slowly learned to connect pleasantly with oddballs I stumbled into. One time, I wandered into the Radio Shack store on the main road and met a polite soldier from Fort Bliss in a not-so-fresh beige uniform.

The guy was awkward; looked down all the time and walked as if the floor was being tilted back and forth just a little bit by a supernatural prankster. He noticed me yearning at the drawers of electronic parts and said hello.

He seemed young, even to me. Not quite a real mustache yet. Worked with radar equipment. Wouldn’t tell me more.

What makes people generous? What makes a stranger take a chance? This fellow took it upon himself to introduce me to electronics. He brought a few parts to our little house a couple of days later; resistors, capacitors, wire, solder, transistors, potentiometers, a battery, a little speaker. We made a radio.

First Experiment

Right next door to that Radio Shack was a drugstore with a magazine stand. Before the Internet, magazine stands were a big deal. You could go and browse the covers without even touching; show dogs, fancy boats.

The rack itself was made of thick, curled, shiny wire; fancy but cheap-looking at the same time. You had to be choosy about when to go, because the afternoon desert sun blasted through the big window. It would glint off the wire, making it painful to look.

There was always more to learn about the magazine rack. The magazine covers lost their warm inks and turned blue after about a week in the strong sun, so you could always tell how long an issue had been out. There was supposedly a room with dirty magazines in the back, but I never saw it.

A few titles covered the amateur electronics beat. Most of the articles were about building radios, but I found a piece about an early electronic musical instrument called a Theremin and learned how to make them. Theremins are played by moving your hands in the air near antennae; nothing is touched, and playing gives you the feeling of contact with a virtual world.

I was also fascinated by the diaphanous, silky, turbulent images called Lissajous patterns, which can be made by fiddling with musical signals and an oscilloscope. I made a crude Lissajous viewer out of an old television set I found in a trash bin, and hooked it up to a Theremin. Normally, a Theremin makes spooky warbling sounds, but I got one to make spooky warbling images.

As Halloween approached, a plan formed in my mind: I would build a fantastic haunted house out of my electronic contraptions and attract people worthy of being friends! There must be others like that kind soldier, just wandering about, out of sight, like desert tortoises. All you had to do was find them.

I hung sheets around our tiny front porch and set up an old enlarger lens to project Lissajous patterns from the TV onto them.

Once the sun went down and the images appeared bright, I was deliciously surrounded by fantastic dancing forms. The motions of anyone nearby would alter the patterns, as if by the invisible strings of a puppeteer, courtesy of the magical Theremin antennae.

I wondered whether any girls, those beings of utter mystery, might be delighted. Who wouldn’t be?

My haunted house pleased me immensely but attracted no visitors. I watched from inside my palace of imagination and freedom as one child after another crossed the street to stay as far away as possible. It never occurred to me at the time that they were probably frightened. They had certainly seen nothing like it before.

After that Halloween, the bullies stopped bothering me. I had made myself into a scary unknown. Progress.

Burned

Another astonishing thing about my mother was that she was the breadwinner of the family, at least when we had moved out west. In that era it was always the man.

There was a truckload of ill will directed toward Ellery over this matter, before her death and after. “The boy needs to see his father earning an honest living, being strong. You’re letting him down. Keep this up and he’ll grow up funny.” Good citizens were so sure of themselves that they didn’t care if I was right there, listening to it all.

My mother made our money over the phone, trading stocks in New York, back when no one did that. She wasn’t a tycoon, not even close; we were middle class, but not in the upper reaches of the middle. We could eat at a drive-up hamburger joint every week.

Players are usually either well-off or wannabes on Wall Street, or any other wide-open platform, but my mother found a do-well-enough niche. Could she have done better? Maybe she was afraid of standing out, being seen.

I do remember a little more than I’ve said so far. I remember her getting off the phone one day and exclaiming that she’d just closed a great trade, made not just hundreds, but a few thousand dollars. That call persists in my recollection because it yielded the money that bought the car she died in. We went out the very next morning to buy it. I got to pick the color.

After my mother died, there was a secondary crisis, because Ellery and I no longer had income.

While I was in the hospital, Ellery enrolled in a program to get certified to be an elementary school teacher. So that was a solution to the income problem, but then something else went wrong.

We had known for a while that our lease would run out and we’d have to move. This had happened a few times. My parents finally arranged to buy a house so we’d never have to move again unless we wanted to.

It was a tract house under construction in a development at the edge of El Paso, entry level, but upscale compared to where we had lived before. It had a carport! I visited it only once when it was still under construction, but I was fascinated by the blueprints and pored over them. I learned all I could about drafting and construction. I couldn’t wait to move in.

While I was in the hospital this house was completed, then burned down the next day. Ellery told me but the news didn’t register. I thought it must have been a dream, and was still confused about the facts when I was discharged.

The police informed Ellery that the fire was arson, but there were no witnesses and no suspects. Ellery muttered that we might have been targeted, but it could have been random. Bad things happened all the time around there.

There was a screwup with the bank or the insurance. After the incineration we didn’t recover any of the money my mother invested in the property. Ellery felt especially bitter about having to pay to have the ruins cleared.

Thus it came to pass, not long after my haunted house experiment, that we had to move but had no place to go.

2. Rescue Spacecraft

Landed

Ellery did something unthinkable and brilliant. After my mother died, and our new house burned down, and we were broke, and I oscillated between terror and isolation—after all that, he bought an acre of throwaway land in New Mexico.

The lot was cheap enough that he could buy it with the small remaining amount of cash on hand. And he found a teaching job in the same area.

An undeveloped corner of the desert. We had no money left to dig a well, much less build a house. We had to live in tents at first. Our belongings, even my mother’s baby grand piano, were wrapped up in plastic on pallets out in the open on the indifferent desert dirt.

Ellery took to teaching sixth-grade kids in the rough little barrio in the center of Las Cruces, New Mexico, as if it were an art form. He had tough kids build cardboard spaceships to inhabit all day long. They launched model rockets and used sand to explore the ideas behind calculus. He was known as pelón to the kids, for Ellery was shiny bald, like a polished gem.

Whenever I go back to visit Las Cruces, people come up to me, and with that distinct New Mexico Chicano accent, say things like, “Your dad Ellery changed my life. My older brother is in jail, but I’m a NASA engineer.”

We ended up living in tents for longer than planned, for over two years. When Ellery’s earnings as a teacher started coming in, the first priorities were a shed where electricity and a phone could be connected, a well for water, and an outhouse.

The high desert can be bone cold, and I remember shivering like a spring marionette on winter mornings. The people who bought lots around our land wheeled in trailers and mobile homes. We talked about it. We could do the same, but that would mean diverting money from the big plan. Not worth it.

We grew vegetables. Raised chickens.

Tent life was not so bad. It clarified your role in your own survival, and we both needed that. And those rolling homes were so ug-ly.

Where in the Universe?

There was a social anomaly in that part of New Mexico: a population of superb engineers and scientists employed by White Sands Missile Range. They were everywhere. It was a relief to discover the culture of technical people, which was welcoming to an awkward kid like me.

One of our near neighbors was a lovely, slight old man named Clyde Tombaugh, who had discovered the planet Pluto in his youth. When I knew him, he directed research in optical sensing at White Sands.

I learned how to grind lenses and mirrors from Clyde, and I still think of him when I work on virtual reality headset optics today. He built impressive backyard telescopes, and he let me play with them. I will never forget a globular cluster he showed me—a vividly three-dimensional form, a physical object like me, a cousin to me, as real in front of me as anything else in the world. I gained a sense of belonging in the universe.fn1

I went to public school in New Mexico, and I don’t remember a lot about it, which probably means it was okay. At the very least, I didn’t experience terror.

Right after we arrived, before I got to know any other kids in the area, an amazing thing happened. One evening there was a perfect breakdown of the local telephone system. Anyone who picked up a phone could hear everyone else, all at once.

Hundreds of voices—some sounding distant, some close by—hovered in the first social virtual space I had ever experienced. An instant society of children formed, brilliantly superior to any I had experienced before.

The floating children were curious about one another; they were friendly. It was less fraught to communicate with strangers than in real life. The voice of a little boy said, “I’ve hugged every woman in the world as a pillow.” And this was with real girls floating nearby.

It was late at night, and none of us were supposed to be up, though I might have been the only one in a tiny plywood shed that could only be secured by padlock.

The next morning at school, no one spoke of what had happened. I looked around and wondered who I might have talked to the previous night. Was it possible that people could suddenly improve, if the medium that connected us was different?

Ever since, I’ve tried and tried to rediscover that formula. Maybe it was a one-time benefit due to novelty. Whatever was going on, it’s long been clear that it’s easier to design a virtual space to make people worse.

A lot of folks we met near our land beheld apparitions. I’d walk home from school along an irrigation ditch, so I’d come across men taking breaks from working the land. They’d talk about the weather or cotton prices, but just as often about miracles.

“You know Alicia, she almost died in the hospital, but the curandera, she said Mary would come to her, and then she came, shined like the sunset, and Alicia got real better. Now she’s bothering me every day. Like I don’t work hard enough.”

The story went on indefinitely. I’d wait for a moment when I could say ’bye, but there wasn’t one. You just walked on, maybe with a slight upward sling of the head, as if your chin was pitching an invisible ball.

The border region was crawling with spiritualists of every kind; evangelical, pueblo, Catholic, hippie. That could mean trouble. I once became enraged with a shaman from the Copper Canyon region of Mexico. Had a prosthetic agate eyeball and dressed in ribbons; claimed to have made contact with my mother, wanted money. I think he might have even gotten some out of Ellery; we were both still vulnerable, went through patches when nothing made sense.

At least you could trust the sincerity of murderous kids in a schoolyard. Friendly people could be sneaky. A difficult lesson.

There were also secular apparitions; a local culture of flying saucers. Kids would bring bits of fallen alien spacecraft to school for show-and-tell and no one questioned their authenticity, certainly not the teachers. We lived next to the largest missile test range in the world, so peculiar debris fell from the sky all the time. I still have beautifully machined satellite detritus I found in the mountains.

I never believed they were really alien, but I did find myself entering into the cult of local pride about our flying saucers. I still feel an involuntary surge of indignation whenever a rival town, Roswell, New Mexico, gets renewed attention for its inferior 1950s flying saucer crash landing. Our flying saucer crashes were better!

Whence

It must have seemed to Ellery that he had been preparing to live in New Mexico for years.

Before I was born, he had undertaken a wide range of careers simultaneously, just as I have. He studied architecture at Cooper Union and built skyscrapers with his father, who was also an architect. He also had a job designing window displays at Macy’s, and he and Lilly had shown their paintings—cubist—at a few notable shows.

Ellery had a mystical bent. He had lived with Gurdjieff in Paris and Huxley in California and studied with various Hindu and Buddhist teachers.

Hand in hand with his interest in mysticism, which he distinguished from superstition, Ellery liked confronting hokum. He was a minor radio personality in the 1950s, a semiregular panelist on one of the very first radio call-in shows, hosted by the pioneering broadcaster Long John Nebel, who was known for his interest in the paranormal.

They had great fun humoring crank UFO and paranormal enthusiasts on the air, but ultimately exposed con artists. Ellery wasn’t above making up nonsense as a gag, along the lines of the War of the Worlds radio show.fn2 He claimed to have made up the alligators-in-the-New-York-sewers urban myth. Could be so.

Once, he exclaimed on live radio that a noisy purported antigravity device had just maybe lifted a bit. He then had to explain it was a joke after callers took him seriously, but they couldn’t be talked out of believing.fn3

Ellery also wrote columns for Hugo Gernsback’s pulp science fiction magazines of the 1950s. He was, for a spell, the science fact editor of Amazing, Fantastic, and Astounding magazines. He’d explain the science relevant to the stories in each issue—for instance, the latest research on Mars when Isaac Asimov set a story there.