I'm late to the thread but I've been reading Carlos Yu's comments. First and foremost, I'd like to congratulate him on his clear reasoning.

I think that the real problem is that we don't really have a good theory of consciousness, yet. All we know is that there is a type of structure -- the human brain -- which produces it. We really don't have a good understanding of how it produces it nor what the minimal constraints for producing it are.

I think that it's the minimal constraints that are really the question. If producing digital consciousness requires a molecular (or atomic) level simulation of a human brain, then we probably won't see it within our lifetimes, if ever. If, however, the hardware is just natures kludged and inefficient solution, perhaps we will find that it's reproducable with conceivable technologies.

To use an admitedly simplistic analogy, if I wanted to make a pump and my only knowledge of pumps were based on the human heart, lacking an understanding of what makes a pump pump, I might despair of ever making one. Paradoxically, the more I know about hearts, the more I might think that pump-building was impossible.

Once I managed to develop a theory of pumps, however, I would be able to see that pumps are simple even if the heart is complex.

Now, before anyone starts yelling at me, I recognize that there's a lot of holes here. For one, a better analogy wouldn't be creating a pump but creating an artificial heart (which is significantly more challenging); however, I think that the point stands. Having a good theory of pumps is critical to understanding how difficult the problem actually is. In like manner, having a good theory of consciousness is necessary to know whether the problem of reproducing awareness is a tractable problem. Dennett to the contrary, I don't believe that we have such a theory or are likely to for awhile.

That, alone, makes me think that there's not going to be a singularity in my future (except, perhaps, via the Intelligence Augmenting path that Vinge sketched out). For the larger question, though, I don't think we have enough information nor shall we for some time.

Andrew: I've yet to hear a convincing argument that consciousness exists, and/or that consciousness is an exclusive human trait to which no other organism has access. I suspect the pursuit of consciousness is actually a red herring on the road to AI, in much the same way that the pursuit of the Alchemists' Stone was a red herring on the road to the transmutation of lead into gold, and indeed, that the goal itself will (once we understand how it could be done) will prove about as desirable -- and about as closely related to useful activities.

Here's a question to chew over: why do discussions of digital consciousness so often to presuppose that any simulation of the human brain needs to be accurate down to the molecular level? Or that the brain is a digital instrument (when fairly clear cytological evidence says that it's a neural network with components which switch but have a rather analog-ish response curve)? There seems to me to be a huge disconnect between the neurobiology community and the AI folks, to such an extent that many of the AI folks need to do some remedial biology coursework before they open their mouths and start talking about the pros and cons of brain-level simulation.

Well, consciousness is a tricky word. Call it self-awareness, or a sense of qualia, or what have you. Of course, I think that this underscores my point that we don't have a good theory of the subject -- we can't even properly define our terms! I also certainly didn't mean to imply that it's, a priori, a human-only trait.

I would like to clarify that I am certainly not presupposing that it must be accurate down to the molecular level. My view is that we don't know how closely a self-aware "simulation" of a human mind has to resemble the actual wet stuff that our own brains use. I suspect that there's a lot of non-optional complexity that would need to be accounted for, and that not all of it has to due with simulating neural architecture, per se (the whole subject of how the brain interacts with our glandular system comes to mind), but I also suspect that there is a lot of room for simplification. I doubt that we need to be running quantum-accurate simulations of virtual neurons (but I could be wrong).

I really doubt that the brain is digital. The question is how we can map minds (or something sufficiently mind-like to satisfy our desires) to a digital environment.

I do agree that there is a big disconnect between the two communities. A huge one. Right now, I think that the main contribution to the subject, from the AI camp, has been to help us pin down what intelligence isn't rather than what it is. I think that's a useful contribution, but not quite what most dedicated AI theorists really want.

I would be curious to know what your particular doubts about the existence of consciousness are and why you think that it's a red herring. I'm particular interested to know why you suspect that the goal won't be desirable. Could you please elucidate?

Charlie, it seems to me that your two questions answer each other. Or at least represent opposite poles. One lot of disputantes want to simulate down to the molecular level because the other lot insist that the secret of human consciousness resides there. (And when the AI advocates say, okay, that should be possible in another (quick calculation) 25 years, the priests of human consciousness try to retrench and proclaim that consciousness depends on spooky quantum effects in the microtubules, a la Penrose.)

For myself, layman that I am, I feel that it should be possible in principle to create an AI without directly simulating neurons at all, once the neurobiologists have a better handle on how neurons build the effects that build intelligence. But it will probably be easier to model neurons first. When they've modelled a few seaslugs' and nematodes' nervous systems we might have an idea of how closely that analog-ish response needs to be modeled before it visibly impacts on the workings of a brain simulation. 32-bit? 8-bit? 1-bit?

I say "intelligence" above, rather than "consciousness", because it's my feeling that consciousness is an emergent property of having a load of parallel processes working together. A system capable of doing all the things that we would expect an AI to do, would of necessity have a consciousness, otherwise it wouldn't work at all. A conscious machine will not be created by trying to directly program consciousness; it will come from trying to program a machine to walk and chew gum (and monitor its energy usage and perform low-level maintenance routines and think about a problem and avoid mashing puny humans underfoot and...) at the same time.

Eric:

>Accelerando. Yum. Must buy copies for friends...

-- I'd second the notion.

>Stirling, that's an interesting analogy. But transportation hit an energy wall, I think--technologically, we could go faster, but who wants to pay $8,000 for Concorde tickets to save four hours on a Paris-to-New-York flight? Further improvements in transportation speed would need to be accompanied by declining fuel prices.

-- that's sort of the point; but we didn't just reach an energy wall, we hit a whole _series_ of walls in flight technology. Materials, cooling, steep upward curves in the energy cost of each additional mph, and so forth. The radical increases in capacity (and in power-to-weight ratios in engines, airframes, etc.) simply stopped dead between the mid-fifties and the mid-sixties.

Since then we've been making incremental improvements, but nothing even remotely comparable to the introduction of the all-metal stressed-skin fuselage or the jet turbine.

The same thing happened in the other technologies I mentioned. Eg., small arms underwent really revolutionary changes between the 1830's and the 1890's.

Nations anxiously watched the new rifles their rivals produced, and leapt to copy their features; the advantages were potentially decisive.

1830-1890 is the span between the round lead ball, muzzle-loaded with black powder and ignited by a flintlock (Brown Bess)and weapons with modern effective ranges and muzzle velocities in the 2000 fps+ range. By the 1890's, the latest small arms were breech-loaders using box-magazine or belt feed, small-bore jacketed ammunition fixed in centerfire-primed brass cartridges filled with smokeless nitro powder, etc.

Just as early-21st-century weapons are.

Since the 1890's, no fundamental changes -- despite years and years of effort, we haven't got one single issue weapon using caseless ammunition, for example. A good armory workshop in 1895 could duplicate a 2005 assault rifle or machine gun without much trouble, given an example to reverse-engineer.

It would be somewhat heavier and somewhat less reliable, and if it had a plastic stock they'd have to substitute wood, but it would be the same basic animal and perfectly useable. The Enfield or Mauser engineer of the 1890's presented with an AK-47 would understand it instantly.

Going an equivalent distance back from 1895, and armory in 1785 couldn't even begin to make a Lee-Enflield or a Maxim machine gun. They probably couldn't even figure out how the ammunition worked, much less duplicate it; the composition of the metals and the fabrication methods would be similarly beyond them.

Now, perhaps computers will be the first exception to this rule. It's not inconceivable. But it isn't, on the record, likely.

In any case, the above points out that there hasn't been an ever-increasing rate of technological change.

Instead, separate technologies have gone through spurts of rapid change and have then reached a plateau of stability.

In fact, you can make a strong argument that technological change was more rapid and had a greater impact on society in 1900-1960 than in 1960-2005. Certainly the life of an average American or European underwent greater changes between 1900 and 1960 than between 1960 and now.

Charlie: I think the basic problem is that we don't know how our brains produce our minds.

And worse than that, we don't have any convincing explanation of what our minds _are_.

Our descriptions of our minds are at a crudely prescientific level, rather like Aristotelian physics. We do have some knowledge of the biology of the brain, but not at anything like the level needed.

Until we do really understand how our minds work, trying to duplicate them (or their more complex capacities) with computer simulations is pretty damned futile.

A metaphor I like compares it to filling an empty swimming pool with ping-pong-ball models of water molecules and then jumping in. That won't get you wet or let you swim.

We understand fusion reactions far better than we understand our minds, yet we're still struggling to build a working fusion reactor -- and success is still the proverbial 25 years away.

I think we ought to start by admitting that this problem is _very, very hard_ -- like controlled fusion, only much more so.

It's not comparable to our prototype reactors; it's comparable to someone in the 1820's trying to build a fusion reactor, or even just a bomb.

The whole field is probably a distraction from what we should be doing, which is trying to understand the basic phenomenon involved in consciousness. You have to walk before you can run.

For starters, comparing the brain to a computer is a metaphor, not a one-to-one mapping with objective reality. Strong AI is a literalization of a metaphor, which is fine for SF but a really bad idea in the outside world.

I think the basic problem is that we don't know how our brains produce our minds.

And worse than that, we don't have any convincing explanation of what our minds _are_.

Our descriptions of our minds are at a crudely prescientific level, rather like Aristotelian physics. We do have some knowledge of the biology of the brain, but not at anything like the level needed.

Yes, yes, and yes again.

Having said which, there are some promising signs out there -- but the developments that interest me are coming from the neurobiology community. For the most part AI in the comp. sci. field is enthusiastically barking up a variety of wrong trees, or reinventing the wheel. Meanwhile, we've got prototype artificial retinas undergoing human trials, we've reverse-engineered bits of lobster digestive tract control circuitry (and other odds and ends), IBM are trying a brute-force approach at simulating real chunks of human neocortex to get a feel for how they work, and so on.

This is so far away from the original 1950s AI program that it ain't funny -- but at least it's happening. (Some day I ought to do an alt-hist story in which Minsky's devastating demolition of the perceptron -- which stalled neural network research for a quarter of a century -- didn't happen.) Meanwhile I'm going to stick with the quote I stole from Edsger Djikstra for "Accelerando": "the question of whether a machine can think is no more interesting than the question of whether a submarine can swim".

Oh yeah, fusion: I don't buy your analogy. Fusion is 25 years away because we know how to do it, and we can make a stable tokamak; but it'll take 600Mw just to keep the bloody thing burning, which means a minimal power-production reactor will have to be scaled accordingly, i.e. a proof of concept commercial prototype will be in the 3-5Gw range, and there's no such thing as a cheap 3Gw power station. So it takes decades to put the financial infrastructure in place to spread the risk of building a wholly new and untried reactor that has to be 3-5 times the scale of a large PWR. I suspect a Manhattan-style emergency push would get us working fusion reactors inside 5 years, 10 at the outside ... if anyone had $100Bn to throw at it.

> Now, perhaps computers will be the first exception to
>this rule. It's not inconceivable. But it isn't, on
>the record, likely.

I agree. Computers, at least the sort made from integrated circuits, are quickly approaching that point. There are some speculative technologies that offer the possibility of going beyond IC limitations (e.g., spintronics), but I don't think any of them are going to be able to jump into the breach for awhile.

I do suspect that there will be future computer revolutions. Unlike guns and planes, there are a lot of potential ways to make a computer which offers the possibility of a series of dicontinuous computational revolutions spread out over time, each with their own S-curves.

We also need be careful to distinguish maturity of hardware from maturity of software. Even if computer technology stopped dead in its tracks today, there's still a lot of unexplored design space for software. I wouldn't be surprised if there were some radical approaches to increasing the effectiveness of our programming that are yet to be discovered.

I think that Singularitarians have staked far too much on Moores Law. I find it interesting that Vinge, himself, didn't. I think that Vinge was correct in supposing that the real catalyst of a singularity, if one should occur, would be the production of greater than human intelligence. Vinge's paper on the subject suggested that, perhaps, a more plausible way to achieve this would be via intelligence amplification (IA) rather than via AI. I, personally, find that a compelling hypothesis, particularly since there's a number of unexplored avenues by which that could be approached (mind/machine integration, genetic engineering, etc).

>I wouldn't be surprised if there were some radical approaches to increasing the effectiveness of our programming that are yet to be discovered.

-- quite true, but not, I think, relevant to the "singularity" thing.

To be absolutely frank, I think the whole "singularity" phenomenon is merely a secularized version of the old religious longing for transcendence; transcendence of mortality or of material existance and its limitations.

The desire to "break on through to the other side", as the song had it.

But there _is_ no other side.

I don't find the religious version either attractive or convincing, and it's no more so dressed up in a lab coat rather than a cope and mitre. Sort of like a SFnal version of "intelligent design".

>that, perhaps, a more plausible way to achieve this would be via intelligence amplification (IA) rather than via AI.

-- the problem with that is, first, that once again we don't understand the mind. How can we amplify it if we don't know what it _is_?

Second, there's a good deal of epistemological and other confusions among those talking about it.

Eg., there's a perennial favorite, a neural link to access vast stores of information.

If you think about this for a moment, you'll realize that this is only marginally superior to the Web, or for that matter to having a big reference library.

If I have a math textbook, I can look up how to do matrix algebra. But until I _understand_ it, it's so much gobbledygook to me. Information still has to be processed through my consciousness before it's useful to me.

Eg., I understand the Hundred Years War fairly well. That isn't because I've got copies of several good books on it sitting near me, though I do, and I use them to look up details. It's because I _read_ the books, discussed them with people who knew the material, and thought about them.

To give me "instant expertise", an IA system would have to simulate the results of this process.

>genetic engineering, etc).

-- now, genetic engineering is just getting going. From that, I expect really dramatic results over the next generation or three.

>Having said which, there are some promising signs out there -- but the developments that interest me are coming from the neurobiology community.

-- well, sure. It's logical that if you want to know about minds, you should investigate the structures which actually produce them (brains) rather than those which never have and probably never will (computers).

It's important to remember that comparing the brain to a computer is a _metaphor_, not a description.

In the 18th century they compared minds to mechanical clocks. In our time, we compare them to computers. But minds (or even brains) are neither clocks nor computers, and literalizing the metaphor, while a staple of SF, is a really dodgy thing in out here on Planet Reality.

Since the brain is a material object, I expect that _eventually_ we'll be able to duplicate its functions.

(Unless we're simply not smart enough, the way a gorilla isn't smart enough to understand a Coke machine, though it can learn to use one.)

However, I also strongly suspect that when we learn to duplicate brains, we'll find that doing so is largely pointless in the larger sense. Useful for some things, but not world-transforming; less significant overall than the IC engine.

>For the most part AI in the comp. sci. field is enthusiastically barking up a variety of wrong trees, or reinventing the wheel.

-- agreement; see above.

>Meanwhile, we've got prototype artificial retinas undergoing human trials

-- one thing I do expect to see is a lot of "telepresence" stuff hooked into our neural circuitry one way or another.

Artificial vision will start by restoring sight to the blind. And hearing; so much for "Deaf Culture", one of the more memorably silly concepts of the past generation.

But then it'll get better than the natural equipment, and you'll be able to plug into machines that convey your senses somewhere else.

I suspect that military applications will drive the field. Soldiers will remotely-operate weapons and vehicles. That's already started, at the screen-and-joystick level, and it will advance quickly.


>i.e. a proof of concept commercial prototype will be in the 3-5Gw range, and there's no such thing as a cheap 3Gw power station.

-- no, but the larger power-plant complexes have been in that general range for some time now. The basic reason, as you say, is that the necessary research is expensive.

Myself, I think it would be worth financing on a shorter timeframe just to watch the Saudi princes' faces.

>I suspect a Manhattan-style emergency push would get us working fusion reactors inside 5 years, 10 at the outside ... if anyone had $100Bn to throw at it.

-- probably right.

There are two examples which I like to quote when people talk about historical discontinuities.

The first is a letter from about 1,800 BCE, discarded unopened back in the Bronze Age and discovered by archaeologists in Mesopotamia. They eagerly broke the clay envelope and found that the text began:

"This is the third letter I have written you about the silver you owe me for the sheep..."

The other occurred in central New Guinea, where one tribe previously unexposed to Western civilization was first contacted by a light aircraft. These people were so primitive that they were still making and using polished stone tools -- they hadn't even gotten metal implements by hand-to-hand trade.

A day after the plane landed, the 'big man' tried to use an offer of pigs to get the pilot to fly over a neighboring village so he could drop rocks on their heads to settle a longstanding feud.

From the Neolithic to Strategic Bombing in 24 hours...

On a related topic, the concept of "post-scarcity" economics is based on a similar misunderstanding.

"Scarcity" is a _relative_ term.

We're already "post-scarcity" in the modern West, in the sense that you can have a perfectly acceptable living standard -- better than what our grandparents usually had -- without much effort. In many places without any effort at all.

But of course 'sufficiency' is a moving target. We don't want what our grandparents had; we want what's available to us, and so we go on chasing the carrot, hoofs churning and teeth snapping.

This process can and probably will continue indefinitely. Wants are infinitely extensible.

>>I wouldn't be surprised if there were some radical
>>approaches to increasing the effectiveness of our
>>programming that are yet to be discovered.

>quite true, but not, I think, relevant to the "singularity"
>thing.

I don't know if it is or isn't. I am merely pointing out that the even as we're approaching some rather daunting hardware limitations with our computers, we'll still have a ways to go, yet, before we have really plumbed the limits of their capabilities.

>To be absolutely frank, I think the whole "singularity" >phenomenon is merely a secularized version of the old >religious longing for transcendence; transcendence of >mortality or of material existance and its limitations.

Bah. I've heard this claim made again and again and I don't buy it. Technology is about allowing us to achieve our wishes. When people say that the idea of the singularity is bound up in religious longing I think that you could just as easily say similar things about our desire to fly. Prior to our actual achievment of flight, one could have said that those persuing flight were just chasing after an Icarus myth, or some such.

Mind you, anyone who supposes that the singularity will allow for the violation of physical law is being slope-headed and I don't expend much patience on them, but I find that most of those theorizing about it are fairly commited to a materialistic world-view.

The central theme of the singularity is that there will be an explosion of intelligence. While I think that a lot of those hoping for a singuality are hyper-optimistic about what it will take to get a self-amplifying intelligence off the ground, I don't see any reason to suppose that it's impossible baring the conjecture that we're simply not smart enough to get to first base.

>The desire to "break on through to the other side", as the >song had it. But there _is_ no other side.

I've been following the idea of a singularity from the mid-eighties. Aside from Vinge's "Marooned in Realtime" conception (which is quasi-mystic) of a singularity, I don't think that many people really imagine a singularity as being like that.

>>that, perhaps, a more plausible way to achieve this would >>be via intelligence amplification (IA) rather than via AI.

>-- the problem with that is, first, that once again we
>don't understand the mind. How can we amplify it if we
>don't know what it _is_?

You don't always have to fully understand a phenomenon to improve it. Humans were building and using some rather sophisticated ballistic weaponry well before we have a good understanding of the physics.

In fact, I would say that the advent of writing is a good example of intelligence amplification that's already happened. It allows us to offload our thoughts into external storage and to more efficiently share those thoughts with one another. A mathematician with pen and paper is able to perform much more complex operations (and achieve deeper realizations) than an illiterate mathematician whose forced to keep everything in his own head.

>Second, there's a good deal of epistemological and other
>confusions among those talking about it.

I'd say that's true of nearly any topic.

>Eg., there's a perennial favorite, a neural link to access >vast stores of information.
>
>If you think about this for a moment, you'll realize that >this is only marginally superior to the Web, or for that >matter to having a big reference library.

I would agree that the degree of improvement is less than some imagine. On the other hand, efficiency of access to information does result in a quantitative (if not qualitative) improvement to cognition. If I can avoid spending five hours tracking down a tidbit of info, I have five hours to not only address the problem I'm trying to solve, but additional time to think up and address new problems.

>If I have a math textbook, I can look up how to do matrix
>algebra. But until I _understand_ it, it's so much
>gobbledygook to me. Information still has to be processed
>through my consciousness before it's useful to me.

Of course. However, again, if I don't have to hold all that information in my head all at once, it's much easier for me to make better use of that grey matter.

I'm a database administrator. My specialty is something called SQL Server (a Microsoft product). I seriously doubt if anyone on the planet has a full and complete understanding of its capabilities. However, no professional DBA needs to. You need to know enough to understand the core functionality and to understand the documentation but the majority of knowledge of the product is bound up in something called Books Online. Because of BOL, I don't *have* to know how to have the specifics of restoring the Master database comitted to memory. So long as I know how to find that information in BOL (and know enough to interpret it), I can confidently perform the action when I need to.

BOL makes me a smarter DBA. It's a qualified smartness, to be sure. BOL won't make me more creative, and any new info will still have to be integrated into my actual brain, but it still is a kind of external amplification of my capabilities.

>Eg., I understand the Hundred Years War fairly well. That
>isn't because I've got copies of several good books on it
>sitting near me, though I do, and I use them to look up
>details. It's because I _read_ the books, discussed them >with people who knew the material, and thought about them.

Of course. However, even though you've read them, I'm fairly certain that you have only memorized a fraction of their material. It's not necessary. You can concentrate on integrating the broad details of the 100YW while keep the books as references for the lower level details.

>To give me "instant expertise", an IA system would have to
>simulate the results of this process.

I don't think that one needs instant expertise in order to claim IA. One simply needs a system that allows you to think faster and more efficiently. I would also say that anything that improves our ability to share and distribute intelligence serves that goal (consider the fact that no one really knows, or needs to know, every bit of knowledge required to make a modern jet aircraft).

>genetic engineering, etc).

>-- now, genetic engineering is just getting going. From
>that, I expect really dramatic results over the next
>generation or three.

Perhaps. I think that GE is as subject to hype as any other technology.

"his is so far away from the original 1950s AI program that it ain't funny -- but at least it's happening. (Some day I ought to do an alt-hist story in which Minsky's devastating demolition of the perceptron -- which stalled neural network research for a quarter of a century -- didn't happen.)"

I'd be interested in knowing more about this. Do you have a link that you could toss me?

>Technology is about allowing us to achieve our wishes.

-- but it's not about "becoming as Gods", etc. Which, you must admit, seems to be the emotional driving force behind a lot of Singularity-worship. People who just can't face personal extinction and the ontological emptiness of the universe.

It makes my skeptic's nose twitch. Goshwow always does.

>I've been following the idea of a singularity from the mid-eighties. Aside from Vinge's "Marooned in Realtime" conception (which is quasi-mystic) of a singularity, I don't think that many people really imagine a singularity as being like that.

-- well, since Vinge _invented_ the concept...

>In fact, I would say that the advent of writing is a good example of intelligence amplification that's already happened.

-- oh, come now. That's like saying cars enable us to run faster.

>Of course. However, again, if I don't have to hold all that information in my head all at once, it's much easier for me to make better use of that grey matter.

-- not really true. You're talking about different brain functions.

So all in all, technology is _not_ on an ever-accelerating curve.

In 2050, I expect life to be about as different from today as today is different from 1953, when I was born.

That is, the difference will be substantial, but less overall than that between 1905 and 1955, and less than that between 1855 and 1905.

People will still be people, not IA or uploaded or nanotech Gods.

They'll drive cars, fly in commercial jet aircraft at about the same speed, and so forth. Most will live in cities and work in offices or equivalent, but far fewer in factories or on farms than today. Most will believe in some religion or other, and be under the political authority of sovereign states. Medical care will be much better than it is now, and lifespans somewhat longer. Computers and computer-like devices will be more powerful than now, and much, much more ubiquitous, but although your refrigerator may be able to tell you when the lettuce is going off, it'll still be a refrigerator.

There will be more people than there are now, but not a whole lot more, and the world population will probably be dropping.