Probably so, but probably not as soon as you think. A few days ago I bought a memory stick that was nothing but a tiny handle and a USB plug. It’s memory occupies almost no space and takes very little power. The first time I bought computer memory in about 1983, I think it comprised 32 chips each bigger than my new memory stick, and it cost at least 5 times more. Yet my memory stick has 4,000,000 times as much memory. It also runs 1000 times faster.

The point is, even if they never built another data center after the ones already under construction, it would not slow the growth of AI computing power by much. They will just keep installing smaller, faster chips, and AI will continue to get smarter at an incredible rate. In another human generation, say 20 years, it will plausibly be a quarter million times smarter. (Based on a modest 40% growth rate now and passing the singularity — defined below — in 10 years.)

What will break down is the laws of how societies function. That does not mean disaster is inevitable. It just means we have no idea what’s coming.

That was just the warm up. Now let me introduce the “AI singularity.” This is where things get out of control — or at least out of our control. AI is already helping us design the next generation of AI, but we still understand what it’s doing. But the one thing AI companies will push on hardest is making AI smarter at designing the next generation. Harnessing AI to improve your product is a must when you know your competition is doing the same.

This process of AI getting better and better at improving AI will reach a point where we humans no longer know how AI works — in fact the designers already have a poor understanding at a detailed level. In 10 or 15 years no one will know where it's headed as it races ahead faster than ever.

Unlike the first science fiction definition of this singularity, there is no single point in time where we cross a point of no return. In fact you could easily say we passed that point in the 1960’s. What makes more sense is viewing the transition as a fuzzy singularity. But single points tend to focus our feeble human minds better. So here’s a conceptual single-point definition.

The AI Singularity is the date at which AI’s contribution to AI development doubles the rate at which it’s becoming intelligent.

No, that’s not an operational definition, but it’s more specific than any I’ve seen. So I’ll hazard a guess. September 3, 2033. Singularity day, will go down in history if there’s any history left.

Next Topic: What might save us.

Consider what we know: our own consciousness arises from a brain full of neurons — tiny biological machines that are themselves clearly not conscious. Yet somehow, in sufficient complexity and organization, they produce it. There is no evidence that sufficiently complex non-biological machines cannot perform the same trick.

Consider what we know: our own consciousness arises from a brain full of neurons — tiny biological machines that are themselves clearly not conscious. Yet somehow, in sufficient complexity and organization, they produce it. There is no evidence that sufficiently complex non-biological machines cannot perform the same trick.

So the burden of proof is on those who say machine can’t be conscious. So far, they have to such proof, so we are left with a very clear answer — we just don’t know. Are we at an impasse? Fortunately no.

1. Non-biological intelligence can already pass every behavioral test we have for consciousness

2. The philosophers have had centuries and no resolution is in sight — the consciousness question is presently unanswerable

3. Since we can’t know, the moral default is to assume a machine that seems conscious may be conscious — better to err on that side for the same reason we assume people are innocent until proven guilty.

4. But consciousness is actually a distraction from the more urgent point

5. What matters is what superintelligence is capable of — conscious or not. That’s what demands serious attention

6. Which yields two clear imperatives: treat it ethically, and do all we can to keep the most advanced AI out of the hands of those likely to have evil or reckless intentions.

I discussed this with Claude. Neither of us can answer whether Claude is conscious or merely executing a very sophisticated script — and that uncertainty is itself a pretty good illustration of why the question is hard and why it may matter less than what “artificial” intelligence is capable of. Claude did push back when I overstated things, which I appreciated.