MIRI explains that you don’t need a malevolent or even sentient AI for it to be dangerous and worth worrying about – ridiculing “a Hollywood-style robot apocalypse”. The real danger, they claim, is in our machines may simply become too good at science and planning and solving problems; whether they also become self-conscious is unimportant.
I’m all for ridiculing a Hollywood-style apocalypse but I can’t help wondering what “too good at solving problems” may actually mean. “Treating humans as resources or competition” – yeah, I get that, it’s obvious, but that would be simply bad (for us humans at least), not “so good as to be bad”. You don’t have to be smart to be mean; in fact, from what I’ve seen in life, the correlation is rather the opposite.
MIRI gives a glimpse at what they think is a likely failure mode of dangerous future AIs:
“A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable."
That's true, but pardon me, where's the "too good" part in this? Are they implying that an AI will be mindbogglingly, unimaginably, impossibly smart – and yet will readily commit such a dumb mechanical error of ignoring a part of the world simply because it was so (self-)programmed?
Isn't there, you know, a contradiction?
Admittedly, we cannot know what it means to be orders of magnitude smarter than the smartest of humans. We don't even know if it's possible at all. But I think a straightforward undergrad-level function optimizer, even with infinite RAM and clock speed, can be safely ruled out.
I was somewhat amused to learn, from a LessWrong survey, that a great majority of the rationalist community adheres to consequentialist ethics. It’s not that I would like to see them switch to some other system and not that I think it makes them in some way less ethical than I would like. It’s simply that consequentialism is not, you know, very rational.
It certainly looks mighty rational as you’re deducing consequences and calculating probabilities. In the end, however, it’s simply kicking the can down the road. Instead of deciding whether something is good or bad, you’re looking at its consequences. Fine, but why are those consequences good or bad themselves? Look at their consequences… and so on ad infinitum.
To be sure, looking at consequences is not a vain exercise. We do it all the time, whether we’re making an ethical choice or any other kind of choice in life. But it’s not really about ethics. It’s about steering your course – it’s what you do to leave rocks and sandbanks safely behind, but it’s not a way to figure out where you want to travel in the first place. To put it bluntly, consequentialism is not a kind of ethics; it’s just a tool to apply whichever real ethical positions you hold.
Mistaking consequentialism for a valid ethical position in and of itself leads to all kinds of paradoxes. For example, killing people is bad, but what if you save ten people for killing one? Not save ten by killing one, as war-on-terror apologists claim to, but simply save some unrelated ten people and then take this as an excuse to kill just one. That sounds wretched but it’s what you get if you stick to consequentialism: after all, the consequences of your combined action (saving + killing) will be overwhelmingly positive.
“This is a bullet I am weirdly tempted to bite,” concludes the author. “Convince me otherwise.” Well…
As for my own take on the matter, I call it live ethics. (Sorry the link is currently red, please check back in a few months.)
Here’s a bunch of Everday news for the holidays season.
Editing is final up to letter K inclusive. That’s about half of the book, just posted on the site. As usual, the new bunch has some fundamentals (intake, human scale, knowledge, home and garden) and a scattering of smaller juicy bits (guesthouses, hairtangle, grazing, haptic, game of i…). Enjoy.
Also, perhaps it is time to offer the book in more reading-friendly formats. The site is good for browsing and linking, but you may like to read it slowly offline or even print it. In fact, the book was written linearly, and it is intended to be read linearly, for there’s a certain flow and development to it (or so I like to think). Even an attempt at catharsis towards the end!
Here’s the PDF (half-Letter format), PDF (for small-screen devices), MOBI (for Kindle), EPUB (all other ebook readers), and a single-page HTML. These versions contain the edited first half; they will be updated as I post more of the book.
Also, there’s an entry for my book at Goodreads, which is a good site for book lovers. Please add your reviews and ratings there to help promote Everday.
Last and probably least, the book is also available on Amazon (as an ebook). There you get the entire book, including the still-unedited second half; however, there it costs about $1 (Amazon won’t let you sell anything cheaper than that). On the other hand, maybe these two inconveniences can balance each other out? Anyway, please add your ratings and reviews there too.
Oh, and just for completeness, here are my accounts on Facebook and Google+, mostly retranslating this blog. Feel free to friend or add or whatever.
There are two mysteries left in philosophy: time and consciousness. But, purely etymologically, only one of these can be a true mystery: time. “Consciousness” is too unwieldy—too artificial a word to point at something really profound.
Letters D, E, F are up on the site. Copyedit must go on. At rev 12979 now.
There are quite a few important things in this batch: feeleries (we haven’t reached Minds yet, but feeleries may be, in a way, even weirder), evolve, deep sleep, engaging, Fadeout, even Everday itself. And of course dreaming – “what is it but a dream?”
Enjoy!
I’m officially starting to publish Everday. A, B, C are up on the site. The rest will follow soon.
There are already some core concepts (Arf) as well as assorted soul candy (cloud hunting, badminton, Big Trees). But it’s just the beginning of the journey.
It’s going to be a hard plunge. Munching on this text has been central to my life for a long, long time. When the northern summer blooms – as it does now – each breath of forest and sea air is full of Everday for me. I’m going to miss it so.
But enough is enough, time to let it go. Let the cone unfold. Please read it, quote it, link to it. Ask me about it. Laugh or shrug.
Edit it, too. It’s a wiki for a reason. I’m not the real author, after all. You may be hearing something clearer than I could.
Just finished (rev. 12420) the first full pass of copyediting ever since I started writing the book. Previously I only did various overlapping patches and selections; now for the first time I went through all of it, from A to Z. (Well, to W, to be precise, as I don’t have anything at X, Y, or Z.)
That’s the good news; the bad news is that it’s still not finished. I already started yet another full pass, and it’s plowing almost as deep as the last one. Don’t really know how much longer it will take; I hope to release this year but no guarantees.
This copyediting stretch is proving more frustrating than I imagined. After almost six years of work, I can still easily see lots of edits that make it better. Which, of course, only shows how bad it was to begin with.
