As AI gets better, and more and more the choice of whether you use it or not professionally becomes a “use it or be out-competed by those who do” type of not-really-a-choice, many are rightfully concerned that we’re outsourcing too much of our thinking to AI. Arguably, better technology tends to mean that humans can stop doing something they’ve spent a lot of time or effort doing before, like how washing machines mean that we don’t have to wash clothes by hand anymore, and it’s not like anybody particularly misses this ‘ability’. The argument goes like this: as better technology means we don’t have to know how to do some things anymore, then clearly those things are no longer necessary and are OK to forget.

I find it difficult to contradict this argument, but an observation I’d make is that while every individual step along this path might be perfectly justifiable and logical, we do lose something along the way, perhaps something that was actually necessary to be a healthy human. I think losing the ability to think and reason definitely qualifies, and Are we offloading too much of our thinking to AI? (discussed on Hacker News) worries about exactly that. I worry that this might just be another step on the path: mass and social media seem to have already done the bulk of the work. This step seems like it’ll be as inevitable as the previous ones, and I expect it similarly won’t bring any kind of instant doom: but it will be yet another step, and we might not like where we end up.

This worry is hardly new: the author’s post was triggered by having read The Perfect Match, a short story by Ken Liu from 2012 (!), which reads a lot less like sci-fi today than it was back then (a mere 14 years ago).

I heavily recommend reading the whole short story (it will take you about 20 minutes), but this part is what makes me feel like the step is inevitable:

“There’s no such thing as neutrally offering up information. If someone asks Tilly [an AI] about the name of a candidate, should Tilly bring them to his official site or a site that criticizes him? If someone asks Tilly about ‘Tiananmen,’ should Tilly tell them about the hundreds of years of history behind the place or just tell them about June 4, 1989? The ‘I Trust You’ button is a heavy responsibility that we take very seriously.

“Centillion [e.g. Google] is in the business of organizing information, and that requires choices, direction, inherent subjectivity. What is important to you—what is true to you—is not as important or as true to others. It depends on judgment and ranking. To search for what matters to you, we must know all about you. And that, in turn, is indistinguishable from filtering, from manipulation.”

[…] Without Tilly, you can’t do your job, you can’t remember your life, you can’t even call your mother. We are now a race of cyborgs. We long ago began to spread our minds into the electronic realm, and it is no longer possible to squeeze all of ourselves back into our brains. The electronic copies of yourselves that you wanted to destroy are, in a literal sense, actually you.

“Since it’s impossible to live without these electronic extensions of ourselves, if you destroy Centillion, a replacement will just rise to take its place. It’s too late; the genie has long left the bottle. Churchill said that we shape our buildings, and afterwards, our buildings shape us. We made machines to help us think, and now the machines think for us.”

This seems disturbingly true to me.