Bad Problems Don't Stop Being Bad Because Somebody's Wrong About Fault Analysis

It has been a while since I last updated this blog, and from here on I want to try writing a little more often. This piece is a lot like an earlier one of mine, "When Hope Runs Out, Action Begins." That one was also part translation, part recommendation of a good post from a foreign forum, with some passages translated directly but most of it being my own new writing. This time the source is an article from lesswrong.com that has stuck with me lately. I have put the link to the original at the end, so you can click through and read it yourself.

The original title on LessWrong is

"Bad Problems Don't Stop Being Bad Because Somebody's Wrong About Fault Analysis."

While I was writing this, I tried out a lot of alternative titles in Chinese, things like

"Understanding a structural problem can itself block your concern for how serious it is,"

"A gut sense of an urgent problem should not be overridden by rational analysis,"

"Rational analysis numbs us to serious problems."

But as you can see, the title carries so much in it that it resists being shrunk. Every attempt to abbreviate it just made it easier to misread. To put it the other way around: you can read the title a few times, and once you have it, you have already grasped what the body of this piece is trying to say.

So I kept it. What follows is the explanation.

Let me start with a few familiar little scenes.

Story 1:

Person A: Domestic AI companies care way too little about safety alignment and interpretability these days.

Person B: But given the startup environment here and how companies plan their resources, it makes sense that alignment ends up low on the priority list.

Story 2:

Person A: This news headline (or this article's headline) is really misleading, way overblown.

Person B: You know the author doesn't write the headline, right? Editors change those.

Story 3:

Person A: Universities are still teaching outdated material.

Person B: The problems with university education are systemic. There are reasons reform is so hard.

But what we care about is not the org chart. It is the problem itself, and the concern behind it.

Honestly, in daily life I quite enjoy playing the part of Person B.

Every time we travel, the older folks at home tend to marvel: "Whoever invented navigation apps was a genius."

And every time, I start to explain. I say that navigation is a feat of systems engineering, not something a single person could ever have built. On the pathfinding side, you can trace it all the way back to the Seven Bridges of Königsberg in 1736. On the side of observing the world, from mapmaking to remote sensing, there were technical advances and also a great deal of plain human labor. On the communication and software side, the fact that we can use navigation at all rests on the infrastructure and telecom built by large internet companies.

This is also why navigation, a modern technology, is unlike some other fields. Someone can be called the "father of rocketry" or the "father of the internet," but it seems almost impossible to point to one person as the "father of navigation software." The closest you get is Roger Tomlinson, the "father of Geographic Information Systems (GIS)." He proposed the idea of geographic information and built the earliest such system, the Canada Geographic Information System, for managing and planning natural resources. But that is clearly still a long way from navigation. It is only one of the several technical pillars holding navigation up.

By this point you have probably started to sense what is wrong with this mode of thinking, the careful, correct, reflective analysis of a problem.

It is interesting. It is intellectual. It lets you show off what you know, and it is a kind of mental exercise.

But it is very far from the concern.

If I keep following that thread, I can go on about graph theory and the whole history of computer science, about how modern internet companies work, how people cooperate, how a modern company makes a good product. But all of that is very far from the original feeling someone was expressing, and Person A does not care about any of it anyway.

The original post on LessWrong is shorter than this piece. The author's analysis goes roughly like this:

In these examples, people respond as if "to explain is to excuse," treating the descriptive fact of why something happened as if it answered the normative question of whether it should have happened.

This is a cognitive error or logical fallacy so absurd that I hardly know how to respond. In the examples above, people did not even start out by blaming the groups that others rushed to defend! And even granting that they did, how would shifting the blame solve the underlying problem?

The reasoning would have to run like this: "These people are (implicitly) blaming some group G for a supposed problem P. If I can show that they are wrong to blame group G, then I have shown they are wrong. And since wrongness is transitive, we can be confident that problem P is not real (???), so we no longer need to worry about P."

Maybe I am quibbling, but I genuinely do not understand the logic here!

The author also offers some ways to defend Person B.

Because, in fact, Person B's reasoning is correct and rational. And sometimes a problem really is unsolvable, in which case telling the person asking that the problem "cannot be solved" can help them face reality and turn their attention to things that are more tractable.

This short post set off a fierce argument on LessWrong. Some commenters said Person A was dodging the real complexity of the problem. Others thought Person B could have answered in a better way. Some felt the issue the poster found came down to sloppy phrasing. Others felt A and B were essentially worried about different problems.

In a place like LessWrong, the discussion a piece sets off can be every bit as good as the piece itself.

For me this topic has a particular pull. It sits right on the line between reason and feeling, carries a touch of self-reference, and lands close enough to my own life.

"Bad problems don't stop being bad because somebody's wrong about fault analysis" felt like the last piece slotting into place. It finally let me understand what I had been reaching toward, in my own thinking, while reading all these mental tools.

Let me walk you through a few of them quickly.

"Fill the canvas" tells us not to chase perfectionism in a first draft. Just do it, just finish, fill the canvas first.

"Action begins when hope runs out" says that despair is what drives renewal. Holding onto hope keeps you hesitating, and only once hope is gone do you finally start to act.

"Moonshot mindset" means not caring about how to get +10%, and caring only about how to get 10X, 100X, even 1000X. Set the target on the moon, give up incremental effort, and chase exponential compounding and return instead. That is where genuinely disruptive innovation comes from.

The common thread among these tools almost says itself. For a clever person like you, the thing that holds you back is "over-meticulous understanding and analysis," which tends to make you talk yourself out of something before you have even tried.

Structural analysis of a problem really is necessary. But it should not dissolve the problem itself.

You have pictured some bold move in your head, like starting a company, switching into a new field, raising an objection, taking a spontaneous trip, picking up a new hobby. And in the end the thing that stopped you was your own rational analysis.

It is necessary to dissect and discuss a problem rationally. But that habit slides very easily into an argument for the problem's impossibility, and from there into vetoing the proposal altogether.

It is like an immune system that overreacts. Analyzing a problem is, at its root, a form of self-protection. It calls on your reason to examine and understand the problem in advance, so it can offer sensible advice about what to do, and so it can let you back out reasonably when the problem turns out to be extremely hard or complicated.

But in the current climate, this turns very easily into learned helplessness. Right now most people face a life with few obviously workable plans for the future and few stable, predictable paths for personal growth. Especially after your own attempts hit a wall again and again, "reason" leans more and more toward giving you the avoidant answer.

So we become Person B. When we face a real predicament, we have a huge stock of theory and knowledge telling us this is a systemic, structural problem, that it has causes which made it inevitable, and that those very causes make any solution nontrivial and hard, that maybe there is no solution at all, or that trying to fix it will only make it worse.

So it is no surprise that people admire a brave, full-blooded act. Because it is exactly that slightly impulsive kind of idealism that can break free of the shackles of rational reflection.

Bad problems don't stop being bad because somebody's wrong about fault analysis.