I work with a team of truly exceptional people. Of them, there’s one senior engineer (let’s call him “M”), who demonstrates such amazing clarity of thought that I can’t help but be awed.
When debugging a problem, M often formulates clearly defined hypotheses, tests them quickly, and isolates the root cause. Where most people (including me) are guilty of being satisfied with a “good enough” explanation for issues, M’s week of oncall often ends with us uncovering deep systemic problems, because he takes the effort to dig deep and arrive at the real root cause. His documents are also super easy to read, even though they usually deal with some of the most complex parts of our system – you get the sense that after reading a document, you’ve come away with exactly the understanding he wanted to impart.
I’ve had several conversations with M about his process and approach to thinking about problems. Unfortunately I’ve lost most of those conversations due to the super short Slack retention policy at work. But I’ve been incorporating those ideas into how I think about things, so I can still talk to my perspective on them.
How to test or debug a failing system
First, identify every single assumption the system makes. Then go over each of them and falsily them in turn.
For instance, when you’re testing a function, it might assume the input is valid JSON conforming to some schema. Well then, does it behave correctly (fail) if the input doesn’t conform to the schema (e.g. some field is missing); the input isn’t even valid JSON; or the input is null?
That’s a very simple example, but very often the problematic assumptions are quite insidious, and often occur at the interface between two features which individually work “correctly” or “according to spec” (read - Feature Interaction Bugs).
Getting good at isolating these assumptions is a skill that I believe can be honed by deliberate practice – I’ve started thinking about the systems I own and the interactions between them in terms of shared / conflicting assumptions, and (I think) I’ve gotten better at identifying problems before they occur.
How to ensure you understand the problem you’re solving
Quite often, more when I’m working on an exceptionally hard problem, I find myself in a quagmire of my own thoughts. I keep debating options, second-guessing my decisions, and don’t arrive at any conclusive next steps. This is especially hard when there are multiple domain experts and stakeholders I have to consult and get on board with my proposed solution.
What tends to help is if I physically distance myself – by going on a walk, say. Then I come back to write down everything I understand about the problem. More importantly, I write it down as if I were explaining the problem to someone else. This tends to be harder than you’d expect – and the harder it is, the more it’s indicative of gaps in my understanding of the problem. This then gives me a clear list of questions to chase down the answers to.
As a bonus, you can also play the role of the “other person”, and ask questions about assertions you’re making when articulating the problem. I’ve found this to be a great way to uncover non-obvious assumptions I’d been making.
Documenting your decisions
I’ve on multiple occassions found that I made a decision some months earlier, fully convinced that I was right, and on revisiting it now I don’t remember why I made that decision. I then spend more cycles re-evaluating that decision, and possibly arriving at a different conclusion. That burns valuable time and energy that I could have spent elsewhere.
Instead, I’ve found that if I document the original decision along with the reason it was chosen, then next time I look at it I can just skip the reverse-engineering involved in arriving at the original motivation.
As a bonus – often while making a decision I’d have considered other options which I then decided against. Recording those options, and why I decided against them, gives my future self greater confidence in the quality of my original decision because I don’t have to consider those alternatives again. Even better – consider recording what conditions would cause you to revisit this decision. For instance, what assumptions are you making which, if they no longer hold, would mean your current decision is no longer the best one.
Intuition vs rigor
I’m quite an intuitive thinker – I see connections, take leaps of logic, and arrive at conclusions very quickly. However, while intuitive reasoning is valuable when brainstorming, it is not really advisable to base decisions off of “gut feelings” or “instinct”. You see – since you’re skipping steps in your thought process, you’re more likely to miss details, or make hidden assumptions based on your biases Therefore, I like to split my thinking about a problem into multiple phases.
First, it’s important to define the problem and its scope as clearly as possible. You don’t want to spend time thinking about problems you don’t have to solve.
Second, a “brainstorming” phase. This is where I let myself think freely, without censorship or judgement, and just record all ideas as they come into my head, without evaluating their viability.
Third, and this is important, I follow a more “step by step” procedure of identifying any and all assumptions I’m making, listing criteria that I would use to evaluate each option, describing each option in detail, and then ranking them by how well they meet those criteria.
Conclusion
I have by no means tamed the maelstrom of my thoughts, and I still often find myself going around in circles. However, giving myself these templates, and following a more structure approached to thinking about problems, has already helped me get more concrete results (documents, code, etc.) from all the time I spend thinking. Hopefully the ideas I’ve briefly described here will help you too.