It happened again. Someone made a design decision was made that turned out to be wrong. Now we must go back and revert some work. How can we prevent this from happening again?
There are two solutions to this. First: training. We need comprehensive training to all team members so that everyone knows how to design software. This should be documented so that everyone can refer to this. It will serve as guardrails so that no one will make a bad decision again.
Second: design reviews. All decisions should be reviewed by peers and senior people. This will catch any errors and prevent bad decisions from being implemented.
Unfortunately, both these solutions are wrong. They won't work, at least not entirely.
The third option is: Embrace Failure! Mistakes are going to made. That's OK. We can always change and adjust. If we do need to revert something, we probably don't need to throw everything away. And even if we do, at least it was a good learning experience.
Even though training and reviews won't catch everything they are still worthwhile endeavours.
Software design decisions should never be based on opinion. However, they usually are. Decisions are based on emotions and justified with logic afterwards. Quick decisions are made based on feelings. This design does or does not "feel right". This "smells bad". An experienced developer over time does get a "feel" of what is good and bad. But this is tempered by sound principles and lessons learned.
Often, I will make a quick decision how something should be designed. I don't need to put too much thought into it. It becomes instinct. Only if I am challenged (in a review), will I go back and examine where those decisions came from. What software engineering principles formed those opinions?
All my decisions are based on sound software engineering principles. I welcome anyone that would like me to explain and discuss any decision I have made. This reasoning and debate are critical to growth, not only to me, but my team.
Disagreements should not devolve into "well, that's your opinion". "We agree to disagree". Let's bring in other people (often will less training on good software design principles) to vote on what course of action should be taken. Decisions should always be passed on sound software engineering principles, where everyone involved in the decision is aware of those principles and trade offs that come with them. Together the team should be able to make a decision that is devoid of opinion and emotion.
I have written a few public and internal blog posts or talks on how software should be designed. I can do some more, but I probably don't need to repeat the very good content that is already available on the internet or (*shutter*) in books.
A very brief and incomplete list of principles that I use is: 4 rules of simple design, Rule of three, Clean Code, cohesion & coupling, Software design patterns (not just the gang of four), anti-patterns, Domain Driven Design, Agile/Scrum/Kanban, Database normalization, concurrency, Continuous Delivery, compatibility, sematic versioning, Doyle Brunson's Super System, and Sun Tzu's The Art of War. In this blog post I won't go into details of what makes good software design, but understanding these principles is a good start.
Reviews (of all kinds, not just design review or code reviews) should be commonplace in an organization. They shouldn't be so rare that people feel like they are being pulled in front of the Spanish Inquisition when a review happens. Review should happen daily. They should apply to all levels of the organization. Even senior people should have their work reviewed before it is sent out or implemented.
The review process isn't a way to exert authoritarian control in an organization. The Agile Manifesto states we should value "Individuals and Interactions over processes and tools". You can implement a process of design reviews, but the reason for doing this is to ensure individuals to interact with each other. It is meant to encourage discussion and debate of design decisions.
There are different design processes. Most source control systems have a pull request process that requires a code review. The commenting system in the pull request is a discussion forum. The Scrum process has a review process for product owners to review the software before release, not to mention a peer Retrospective. The earlier you can put a design review in the process, the better. It is called "shifting left". Fixing problems earlier is usually less costly than fixing them later.
There are different levels of formality for reviews. A rigid "100% review all the things" policy shouldn't be necessary. For optional reviews it can be as simple as tagging someone in an email to solicit their opinion.
One way of speeding up the review process is to do a self-review. When you are making a design decision, list out what all the options are. List the pros and cons of each. Add a summary sentence at the top to state the decision you have made. Then share it. This is helpful to both more senior and more junior reviewers. Everyone can see the thought process that went into the decision.
Not everything you do needs to be reviewed. You are empowered to make some decisions on your own. But do be careful to know what your level of empowerment is.
Just because you were granted approval once, doesn't imply future approvals.
Was that first approval you received, begrudging or wholehearted? It is good to experiment. But following each experiment there should be a review see how effective the solution was for all involved. You may get approval to do something hacky given current budget constraints, but with the understanding that technical debt needs to be cleaned up later. Be aware of the vision for the product. Are the design decisions made moving toward or away from that vision.
There is a Rule of Three that states after you repeat the same process 3 times, it should be automated. If you believe you are approved to do the same job 100 times over, you probably should have been looking at ways to improve the process.
Don't stay in your pre-approval lane. This is also known as the Golden Hammer anti-pattern. Although a design might be a perfect fit for one scenario, this doesn't mean it should be applied to other scenarios.
Realize that everything is design. Let's say you need to store a user's password. The software already has a User Configuration screen. There is an eye colour field, which nobody uses. We can just store the user's password in that field. Then we don't need to make any changes to the software. No software redesign needed; no design review needed. Although the software may not be changing, you are redesigning the process in which that software is used.
Don't think that avoiding the review process means that you are more efficient. Always stay collaborating with your team.
Training and reviews won't catch all problems. So be prepared for failure. It will happen. It won't happen all the time. The successes will outweigh the failures. But still be prepared for when it happens.
Don't worry about blame. Everyone will cross a red line at one point or another. Anybody that is completely safe, isn't being innovative enough to be a good developer. Create a safe place where failure is allowed. Mistakes can be fixed.
Make sure people are learning from their mistakes. Mistakes are OK, but making the same mistake repeatedly, is very bad.
Create guardrails in your systems. Wrap your junior developers in bubble wrap and let them free on the system. These safeguards make sure they don't hurt themselves or others. These are metaphors for automated testing. If a junior developer encounters a failing test, they should take this as a clue that there is an aspect of the design they didn't consider. Don't just delete failing tests or change them for the new behaviour of the system without reviewing what compatibility issues that would cause.
You can't fix many problems with a new process (e.g. training and reviews). Firstly, it only works if every follows the process. Sometimes people aren't even aware the process exists. If everyone is using the process, over time people don't even remember why they are following it. They just following it blinding, which lacks innovation. Or they do finally abandon the process, which works fine for a while but misses that 1% failure rate the process was supposed to catch. Automated tests can catch those 1% scenarios that people don't typically think about.
Encourage training. The discussion that falls out of reviews often leads to better training. This will avoid many problems. But be prepared to embrace failure when those break down, which they will.