Is Strict Release Management Evil?
How does your current organization manage release? Do you have a strict release process like CAB (Change Advisory Board)? Or, you can make a release plan by themselves as long as the plan is published ahead of time? Or do you have no process at all?
I would like to talk about my opinion on how the release process should be.
Unexpected Side-effect of CAB
I would like to present controversial data which shows the fact that the release process by external members (typically called CAB) does NOT contribute to the success of changes. Rather, it results in slower MTTR (mean time to recovery).
Accelerate (Nicole Forsgren et al.) presents a large-scale survey that analyzing 23,000 data collected in 2014 to 2017. Their research found the key factor of high-performance organizations in system releases. Based on their research, they found the following interesting fact:
We investigated further the case of approval by an external body to see if this practice correlated with stability. We found that external approvals were negatively correlated with lead time, deployment frequency, and restore time, and had no correlation with change fail rate.
This is a very annoying result for the organization which relies on a strict release approval process. The strict release management won’t work. In addition, it can even make the situation worse. What the heck?
Gemba Knows the Change
Let’s calm down a bit. This is actually obvious. Who knows the change? The answer is Gemba(*). What is the scope of the potential impact of the change? How much the risk of the change is? Who is the most knowledgeable person? Who can make the right decision? Developers? Or external advisory board members? The answer should be clear.
Essentially it’s much difficult for external members to judge the risk of the change even if we put sophisticated people in the advisory board.
(*) Gemba is a Japanese word that means “actual place”. The word of Gemba is used in contrast to top management typically. In this case, the developers should know the change.
Release Management should Be Strict. But by the Wisdom of Gemba
I had working experiences both in a successful organization that is good at change management, and another organization.
In the successful organization, changes were managed by the wisdom and the leadership of Gemba. Release procedures were precisely reviewed by peers. High-risk DDLs were reviewed by DBA, and release times were carefully estimated and scheduled. Changes were pushed to the canary first, and fully automated tests were conducted before pushing the change to live traffic. If no issue detected after the test, the changes were finally pushed to customers. We sometimes failed releases but that did not happen frequently. So, nobody made complaints as long as we share the release plan ahead of time. Gemba could make a release plan.
In the other organization, I face multiple release failures because Gemba could not make appropriate risk evaluations. We pushed changes while we did not understand the impacts and dependencies on other systems. We mess up systems during critical sales events. As we made more incidents, the release process became much heavier. The issues were detected by the CS department instead of automated test case failure.
Based on those experiences, here is my conclusion. Release management should be strict. However, by the wisdom of Gemba.
We need trust from businesses and management to ask them to delegate the authority of release management. Gemba needs to take leadership in risk control as well as implementing better engineering mechanisms for safe releases. This requires the power and wisdom of engineering. If we get the trust, the organization can make a much stable release, much faster recovery from failure, and even much faster value delivery to the customer.
