Is Strict Release Management Evil?

How does your current organization manage release? Do you have a strict release process like CAB (Change Advisory Board)? Or, you can make a release plan by themselves as long as the plan is published ahead of time? Or do you have no process at all?

I would like to talk about my opinion on how the release process should be.

Unexpected Side-effect of CAB

Accelerate (Nicole Forsgren et al.) presents a large-scale survey that analyzing 23,000 data collected in 2014 to 2017. Their research found the key factor of high-performance organizations in system releases. Based on their research, they found the following interesting fact:

We investigated further the case of approval by an external body to see if this practice correlated with stability. We found that external approvals were negatively correlated with lead time, deployment frequency, and restore time, and had no correlation with change fail rate.

This is a very annoying result for the organization which relies on a strict release approval process. The strict release management won’t work. In addition, it can even make the situation worse. What the heck?

Gemba Knows the Change

Essentially it’s much difficult for external members to judge the risk of the change even if we put sophisticated people in the advisory board.

(*) Gemba is a Japanese word that means “actual place”. The word of Gemba is used in contrast to top management typically. In this case, the developers should know the change.

Release Management should Be Strict. But by the Wisdom of Gemba

In the successful organization, changes were managed by the wisdom and the leadership of Gemba. Release procedures were precisely reviewed by peers. High-risk DDLs were reviewed by DBA, and release times were carefully estimated and scheduled. Changes were pushed to the canary first, and fully automated tests were conducted before pushing the change to live traffic. If no issue detected after the test, the changes were finally pushed to customers. We sometimes failed releases but that did not happen frequently. So, nobody made complaints as long as we share the release plan ahead of time. Gemba could make a release plan.

In the other organization, I face multiple release failures because Gemba could not make appropriate risk evaluations. We pushed changes while we did not understand the impacts and dependencies on other systems. We mess up systems during critical sales events. As we made more incidents, the release process became much heavier. The issues were detected by the CS department instead of automated test case failure.

Based on those experiences, here is my conclusion. Release management should be strict. However, by the wisdom of Gemba.

We need trust from businesses and management to ask them to delegate the authority of release management. Gemba needs to take leadership in risk control as well as implementing better engineering mechanisms for safe releases. This requires the power and wisdom of engineering. If we get the trust, the organization can make a much stable release, much faster recovery from failure, and even much faster value delivery to the customer.

Global team builder from Tokyo. Engineering manager to build international engineering organization.