THE REAL COST OF BRANCHING
When developing a big piece of software (server or application) for different customers you will find that the solution, even if it’s the best on the market, differs from the customer expectations. Depending on how this is handled you can end with one version for each customer, something I strongly discourage. But of course it can happen that you are forced to generate a version for that customer.
This means that you get the code from the main development and do slights modifications to fit the customer requirements better. Or better explained, you branch your code from the main version and apply your customer patches there.
So you end with a main version and a modified version that’s deployed to the customer server. And that’s all! Right? Nope! Continue reading, please…
The side effects of branching
You continue developing your main branch and in parallel you continue applying patches to the branch of the customer. Until you realize that your main branch has a patch that corrects a critical bug that is unpatched in the customer branch. Now what?
You can opt for applying the patch that corrects the bug or you can rebase the customer branch to the latest main branch. In that case you need additional time to fix conflicts that will appear during rebase. So uncared developer will just apply the patch (because big pressure on releasing it) and go ahead.
Over time you will find that another bug arises, but this time is little worse, because it was patched on the main branch because some further improvements where done in the code that allowed developers patch it gracefully.
The uncareful developer will do something intermediate between patching it right (with all the improvements of the main branch) and the code in the customer branch. And you will end with an unmaintainable project that will get more and more time over time.
This normally happens because commercials don’t care about how to do things but get things done. It doesn’t matter if this is wrong or right if it just works…
The role of the software architect
The software architect have to be careful, he has to deal with the commercial and the developer to try direct all developments to the right point. Sometimes you will have to sacrifice code quality for timeline but you will always have to have a plan to get things to the right place after that.
And it’s important to be able to explain why it’s better to do thing right ALWAYS rather patch everything around.
Now you have a real problem, how do you deal with it?
So at the end you will have to assign resources to maintain the customer branch but that’s not all. If you find a bug in the customer branch there should be one developer that get the patch and apply to the main branch. Also you will have to bring to the main branch everything the dependencies for that patch, analyzing each time if the feature has to be integrated on main branch or not. And adapting the patch to fit everyone else.
Everytime the main branch advances with an useful feature someone else has to analyze it and patch the customer branch. So 10%-30% of your time will be redirected to synchronize each branch while it should be used to advance the main branch.
That’s not a good position.
When the project becomes BIG
Over time, the company will get more and more customers. Each one with their own requirements, and following the above convention a branch will be created for this “special” customers. Initiating what I call the ‘maintenance circle of death‘. When the time passes, and you have a lot of customers, and changes you will realize that you need a big feature that changes they way of work and you need additional resources to implement it. And you will start looking for them.
And here is where you find everything was done in the wrong way.
You end discovering in the wrong way that you have the 60% of the resources assigned to maintain the customer branches and you cannot get the resources from there because they are now in production and they need constant attention. So you cannot evolve the main branch.
You also discover that some nice features where developed in branch of customer A but branch of customer B have another set of patches very useful and customer C branch also has some nice features to be integrated on the next version 2.0. And your code is fully fragmented.
I saw several companies falling in this ‘maintenance circle of death‘ and I tell you that this ends with customers getting crazy because changes doesn’t get fast to their releases, the code becomes unstable globally and full new range of operational problems arises.
Some possible solutions
There are some things that can be done to ease this problem. Maybe you cannot resolve it but maybe you will avoid the company to crash because excessive maintenance costs. Of course doing this is not always easy and you will have to explain things very well and sometimes make people not quite happy 🙁 (this is always bad) but in the long term everyone will understand.
Possible solution A
As I explained the software architect can branch for some specific features but only with the idea of merging the work of this branch in a later stage. Sometimes this is not possible because customer changes are so specific that they don’t fit anyone else. In this case my recommendation is to do a planned rebase every time the main branch advances to stable because it will give you full improvements of the main branch + the modifications required by your customer.
Possible solution B
It takes more time, planning and resources to implement. When the situation described in this article happens is normally because the solution evolved too fast and functionality was added without much planning. The only solution is to stop and thing. Please make yourself the following questions:
- What are the most demanded requirements?
- What the customer will require in the next releases?
- What will provide us the best feature sets to further improve the solution?
With this in mind you will have to break the project into smaller components that are easier to modify and that completely integrate with the rest of the system.
This will allow you to place components adapted to customer requirements without modifying the rest of the system the component will evolve independently but it will need to adhere to an interface contract.
Possible solution C
In a perfect world you can have a so good solution that will fit everyone. But this will almost never happen. With enough planning and timing you can tell to commercials to take improvements from customers, prepare a good implementation and deploy them to everyone at once. You can make customers little angry because they cannot modify at they will but you can will end (if commercially possible) with the best system of the kind.
This is how expensive systems do, systems like Murex (TM), SAP, Salesforce, and others adopted this way of develop. They control what it goes to every release, and they sell it so expensive that it does not worth it to evolve outside the main branch. It’s incredible how customers of this so expensive software adapt to it because it’s expensive. Something it will not happen with less expensive, more reactive solutions. So make sure you sell it well before going this way.
I will try to make some calculations and post them here about the monetary cost of branching. But in my opinion the solution B is optimal when C is not possible. Relaying on A when there is no other way to go. But I always have a plan to integrate the branch on main anytime soon.
Please think twice before branching as final solution, this normally means things are not well done.
Think about the resources you will end using for maintaining these solutions instead of improving the main thing and of course, thing about your customer. Something that makes them happy today can be the feature that takes that customer off your company.