It seems that not much people evaluate alternatives when choosing a technology. The situation gets worse when lazy developers are involved because we all tend to stay in our comfort zone and avoid new “ways of do” that can give us troubles in the future.
I totally disagree about saying inside comfort zone. I constantly get out to see how others do it. What are they doing and how problems got addressed. Normally learning a totally different way of addressing problems.
I find very important to evaluate and find the best solution for each problem we find. And where possible avoid all hype that tend to happen with new technologies.
In this article I will try to go thought innovation path we followed in one of our project explaining each decision and the outcomes of every decision. The good, the bad and the ugly…
WHAT I FOUND WHEN I ARRIVED THE PROJECT
When I arrived the project I found a project that was built on JBoss Server using SOAP services for the customer API, MySQL database using JPA as ORM. The configuration system was EJB over the JBoss server and the code generated for JPA had the annotation in the code. The project was built on MyEclipse and everything seemed to work well at that point.
I tried to get some justifications for the decisions and what I remember is something like this:
- JBoss was choosed just because financial market trust on it. Tomcat doesn’t seems to be too much ‘commercial’ for them.
- JPA was choosed because MyEclipse generated the code just well.
- EJB because it was already included in JBoss.
- MySQL because is a nice to use, reliable database.
- SOAP because also in financial market is a trustable tech.
- The rest was done over the road, because there was not much time to check for alternatives and current solution just works.
So the project was working well and why to change anything. Because reality is normally different than we think. When I took this project the first thing I had to do is to change a couple of tables on the database to fix a problem. But surprise! I wasn’t able to generate code anymore, and the people that did it doesn’t was there anymore. So I discovered that the MyEclipse environment used to build was a paid solution and only a few licenses available. So if you don’t have the license you cannot generate the code, and no licenses available. Bad news. I had to modify the code by hand instead of generating to introduce a patch.
So I decided to revamp the system to overcome these problems. The first decision was to move the whole project to maven. I found resistances saying that the MyEclipse solution just worked and maven was rather complicated. And it is for this kind of big problems. I spend some weeks educating team about dependency resolution, project decencies, profiles, and configurations. But it worth it.
I had to resolve also a dependency madness that maven helped a lot to resolve.
When maven was in place and able to build the whole project I moved code generation to Hibernate + JPA (to not change much solution), and code generation was done with hibernate-reveng and maven plugin. I used Level2 libraries that provided a lot more functions thanks to HibernateDAO library.
OUTCOMES OF DECISIONS
The project was dependency management properly done, it allowed us to upgrade libraries easier and build from command line. Hibernate Reveng allowed us to generate code for MySQL from command line perfectly and customize code from xml files. Everybody was able to build the project now.
The bad of these decisions was a greatly added complexity of the project and configuration. And new technologies to learn.
The ugly was that not all the people knew the technologies and the learning step was rather high, this caused team tensions for a while but after some weeks it was paid of.
Last thing to tell is that hibernate allowed us to do transaction management something was not properly done before nor understood.
PROJECT STARTED TO GROW
Because things started to work as expected the project grew fast. A lot of versions where generated and each developed had a different version from the rest of the team. Postgresql was added to be supported because hibernate was there. Compilation was more complex and took more time to build. That caused more troubles.
The project was shared over cvs as best (sometimes even zips).
We started to reorganize the project internal structure but also wanted to fix the new problems we where facing.
The first thing I did at this time is to move from cvs to git. This allowed us to have local repos where we were able to test our developments and merge branch before sharing to the rest. Also better management and a lot of useful stuff was added by git. Not to say that we found GitLab that’s the perfect tool for developing over git.
I also decided to go through CI and added Jenkins + Nexus to the equation. The system started to build automatically on every commit and the result of builds where left in Nexus for everyone. That made us improve support for our customers as every version was tagged and stored.
Just great. It started to see as a serious project at this time. Not without troubles, of course.
OUTCOMES OF DECISIONS
Not everyone was happy with Jenkins and Git. They tend to see them as something not necessary, complicated. Zip+Eclipse builds were just right. But that was not true. Jenkins allowed us to know exactly what was built and sent to the customer and Git is the best version control system I could find. For Nexus, that was another history everyone agreed it was a great solution because they saw it as a repository for all our code as a web (and not a samba share we already had in place). So was accepted almost immediately and looked just right.
The bad was that maven + eclipse where sometimes broken and Nexus code was not retrieved all the time correctly. So it caused a little pain on developers until everyone understood that building on maven and eclipse at the same time was not a good idea. A lot for clean/build cycles where added to development.
The ugly was when we discovered that team didn’t understand what is git and how it works. And team tend to wipe repo and re-download when something didn’t seem correct. Something was fixed by specialized training on git. And people discovered you can do all without loosing anything ever… That was a great jump forward.
BIG CHANGES, GREAT IMPROVEMENTS AND PROBLEMS
Great, project was launched. Everything seemed to work and development was progressing really fast. More internal deficiencies appeared. We where not able to fix some problems without extensive debugging live! Quality of the product started to decrease because the amount of changes, the time to implement and the poor testing we were able to do in that moment.
So I decided to stop the way things where done and introduce a test system. Until that moment every time we wanted to test something we had to go through the whole dataflow inside the system to get to the point it was failing. A lot of crypto was involved and was really difficult to test. I have to say that JBoss live reload of code helped a lot but was really painful.
Unfortunately JBoss and EJB didn’t help anything. It was impossible to bootstrap the system for running a test. Because EJB was not available outside the container. Also a lot of configuration was set by means of annotations on the code and was difficult to adjust.
My decision was to move the whole system to Spring, well known technology and a delight to use. I also moved the configuration to XML instead of annotations. Because changing the configuration over annotations required a build, meanwhile the xml was just a matter of writing to a file. I really cannot understand why someone will want to configure applications though annotations.
JUnit was then introduced to the team. And because everything was handled by Spring we were able to bootstrap only the parts that were required for the testing. That was really great.
OUTCOMES OF DECISIONS
Testing was greatly improved. Resolution times also improved. And develop time was sadly also bigger. Because now testing where created when required. The tests broke from time to time and required to be rebuilt. But also some tests ran for MySQL but not for PostgreSQL.
Spring allowed us to further configure and customize the solution.
The bad was that developers tend to forget to do the Spring configuration causing nasty problems difficult to track. Unit testing caused a lot of overhead in teams. But at least the quality improved a lot.
The ugly was because MySQL and PostgreSQL were not in sync all the time. And reflected the situation a lot of times. We discovered that a lot of problems where caused because the way the project was built and because how JPA annotated the classes.
We discovered that the system was tightly coupled. Transactions were too long, causing troubles in a multi-tier environment. And testing became more and more complex and difficult to do right.
We reached a point where the project was a pain to develop and maintain. A lot of code was duplicated because bad structure. Communication layer was difficult to maintain and highly coupled to the code. And monolithic approach caused one fix to break a lot of parts.
At mobile side talking SOAP with the server caused a lot of roundtrips between our C solution and the Java Android that talked soap. And communication between C client and SOAP server was cumbersome.
I took the most difficult and risky decisions:
- Move communication layer to Apache Thrift.
- Split the project and make it microservices approach. And to communicate microservices I decided to introduce RabbitMQ.
Apache Thrift was great. It allowed us to generate code directly in c, thanks to the c_glib library and Java. We were able to multiplex services over the same port. Be more responsive to customer requests and communications where faster.
Splitting the solution was not easy. A lot of architectural decision were made to split the right way. It caused new developers to start coding immediately a lot of common errors where avoided and transactional system was now properly done and effective. No more troubles.
RabbitMQ introduced a little bit complexity on clients. But it was paid of by the fact that we never loose a message. Every operation was processed. And we were able to decouple what the customer sees and what is processed internally. We where able plug SOAP, REST and Thrift on the front with 0 cost. Was just magical. Rabbit was so fast that we discovered that some operations where forwarded to the endpoint before the transaction of the previous tasks was persisted to database. Really fast!
OUTCOMES OF DECISIONS
Development speed was increased a lot because the rearchitecting. Complexity was decreased and the system performed very well. Moving to Apache Thrift caused a side effect I was looking for. Because Thrift manages the connections JBoss was not really doing much work. Remember that we moved to Spring before, transactional system can be managed by Spring also. Hibernate is bootstrapped inside Spring and at the end the communications where handled by Apache Thrift.
So why the hell do we need JBoss. The response is always the same. Because in the financial market JBoss solution is well trusted. But it’s not really required except to provide SOAP frontend. But that’s a different history…
The bad was that due to the full throttle we acquired developing the great amount of versions and developers involved the communication layers started to desynchronize. Because client was compiled against a different version of the server and problems arised. Nowadays I’m very happy with Thrift and surely we will use for other works. We want serialize and deserialize RabbitMQ messages over thrift (currently GSon). That will allow us to provide different protocols like Binary, Json and others. And switch them over configuration so we can achieve performance or debuggability depending on the case.
The ugly was caused because the MQ client was required to be reachitected several times to comply with our requirements. I discovered that create a client that’s performant, fault resilient, multithreaded and operational is not easy to do. I’m still on it. But it’s on a great shape.
THE FUTURE IS BRIGHT
Not all people understands the deep changes that were introduced on the project. The pains and the refactorings. But the effort was paid. The server is responding every request in less than 100ms (included HSM crypto). It’s very stable. Never crashed. And developing it is a pleasure.
The code is under control and new features are added every day.
The guys that are doing the Android part cannot cope with the full range of features we are adding and they decided to rearchitect their solution so they can code faster. It’s normally strange that the complex part (backed) goes faster than the frontend. But we achieved a lot on this project.
We don’t really know what’s the limit of the current system. But the good news is that we are in position to migrate to Apache Storm, Spark or whatever we need to continue improving performance.
We can remove JBoss and do load balancing by default over RabbitMQ without any effort.
We separated the way code is generated on the database, one single set of Java classes are able to manage every database supported by Hibernate. A project to authorize requests automatically was introduced. It’s called Keystone because does the same than the homologue in Openstack (still much worse out version) but enough for us. Unit testing is automatically done in every database supported. And data loading and database versioning is in place. An integration testing is ongoing. And the quality is superior.
We only miss to do stress testing to see what are the limits. We are on planning over our cloud… 😀
Comments are welcomed…