Apache thrift for Android c_glib

APACHE THRIFT

 

When developing Web applications it reaches some time when you need to interconnect several systems. Sometimes you have a mobile application that needs to retrieve data from the net, or a client that want to interchange data with your platform.

 

Normally I should use REST, SOAP or RPC. But this means that overcomplexity and burden can make your platform to not evolve enough fast. Sometimes the language you are using to connet to the platform does not have bindings for the protocol you are using. This happened to me with SOAP and PHP some time ago. While it had complete support for SOAP it failed to provide WS-Security in a reliable way.

 

REST came to solve complexity and to easen things. But with REST it means you have to build a schema and the software to parse and ensure everything is right. I found REST to be most convenient way to do remote data interchange and Apache CXF helps a lot here. It provides REST and SOAP for the same price. If you do it right you don’t have to code anything more to provide support for both. And this is a plus. But for a banking environment REST is not a good choice. You know why? Because it’s simple!!! That’s the banking way to thing…

 

Normally with REST you encode everything as JSON and this means that with bigger and bigger data interchange you will have bigger and bigger messages. This is not scalable. But makes life easier.

This is where it helps Apache Thrift.

A good comparison is here. Even if Avro and Protobuf (specially this) are good choices I prefer Thrift.

 

WHERE DOES IT HELP?

Apache Thrift provides  the way to interchange data by using common communications protocol, decoupling data management from transport and providing bindindings for a lot of languages.

Basically you defines an schema and compiles it with thrift compiler. The result is a ready to use binding for the chosen language as you will do with SOAP and CXF tools. But language options are much greater and data interchange format can be much more concise.

And it does it all for free!!!!

We found a project that we can use to conver our XSD to Thrift IDL and this is great. But I have to modify it a little to handle better the enums. And maybe date and time types. Anyway it’s great.

There’s also a good IDL to xsd compiler. But we are evaluating if it’s better to handle XSD and convert it to Thrift IDL or the opposite. I suspect that XSD is more concise so maybe it will be a good choice use XSD for definition and the port it to Thrift. We only want to maintain one schema.

 

 

A PERFECT FIT FOR MOBILE

 

Of course you can generate code for Java (Android), Objective C (IOS), .NET (Xamarin Multiplatform) and many other languages. And use it natively.

 

But in our case this is a little more complicated. And for us Thrift is even better because we interact with netowrk in pure C for the ARM platform in Android.

 

Why do it instead of using just plain Java. Well, this was a design reason. We wanted to provide some lowlevel access to the operating system to our Java application and we wanted to do it fast and in controllable way. So we needed to code it with C. And Thrift was the only option to do this.

 

At the begining we did it with SOAP but we found there are not good SOAP clients written in plain C that can be ported to ARM platform so we did a trick to connect C with Java by JNI and this makes application less robust and slower. So now we are migrating. This will make us handle network the fastest way possible, in a more compact way and more reliable.

 

THE PROBLEMS

 

Of course no change comes with own problems. The support for C in Thrift comes with a little problem (or advantage) it depends on how do you see it. It depends on GLib. This means that you have to compile it with all requirements to make everything work on ARM. Your apk will take more space and you have more probability of breaking something. But as opposite you have several other functionalities for free:

  • Module loading (plugins).
  • Object oriented like programming environment.
  • Common libraries (hash, linked lists, date time functions, string functions, etc).
  • Threads and more…

Our idea is to rip off everything we don’t need and make it lighter for mobile environment. Because glib depends on gettext (not needed for us), libiconv (not needed for us) and libiff (maybe for module handling?).

But for now it’s okay.

 

 

 

COMPILATION

The compilation took us two days. It was a pain because no system is really ready for mobile environment platform. Thanks to everyone in the open-source community all libraries and dependencies where compiled with fantastic gnu autotools. It made our life a lot easier. But sometimes we had to touch config and code by hand to make it compile as we need.

The most important lessons learnt were:

  1. Sanitize your autotools environment before building.
    1. We had to use an script to setup our environment and ensure everything is in place before running autoconf script.
    2. Remove all libraries from host that can confuse autoconf. Sometimes it takes libraries from host instead of the ones provided for cross compile. And is difficult to make sure it does not got confused.
  2. Android need remove libraries version.
    1. We had a lot of problems because linux uses libraries versioning and android does not. If you don’t get rid of it android will not load library. Something like libiff.so.6 will not load. The same for the rest. And this name is encoded inside de library.
    2. Normally removing -version-info of the libtool is not enough you need to set the parameter -avoid-version because if you don’t do it, it will create something like libiff.so.0.0.0 that does not work.
    3. The libraries are not ready for this change (at least I didn’t find a common way to do it) so prepare to get your hands dirty.

Once we managed to handle compilation it was piece of cake. You just drop the files into your android project and everything got installed and ran as usual.

JNI interoperativity was great and easy and performance is so great…

 

WHAT’S THE RESULT

We have a lightweight Java Android application that handles all UI without any pain. And does it right. And a set of C functions in a SDK that gives us a lot of core functionality of our platform. We also have a set of JNI interfaces that makes communication between Java and C easy and reliable. And a fast, secure (SSL crypt) and performant connection to server.

We didn’t had to modify our server because Thrift gave us the template code to fill in, and we just got our code from MQ server/SOAP (it’s same basecode) and copy & paste. It worked like a charm!

We have several threads for serving data and read network (TThreadedSelectorServer) and works really well.

The use of glib gave us a lot of functionality we used to do by hand. And memory management that we can control better. We also are able to compile everything for the desktop, test on our x86 platform and take it to the android when everything is ready and bugfree.

We plan to put the binary basecode for Android somewhere in our page is you have interest. And build scripts in case you want to build yourself. When security is a must you should do it.

So tell me if you want more detailed instructions and I will write it.

 

Good coding!

 

Leave a Reply

Your email address will not be published. Required fields are marked *

*
*
Website