This is a tale of CBRC's Linux operations transformation using a magic wand called GitOps@KAUST.

Imagine having to maintain more than one hundred virtual machines. Now imagine that each virtual machine belongs to a student who goes away from KAUST without leaving behind clear instructions on how to maintain what is running on those boxes. Did we mention that each virtual machine hosts a web site that has to always be available because they have been referenced in published papers? This would be enough to scare even the most tech-savvy administrators.

Our colleague, Antonio Arena (Linux & Advanced Platforms Lead, KAUST IT) met recently with Arnaud Hungler, a former KAUST employee,  who worked as a software engineer/system administrator in the Computational Bioscience Research Center (CBRC), CEMSE division. Arnaud reached out to the IT Research Computing for help and guidance to architect a solution that would allow him and who replaced him to maintain all these virtual machines.

Could you tell us what was your work history here in KAUST?

I spent my first three years working as software engineer for Prof. Stefan Arold. Then the director, Prof. Vladimir Bajic (aka Vlad in KAUST) called me to become CBRC's system administrator (sysadmin). I worked as sysadmin for two years. During my stint as sysadmin, I took care of almost all of the web applications developed by CBRC.

What job are you doing now?

I moved to Switzerland with my girlfriend. Thanks to the knowledge accumulated in KAUST, I now work as HPC (High Performance Computing) sysadmin and DevOps engineer at Genome Center - Health2030. I have to admit that KAUST was a great learning experience. I also want to thank IT Research Computing for helping me with this migration project that helped me get where I am today.

How come you reached out to IT Research Computing?

CBRC already uses plenty of resources provided by IT Research Computing. Prof. Vlad had resources (as former center director), Prof. Takashi Gojobori (CBRC's current director), Prof. Xin Gao, Prof. Stefan Arold, Prof. Robert Hoehndorf, just to mention a few; all these were and are using resources provided by IT Research Computing. So reaching out to the team who was already providing so many resources and helping us seemed the perfect fit.

What problem were you trying to solve?

We had more than 100 virtual machines that had been configured by students who had left KAUST a long time ago. These virtual machines were up and running but without any regular maintenance. This created a bunch of critical problems, ranging from applications going out of service (and nobody knew how to fix them), to security patches that would break everything, to security flaws. Some of these security flaws were actually exploited and led to severe security incidents. Trust me, you do not want to deal with issues like that!

What challenges did you face solving your problem?

I clearly remember two meetings I had at KAUST while I was working as sysadmin. The first one was with Prof. Vlad who told me to fix the 100+ virtual machines and migrate them out of the research data center (this data center hosts Shaheen and IBEX clusters) which at the time was going under maintenance for a few days each month. Our first solution was to migrate and consolidate all our virtual machines on two beefy virtual machines provided by IT Research Computing. At least, the host virtual machines would be maintained by someone else, and the data center was not shutting down every month. But the variety of applications meant a variety of dependencies. A huge mess to manage. A security flaw on one application would risk corrupting the whole host virtual machine. Not the best solution.

The second memorable meeting was with Hassan Al Zahrani and Nasr Hassanein (two of our greatest engineers who work in the Automation Team inside IT Research Computing) who showed me how to deploy a service on Kubernetes (Kubernetes is a container-orchestration system for automating application deployment, scaling, and management). At that time, I barely played with Docker, and thought it was a new technology hype that everyone wanted to have, to be forgotten two years later. Being nice, I obliged and tried it anyway. One month later, I had the goal to migrate the old unmaintained applications to Docker and Kubernetes. This also launched my current career.

Can you tell us what comes next?

I'm not in KAUST anymore so I cannot speak on behalf of CBRC. But I hope someone will implement a proper way for student developers to deploy their web applications on Kubernetes instead of their own workstation. Or at least make it work on Docker, and then a sysadmin deploys it for them.

GitOps@KAUST

IT Research Computing helped Arnaud by applying its GitOps knowledge and expertise to design a solution that would allow him and his team to easily maintain it in time, for the long run. Using these GitOps principles allowed who replaced Arnaud to maintain this environment without a sweat. All the configurations, codes, secrets, etc. are stored in GitLab repositories and pipelines. Everything can be recreated with one click. All steps used to create and deploy applications are outlined in the GitLab repository. Some other benefits of using GitOps are Docker and Kubernetes. Docker allows you to share your code with anyone without being afraid that it will not work on their computer. Kubernetes takes care of orchestrating these containers for you. It makes sure that your applications are always up, it can also spawn more instances if the load on your application is too high then deleting them once the load decreases. Arnaud's pipelines also made him and his team independent. They can push fixes and new features to their customers all on their own without any further help from IT Research Computing. Arnaud was able to delete more than 60 virtual machines converting them to containers running on our Kubernetes cluster. This is efficiency at its best!

Reach out today to IT Research Computing to automate your workflows.

Arnaud Hungler (in photo) has joined KAUST from France. Engineer, software developer then sysadmin, Arnaud now spends most of his time eating chocolate in Switzerland. Unfortunately, chocolate eater is not a well-paid job. He therefore chose to spend his extra time doing more sysadmin and DevOps stuff.

Article written by: Antonio Arena | July 26, 2020