The KAUST Information Technology Department blog
06 September, 2020
KAUST gives its researchers a variety of technologies to help them achieve their goals. The Shaheen supercomputer is one of the most important of these technologies. This system allows you to execute calculations quickly. This also means that, depending on your workflows, you could generate petabytes of data. The problem that some researchers have is accessing those datasets to interactively pre- and post-process them. The best solution would be to leave your data on the storage medium and access it for visualization from systems in the data center. IT Research Computing's Remote Workstations were created with this scenario (and many others) in mind.
We met recently with Prof. Ibrahim Hoteit, professor of Earth Science and Engineering, PSE division; Dr. Peng Zhan, Research Scientist in Prof. Hoteit's group; Dr Sabique Langodan, Research Scientist in Prof. Hoteit's group; Yixin Wang, Ph.D student in Prof. Hoteit's group.
Prof. Ibrahim Hoteit: Good question! Somehow, we became specialized in a region more than a discipline. We started studying the Red Sea and we realized that its circulation is mainly driven by the ocean atmosphere. So, we also started working on the atmospheric circulation. We are interested in developing forecasting systems for the short- and long-term environmental conditions of this region. This requires expertise in applied mathematics, estimation methods, and statistics. We ended up with quite a multi-disciplinary group. We have expertise in ocean, atmosphere, applied mathematics, statistics, and the impact of these on the ecosystem. How do the circulations drive the ecosystem? We are interested in the large-scale picture; how the coral reefs or the Red Sea ridges are connected to each other through circulation because this is driven by currents. We want to find out how the ocean conditions drive the productivity of the Red Sea, the exchange with the Indian Ocean, the atmospheric circulation, how the global climate affects the Red Sea. In summary, we are interested in studying the ocean atmospheric circulations and variability of the Red Sea, in predicting its conditions for operational use by the government and the industry, and in understanding and projecting the climate impact on this region.
Yixin Wang: I am a Ph.D student in Prof. Hoteit's group. I study the circulation impact on the ecosystem. I am currently focusing on studying the Red Sea connectivity using a high-resolution numerical modeling and available observations. This project describes, in extremely high resolution, how two different locations in the Red Sea are connected through ocean currents.
Prof. Ibrahim Hoteit: We are interested in how distinct species of the Red Sea are connected through circulations in the Red Sea itself and environmental conditions. This is particularly important for designing protected environments, and to understand why these are genetically connected, and in case we need to protect coral reefs due to a projected bleaching event. This is quite important for scientists and decision makers.
Dr. Peng Zhan: I am a Research Scientist in Prof. Hoteit's group. My main research concerns the mesoscale activities in the Red Sea. We use numerical models to simulate these phenomena so we can understand the mechanisms, the impact on connectivity, oil spills; then we try to predict these activities. I am also working on the coastal circulation in the NEOM and the Red Sea Project (RSP) regions where we design very high-resolution models embedded with my existing Red Sea models. We provide high-resolution simulations for those regions to study their circulation and help in the design of the new projects along the coast.
Prof. Ibrahim Hoteit: Peng and Sabique are important researchers in the large ARAMCO funded projects for developing operational systems for the Red Sea and the Arabian Gulf. These systems are built to interactively predict oil spills on-demand. In two years, it will be made available so ARAMCO can predict oil spills on-demand. We need IT's help with this, big time.
Dr. Sabique Langodan: I am a Research Scientist in Prof. Hoteit's group. I work on developing a wave forecasting system for the region which will especially be used by ARAMCO. This system also includes oil stream modeling. My expertise in this project is ocean, waves, and winds. I work on the prediction of waves and winds and their variability over the region. I also study how global climate changes affect the parameters in our region.
Prof. Ibrahim Hoteit: We are trying to use all available information for this region; from in-situ observations, satellite measurements, and also numerical modeling because observations in time and space at these large
scales are scarce in this region, where it is almost impossible to find historical data. So, we are trying to use whatever available information we can get our hands on to study and understand how the Red Sea system
functions, how it is affected by the climate, how the circulation drives the ecosystem, and can we predict these conditions and circulations on short-term scales for operational use and also on long-term scales
to see how the system reacts to climate changes. We are also interested in what we call the sub-seasonal to seasonal forecasts. Can we predict the weather for two weeks up to three months? Basically, can we predict the weather for a season? This
is important to manage water resources, agriculture, or for risk management. We have become experts of the environmental conditions
in this region. We are of course doing that for science and also to give back to this region by developing and providing necessary tools and information to support the management and future development of the region.
Yixin Wang: I am going to mention an example of one of my challenges. I am investigating the high-resolution connectivity in the Red Sea. We use a distance-based matrix to describe our model. Each element in this matrix describes how two specific locations in the Red Sea are closely related. However, due to the high resolution (we are talking about a one-kilometer resolution) of our models, this distance matrix ends up huge in size; up to 15000x15000 rows per columns. I cannot load this matrix into MATLAB on my local workstation even though I have 256GB of RAM installed on it. I need to load the matrix, even if it is only an intermediate step because I must inspect the data. Sometimes I can load the matrix into memory, but it slows down my local workstation to a halt. I had to use IT Research Computing's Remote Workstations (author's note: these nodes have a minimum of 512GB of RAM up to 1TB of RAM) to work on my data. These workstations allowed me to continue inspecting my data so I can later on extract the part that I need and save it into a smaller matrix that I can now load on my local workstation. The Remote Workstations made my life easier.
Dr. Peng Zhan: We study the Red Sea, but we do not have that many observations. So, we must use numerical models to simulate a high-resolution environment to get a clearer picture. Our models generate lots of data, and we often output results for an extended period. Our biggest challenge is moving data out of Shaheen so we can pre- and post-process it. We used to move the data to our local workstations but copying data was slow and led to data duplication. The Remote Workstations helped us solve these problems. We leave the data on Shaheen and access it through extremely fast networks (author's note: 10G connections setup by IT Networks Team) available on Remote Workstations. We just load the data from Shaheen into the software running on Remote Workstations.
Dr. Sabique Langodan: I face the same challenges as my colleagues. We run our simulations on Shaheen. These simulations are quick to run (Shaheen is a crucial resource for us) but we generate lots of data. I need to visualize that data somehow without having to wait hours to transfer it to my local workstation. The Remote Workstations saved the day.
Prof. Ibrahim Hoteit: In a nutshell, we have around six petabytes of data stored on Shaheen. We limit ourselves as much as we can from generating further data otherwise, we would take over all Shaheen's storage. Even limiting ourselves we are at six petabytes. We were approved for an extra six petabytes of storage medium that we are waiting to arrive. This is a big challenge for us. We would like to run ocean-atmosphere coupled systems in ensemble mode and this requires much more space. Our technical challenges can be summarized as: compute time, data storage and analyzing these large data sets from our local workstations as Yixin mentioned. Scientifically the lack of data is a big problem. The complexity of the system we are dealing with in the Red Sea is also a big problem because it is dominated by a mesoscale activity as Peng mentioned. Simulating and observing this is quite challenging. And the system is also coupled in every way, ocean and atmosphere that drive the ecosystem. The complexity of this system makes it also difficult to understand. It is a small sea but as we say in French: "elle a tout d'une grande".
Prof. Ibrahim Hoteit: The first thing that we did, spending our first four/five years, was generating historical data. You want to know what happened in the past in this region, so we can study it. You need to find out how the system functions: seasonally, inter-annually, even decadal trends. The Red Sea lacks long-term datasets. You can get some proxies from corals but only in some locations. You can also use satellite data, but the information is limited to the surface and the last two decades. We collected whatever data was available then we generated a climatology of the Red Sea conditions from ocean, atmosphere, and waves by combining all available data and numerical models; we generated roughly forty years of data at the highest possible resolution. We look at the models as extrapolators of the data; backward in time (i.e. historically) and also forwards (i.e. to forecast). Whatever we do is driven by available data. So, the models are not just allowed to run free without looking at what is really happening and how they are representing the reality. We have the best available products for the region for forty years climatology. We generated this data to study the circulation as I mentioned previously and to understand the system. The big projects came along, e.g. NEOM, RSP, etc. These projects need to know the environmental conditions to design constructions, facilities. Since there is no available data, our products are the best to optimize their constructions. A real-world example is KAEC. They were interested in the highest sea level and wave that could arrive at the coast because accordingly they will protect the coastal area. They were about to put four meters of sediments but according to Sabique's study we found out that over the past forty years, one meter could be enough. So they reduced their plan to two meters, saving them about half billion dollars! Knowing the conditions allows you to optimize resources and constructions bringing huge savings.
Prof. Ibrahim Hoteit: We can not be experts in everything. We of course collaborate with many Universities. I can cite Scripps Institution of Oceanography and Woods Hole Oceanographic Institution; these are the two most prestigious Oceanographic Institutions in the world. We also work with MIT for forecasting, Cambridge on downscaling, TUDelft for how the climate affects the coast on very high-resolution scales. The work with TUDelft is part of the new Center of Excellence in NEOM. We are leading the component of modeling at the urban scales in collaboration with Imperial, e.g. how the wind moves between the buildings. This work avoids creating heat islands in a city or to avoid pollution stagnation. As I have said multiple times, not only do we try to understand the functioning of the system but we also want to have real world applications of our studies. We also work with Plymouth Marine Laboratory, University of Athens, and University of Essex on super high-resolution remote sensing of the Red Sea coast. We have a lengthy list of international collaborations. We cannot do it by ourselves. KAUST's connectivity and funding system allows us to connect in a wonderful way. IT is strategic for us; we need to access our data in a secure way, we cannot move it out of KAUST. That is also why I always ask for VPN accounts, people from outside need to access our data to collaborate with us.
Prof. Ibrahim Hoteit: The core of our computations run on Shaheen. We use IBEX for the GPUs. A part of the team does visualization and Machine Learning. Sometimes we also use IBEX for smaller jobs. On Shaheen we can run our big models plus it has more space to store our data. Our main problem is storage rather than computing, thanks to all those cores on Shaheen. Our work does not end with the simulation but analyzing the results to understand what is happening.
Prof. Ibrahim Hoteit: Let me summarize the situation. We developed unique modeling capabilities for this region. We also developed a climatology that allowed us to understand how the Red Sea functions; we have generated data useful both for science and for industry. In the short-term, we are developing the first operational systems equipped with online interactive visualizations for ARAMCO, but this system could be useful for the whole Kingdom.
The next step is to expand domain wise. We are going to the other side of the Arabian Peninsula. We will expand our system to include the Arabian Gulf and the Arabian Sea. We will also develop a fully coupled ocean, atmosphere, and waves system. This should allow for better understanding of the interactions between ocean and atmosphere. We will definitely need more computing and storage capabilities to achieve further high resolution with these new larger and coupled systems.
We also want to push the predictability of the system. How far can we predict? Now we can predict up to one week; sometimes two weeks. Can we push this limit to one season? One-week predictions is what we are implementing for ARAMCO today. The sub-seasonal to seasonal predictions are in a development phase. We have a project with the University of Arizona where we are studying the predictability of extreme rainfall events over seasonal scales. Can we predict these extreme rainfall events over Jeddah two to three weeks in advance to avoid human deaths? This would provide authorities enough time to prepare for these extreme events. We are also using Artificial Intelligence for these predictions trying to link them to some large-scale circulations in the global climate. How is our region linked to the global climate? We are trying to build the whole puzzle. We understand our region well but now we must link it to the changes in the global climate.
We also want to make our six petabytes (soon to become twelve) accessible even to non-experts using visualization. We have two researchers dedicated only to making data easily accessible online to the community. I guess this will be the topic of discussion for another User Story :)
Yixin Wang: I would like to use MATLAB interactively while running calculations on Shaheen. Today I must submit my job then check the results once the calculation is over. I would like to use something like Jupyter Notebooks that run on Shaheen but allow to interact in real time with my data. On the other hand, I realize that real-time interactions from MATLAB are almost impossible to realize; thus if interactive MATLAB on Shaheen remains impossible, moving on to Python/Jupyter Notebooks for data analysis could be a potential solution for me.
Prof. Ibrahim Hoteit: I want to mention something that people outside our group do not realize. It is not only what we have developed over the past ten years. We are also building on top of what has been developed in the past forty years outside of KAUST by other research groups. We reuse or extend what already exists. This means we are not very flexible moving away from our core programming languages. I know GPUs exist, but I cannot afford to move our codes on them; it might take us ten more years to adapt it. Then a modern technology might supersede GPUs, so you see we cannot afford following the latest trends. It is extremely challenging for us to move to recent technologies. We are a FORTRAN and MATLAB powerhouse. This does not apply only to our group; many research groups suffer the same fate. If someone tells me to change language, I might consider quitting :)
Dr. Peng Zhan: We are expanding our region and moving more into predictability. We are also trying to apply this to oil spills, plumes, and support coastal engineering requests. My challenge to approach these will still be how to get a fast and reliable connection from the Remote Workstations to data on Shaheen. Other than the computational part, this is the most time-consuming part for me. I hope that a maintenance session on either Shaheen or Remote Workstations would never kill the connection as it has sometimes happened.
Dr. Sabique Langodan: I agree with Peng, high speed, and robust connectivity between Remote Workstations and Shaheen are vital for our research. I want to thank IT Research Computing because it was exceedingly difficult at the beginning for us to connect to data on Shaheen. Once the Remote Workstations were introduced, they made our life easier because we could access the files for pre- and post-processing as if we had them on our local workstations.
I would also like to ask if you could reduce the maintenance windows on the Remote Workstations. I think I can also speak for Peng; these systems are the most security patched ones in KAUST!
Yixin Wang: I second what Sabique just said. The Remote Workstations solved most of my problems; it complements my computational needs on Shaheen. I want to emphasize how much I appreciate your help and support with the Remote Workstations.
Let us take a stroll down memory lane. Antonio Arena was reselling solutions like Remote Workstations to many Oil & Gas companies in Houston before joining KAUST. He asked if this solution would be useful here in KAUST. Mike Young (we hope you remember Mike; he is missed in KAUST, NVIDIA is happy to have him now) jumped on the idea right away. He suggested as first customer Prof. Ibrahim Hoteit. Prof. Hoteit was skeptical at first but was willing to evaluate it since he was seeking solutions to process data on Shaheen. Mike (with Nasser Mohiuddin's help) placed three physical workstations (shh...you are not supposed to do that, but Mike had wasta; thank you Peter Godsafe, the Data Center Lead at the time) inside the research data center. It is crucial that the Remote Workstations live in the same data center where storage and application servers are. Proximity and a fast network connection help this solution. This is where Greg Wickham comes into the picture. He was the KVL Systems Lead at the time (now enjoying his time as the IBEX Systems Lead) and helped us by connecting those first "Remote Workstations" via a 10G connection to Shaheen storage. Now that we have fast connectivity, we also need access to Shaheen storage. Now we welcome on the stage the KSL system administrators, Andrew Winfer and Habeeb Khan. They helped us as soon as we asked access to Shaheen Lustre file-system from the Remote Workstations. Once Prof. Hoteit's group beta-tested the solution, we finally were able to get funding to buy the first servers that would host the Remote Workstations. What had started as an ugly duckling, now had grown into a beautiful (at least to our eyes!) swan. This solution became ever more a team effort. The IT Storage Team (led by Mohammed Naseemuddin) provided access to Noor Homes and DataWaha. The IT Data Center Team (now led by Mani John) moved all the servers to the business data center. This ensured that the Remote Workstations are up 24x7x365 (except when we patch them; we feel you Sabique!). The IT Networks Team (thanks to Juan Carvallo and Luis Barreiro, former and current Lead) created a new super-fast and reliable network that enables high speed connectivity from the business data center (where Remote Workstations now are) to the research data center (where Shaheen resides). Finally, the Linux & Advanced Platforms Team uses GitOps techniques (expanding Dr. Gianluca Castellani's seed work when he was managing Linux Workstations replacing Mike Young) to manage all the Remote Workstations. All changes pushed to these servers are just one merge away. Their GitLab repo is the sole source of truth used to manage these servers.
The bottom line is that when teams work together for the researcher's good, you can touch with your hands the tangible value created. The Remote Workstations is a living example of how Synergy@KAUST makes cool things happen.
Start using Remote Workstations today!
Ibrahim Hoteit (in photo) is a professor in the Earth Sciences and Engineering program at KAUST. He is currently the lead of the Virtual Red Sea Initiative, a joint initiative with Scripps Institution of Oceanography, MIT, and Plymouth Marine Laboratory. He is also serving as the director of the Saudi Aramco Marine Environment Research Center at KAUST. Dr. Hoteit's research interests focus on the modeling of oceanic and atmospheric systems on supercomputers and the analysis of their circulation and variability, with specific interest in data assimilation and uncertainty quantification for large-scale systems.
Yixin Wang (in photo) has joined KAUST from China. Yixin is broadly interested in the interaction between physical oceanography and marine ecology. In her Ph.D., she will use a coupled physical-biological model to simulate the evolution of the Red Sea ecosystem and explore the physical mechanisms that drive it. Yixin is currently investigating Red Sea coral connectivity using a particle-tracking model.
Sabique Langodan (in photo) obtained his Ph.D in Earth Sciences and Engineering from KAUST and working now as a Research Scientist in KAUST’s Red Sea Modeling and Prediction Group. Sabique's research focuses on modeling and understanding ocean processes in the Red Sea and the Arabian Gulf, especially those related to ocean waves.
Peng Zhan (in photo) is a Research Scientist at KAUST. His research focuses on ocean modeling and data assimilation, with an emphasis in exploring mechanisms and predictability of the Red Sea circulation dynamics.
