Windows Ports and Binaries
ECMWF
ECMWF (the European Centre for Medium-range Weather Forecasts) is both a climate research institute and a 24/7 operational service, producing global numerical weather predictions and other data. ECMWF is an independent intergovernmental organisation made up of over 30 member states and co-operators. The Centre has one of the largest supercomputer facilities and meteorological data archives in the world.
Towards the end of 2018, ECMWF put out a public tender for the provision of Windows binaries for their software packages.
As well as collecting petabyte-scale climate data, ECMWF offer open source software libraries to enable users to interact with the data through numerical analysis or visualisation. At the time, these libraries worked well on Linux and MacOS but were unsupported on Windows.
A Modern Approach
Supplier risk is a concern for any organisation and ECMWF is no exception. When we bid for this work we were a freshly-incorporated company with no previous contracts - that alone would be enough to put many organisations off.
However, ECMWF could see the passion of our team and the advantage of collaborating with a dynamic, fast-moving organisation. We offered a fresh perspective on modern tools and best practice. Of course, it didn’t hurt that our team had past experience working with complex build systems and large codebases.
Porting ECMWF core software packages to the Windows platform and integrating them through Conda has been a very welcome new feature for us and our users. This work allows users to use these powerful packages for the first time under Windows to access and visualise a wide range of meteorological data provided by ECMWF. ORT's work did not only enable these new possibilities, but also lay the foundations for automatic continuous integration testing and a setup on GitHub to allow for external code contributions. Old Reliable Tech's expertise on Open Source developments and tools enabled a fast and successful implementation.
Problem
We were tasked with porting ecCodes and Magics to Windows, two core libraries upon which many other libraries depend. As well as porting the libraries, we made them available for users to easily install by distributing them using conda-forge, the community-driven Python package manager.
The codebases in question are a mix of C and C++. Both have Python interfaces and use cmake as their build system.
Solution
We successfully ported ecCodes and Magics to Windows - a feat which has never before been achieved. We used pre-existing tools in order to run ECMWF’s existing tests - which use bash and standard UNIX tools (sed, awk, cat, diff, etc) - on Windows. This left ECMWF with a minimal maintenance burden going forward.
We delivered our work on-budget and ahead of time.
As a result, our project was extended to include a Windows port & package for CodesUI and a package for Metview.
As an extra value-add we set up Continuous Integration (CI) on ECMWF’s GitHub repos so that they could start accepting open source contributions. CI allows the code owner to make sure that a proposed code change has not broken their codebase before having a developer look at it. This has the dual advantage of saving ECMWF developer time as well as encouraging an open source community around ECMWF’s software.
We also found time to add an xarray interface to Magics. Xarray is a popular library which allows scientists and developers to easily work with n-dimensional data arrays. Thanks to our work, Magics users can now directly visualise their xarray datasets without having to first transform the data.
Looking Forward
We’d like to thank our friends and collaborators at ECMWF for supporting a young company. This was a brilliant first project for us - we would not be where we are today without your trust!
This project perfectly aligned with our mission to work on projects with positive social impact. We’re grateful to ECMWF for the opportunity to help and look forward to collaborating with them again.
You can read more about our work for ECMWF in their newsletter and on their blog.