The capacity of the Janet network has always seen massive growth – in my time at Jisc it’s gone from the 10Gbits/s SuperJanet4 network in 2006, to the 100Gbits/s SuperJanet5 network in 2011, to the 600Gbits/s Janet6 network that operates today. We also made a bit of noise last year when we upgraded the core of the network to 400Gbit/s, which was a complex and time consuming piece of work, but ultimately one that put us at the forefront of R&E networking globally.
So, in summary, we’re almost constantly upgrading Janet. But why do we do it? In simple terms, this is why:
Traffic on the network just keeps rising, thanks to all of our lovely members and customers doing more and more exciting things that require more and more bandwidth.
It used to be the case (no more than 5 years ago) that even our biggest users were connected to Janet at 10Gbits/s and that was plenty. The past few years however we’ve seen a step change in network requirements, and the number of 100Gbits/s connected customers is on the rise! This post looks at those big users, when they made the leap to 100Gbit/s, and what they’re doing with it.
First up was the Science and Technology Facilities Council (STFC) Rutherford Appleton Laboratory (RAL). STFC is a world-leading multi-disciplinary science organisation. Its research seeks to understand the Universe from the largest astronomical scales to the tiniest constituents of matter, and creates impact on a very tangible, human scale. RAL’s 40Gbit/s of Janet connectivity was bursting at the seams by Q3 2018, and adding yet more 10Gbit/s channels was no longer the most efficient way to increase capacity, so 100Gbit/s upgrades were implemented. RAL is one of the five largest computing centres that make up the WLCG collaboration; a group of ~200 universities and research institutes around the world providing computing to The Large Hadron Collider (LHC) experiments. RAL archives around 12% of the LHC data produced at CERN. Whilst RAL has dedicated connectivity to CERN for the purposes of receiving this data (which is expected to be upgraded to 100Gbit/s by 2021 to meet the LHC Run 3 requirements), it then uses it’s Janet IP connectivity to share that data with the rest of the UK, hence needing high bandwidth Janet connections.
Next was Imperial College London who made the leap and upgrade Janet connectivity to 100Gbit/s in Q1 2019, and also implemented 100Gbit/s connectivity to the Jisc Shared Data Centre in Slough at the same time. Imperial’s previous 20Gbits/s of Janet connectivity was filling up, and behind the scenes there was a lot of throttling back of the particle physics researchers going on to avoid flooding the connections completely, so on upgrading Janet connectivity to 100Gbit/s all limits were removed, and the traffic graph below shows what happened.
(image taken from a presentation given by Imperial College London at Networkshop47)
University of Edinburgh, specifically its Advanced Computing Facility on the outskirts of Edinburgh – the high-performance computing data centre of EPCC housing a range of supercomputers. As of Q1 2019, connectivity via the University of Edinburgh 10Gbits/s Janet IP connections was no longer suitable given the increase seen in traffic to/from the ACF. Dedicated 100Gbit/s connections were successfully deployed from the ACF directly onto the Janet backbone in July 2019, relieving the pressure on the University and the Janet regional network in Edinburgh.
Finally the European Bioinformatics Institute (EBI) on the Hinxton Campus south of Cambridge – is part of the European Molecular Biology Laboratory (EMBL) Europe’s flagship laboratory for the life sciences. No prizes for guessing that this customer generates, processes and transmits an enormous amount of data. Work is underway to upgrade raw bandwidth to the site from N x 10Gbit/s to 100Gbit/s (whilst also retaining the N x 10Gbits/s as well). On top of that 100Gbit/s connectivity will be provided from the Hinxton Campus to EMBL-EBIs new data centre to further support its activities.
So, what next?
Well we’re in discussion with a number of other institutions about upgrading from N x 10Gbits/s to 100Gbits/s, all of which we expect to come to fruition over the next 12 months – the traffic growth speaks for itself and there’s never really any option other than keep upgrading.
We’ll also continue to upgrade the Janet backbone to cope with the steady flow of upgrades at the edge, in units of 100Gbits/s and 400Gbits/s where appropriate.
Finally, we continue to crunch the numbers, run the reports and predict the future growth, so that we’re always ahead of the game in terms of knowing how and where the traffic levels are increasing. We also work closely with our optical and routing hardware vendors to understand the next generation of products they’re working on, as well as monitoring wider industry activities.