100Gbps of CERN data over IPv6 on the Janet network

Today, 6th June 2022, marks the tenth anniversary of the World IPv6 Launch, an event that saw many ISPs and content providers turn on IPv6 access to their services for the first time. In this article, we reflect on an IPv6 use case that’s now seeing 100Gbps of IPv6 traffic traversing Janet.

The Large Hadron Collider (LHC) at CERN has recently restarted, which is likely to result in larger volumes of particle physics data than ever being collected by the four main experiments prior to being transferred to participant sites for processing and analysis.

In the UK, some 19 universities and research sites are part of the Worldwide Large Hadron Collider Computing Grid (WLCG). These sites, together forming the UK GridPP community, have some of the most demanding requirements for data transfers over the Janet network. One, the Rutherford Appleton Laboratory (RAL), is now connected to Janet at 2x100G, while some others have been or are in the process of being upgraded to 100G.

One of the universities connected to Janet at 100G is Imperial College London which is on the CMS and LHCb experiments at the LHC. Imperial College has one 100G link to its data centre at the shared Slough DC facility, and one 100G connection to its London campus in South Kensington. Imperial is also one of the leading Janet sites for IPv6 deployment on campus, having dual-stack IPv4/IPv6 through most of its infrastructure. This is of particular interest as the WLCG has over the past few years been moving its networking to both support and prefer the use of IPv6, with the aim of running IPv6-only in the future.

The status of the WLCG’s move towards IPv6 is reported on its IPv6 Working Group site, which shows 15 of the 19 UK sites have IPv6, and that overall over 80% of the Tier 2 WLCG storage sites have IPv6 enabled. The main driver is simply address space, to ensure the compute and storage nodes can all communicate directly, and not be impaired by NAT. It’s a significant achievement, and an exemplar for other communities who may choose to follow.

But what does this mean for Imperial’s WLCG traffic? Imperial is connected to the LHC overlay network, LHCONE, which is essentially a virtual network (a VRF) sitting on top of the general R&E networks, but only carrying WLCG data. There is also a separate dedicated optical network, LHCOPN, connecting the Tier 1 sites to CERN, of which RAL is one.

And this leads us into being able to report that, at the end of March 2022, a bulk file transfer to Imperial filled the incoming 100G link at Slough, such that 100% of the traffic coming in over LHCONE was over IPv6. Until recently, that 100% figure hadn’t been reached, but a recent change to the dCache software used to manage the disk servers at Imperial, in which the Java elements were configured to ensure they prefer IPv6, has pushed the inbound transfers to now be predominantly IPv6.

The plot below of Imperial’s monitoring system at Slough shows the two-hour period where the 100G link was filled and where (the green line below) 100% of the LHCONE traffic at that time was IPv6.

The traffic levels seen in the network view correspond to those seen by the WLCG File Transfer Service (FTS) visualisation tools, as shown below. It’s useful to consider both views, to ensure the network performance is correctly viewed as part of the overall true end-to-end transfer picture. The FTS view is also a reminder that while network administrators think in bits per second, researchers think in bytes, so make sure you’re comparing Gbit/s and GB/s correctly!

It was also interesting to see this traffic reflected in BRIAN, the new monitoring platform for the GÉANT pan-European R&E backbone network, as shown below. GÉANT provides connectivity across Europe for Jisc and the other National Research and Education Networks (NRENs), and internationally to other continents and regions. BRIAN has both public views and login-restricted views of different traffic paths, including the Janet traffic to/from GÉANT, and the LHCONE traffic to/from Janet.

Seeing 100Gbit/s of traffic, and 100% of the LHCONE CERN data, coming into Imperial over IPv6 is a great demonstrator of both the performance and capability of IPv6, and of course of Janet.

To find out more about deploying IPv6 at your site, see our Jisc IPv6 page, which includes information on acquiring address space and advice and guidance on how to start a deployment. IPv6 is available to all Jisc members as part of their standard Janet IP connection.

Leave a Reply

Your email address will not be published. Required fields are marked *