We have had a few tickets pointing us at Cloudflare’s Is BGP Safe Yet website (one describing it as “a new website that details internet service providers that don’t tend to their routing“) and telling us that we do not implement BGP safely.
We do not yet drop invalid routes using RPKI data, the website is correct in saying that.
However, Cloudflare’s test is very narrow, and their claims for what RPKI provides are overly broad.
A brief introduction to inter-domain routing on the Internet
Each network that is part of the Internet, that in turn is connected to more than one other network, will have what is called an Autonomous System Number (ASN). Janet’s ASN is 786.
The protocol that routes between ASNs is called the Border Gateway Protocol (BGP). This ties IPv4 and IPv6 prefixes to ASNs. So, for example, all networks in the world that have a full routing table will know that 212.219.0.0/16 is originated by AS786.
Now another network connected to, say, Telia Carrier (one of our transit providers) will see that 212.219.0.0/16 has an autonomous system path of “1299 786”, i.e. as the routing information propagates between networks, each network adds its own ASN to the path.
Who is allowed to announce what?
For a long time, all that data has been stored in Internet Routing Registries (IRRs). In the case of Europe, the Regional Internet Registry for Europe, the RIPE NCC, operates one with a reasonably close authentication that ties together the owner of an ASN and the owner of a prefix. It was (and is) possible to generate prefix lists from that to ensure that your neighbour network is only advertising the correct prefixes to you — I gave a presentation on doing that somewhere around 20 years ago.
However, building static prefix lists doesn’t scale anywhere other than the edge of the network, and not all IRRs had a tight integration between owning a prefix and being able to announce it, so for many years work has been going on to secure route originations, the tie between the prefix and the ASN at the very right hand side of an AS path (the “786” in “1299 786” above).
This is done via the resource public key infrastructure (RPKI), which as the name suggests is based on public key cryptography. At its simplest, the RIPE NCC has a certificate that says which prefixes it holds, and that is used to sign a certificate that says which prefixes we hold (e.g. 212.219.0.0/16 above). We can then use that to sign another certificate (a “route origin attestation” or ROA) that says we permit AS786 to announce 212.219.0.0/16.
The observant will notice that none of this is connected to BGP yet, it’s just a PKI.
To tie it to BGP, the router must be able to check the routing table it sees against the PKI. That validation is done off the router on a server using one of a number of packages, and routes are marked ‘valid’ if there is a correctly signed ROA, ’not found’ if there is no ROA, and ‘invalid’ if there is a matching ROA with the wrong origin ASN. This means that if Telia accidentally decided to announce 212.219.0.0/16 itself from AS1299, that would be invalid.
Increasingly service providers are enabling this check, and dropping ‘invalid’ prefixes, and this is what the Cloudflare test is checking for — they announce a correctly signed prefix and a prefix with an invalid ROA.
Some limitations of RPKI Origin Validation
This doesn’t stop hijacks fully. If someone nefarious at Telia wanted to hijack 212.219.0.0/16, they could set up another router (which could just be a PC running routing software) and configure it to announce the prefix from AS786, which would pass validation.
So it’s not perfect. It’s more likely to protect against accidental hijacks than it is nefarious ones, and it only protects the origin (i.e. it doesn’t say that 1299 is permitted to announce routes from 786).
It is, however, a step in the right direction, and one we are looking at deploying on Janet when all the pieces are in place — which includes the server infrastructure to perform the origin validation.
We have ROAs for most of the Janet aggregate routes, but we need to work with our members to create ROAs for prefixes that pre-date the RIRs (e.g. a lot of the “legacy” /16s which are allocated directly to the Universities rather than to Janet), as we cannot create them without interacting with the RIPE NCC as there will be no resource certification in place.
One of the complications we have to consider is cloud DDoS mitigation, which is something else we are looking at. Suppose a smaller prefix within 212.219.0.0/16 is being attacked (say 212.219.64.0/24). To be able to absorb the attack, the DDoS mitigation provider will need to originate this more specific prefix, either from AS786 or from their own ASN. Now, unless there is a matching more specific ROA, that route will be invalid and dropped, and so the DDoS mitigation provider will not be able to do their work.
However, creating a more specific ROA means that the opportunity for a nefarious hijack is back, as we now have a valid ROA for the /24 which — unless we are constantly advertising (and therefore deaggregating all of our prefixes) — can be hijacked as above by faking the origin ASN, and even with a longer AS path will attract all the traffic by virtue of being a more specific prefix.
This is one of the factors that we’re thinking about in our rollout of RPKI, and you can be sure we will drop invalid routes when the time is right, but we’re not there now, and that’s nothing to worry about.
It’s also worth noting that not all address space is covered by a ROA yet. Data from NLnetLabs shows that in the UK it’s about 28.5% of announced prefixes, in the US where Cloudflare is based it is 7.1%, and to use ARIN’s (the RIR for North America) Trust Anchor Locator (TAL), you need to sign an agreement with them. Perhaps the first push should really be to get more address space covered by ROAs before nagging networks to enable origin validation.
Links to other comments: