Switching from MPLS core to EVPN core

VPN
1

Now that quite a few vendors have BGP EVPN over VXLAN capable access switches, I’m wondering if anyone is doing their core network with these technologies?

We’re currently running MPLS in our own network and routing between VRFs happens on the DC firewalls:

https://pasteboard.co/IKVL8wT.png

Each building has an aggregation switch that also talks MPLS towards the core, and terminates all the VLANs from the access layer. Access layer is L2.

We have lot’s of different buildings and 50+ different segments for different use cases so just configuring L3 on the access layer would be somewhat nightmare to manage with all the ACLs etc. Also we would lose visibility over the traffic between the segments.

Wondering also how do you do traffic engineering, for example have workstations to use core link 1 in the picture and cameras to use link 2 as the primary path.

Not really here trying to solve any major issues but rather wondering how EVPN would work and how it would differ from running MPLS. Any thoughts?

Thanks!

2

Just to clarify a bit, MPLS is an underlay which can run a bunch of different overlays (including BGP EVPN!). VxLAN is an alternative underlay, which is typically used to run BGP EVPN as its overlay.

So you could run MPLS (especially Segment Routing MPLS) on every single node in the network, and still run BGP EVPN wherever you desired to, with L3VPN or L2VPN wherever else you desired.

MPLS is well suited to scale over arbitrary topologies (hub-spoke, daisy-chain, ring, big amorphous blobs, spine-leaf, etc) and has various design options for scaling it larger than your individual IGP (Unified/Seamless, SRTE with PCE, etc)

VxLAN is typically designed and scaled for Datacenters or LANs, and typically in spine-leaf topologies.

Routers marketed for WAN/edge typically don’t speak VxLAN, and switches marketed for Datacenter or LAN typically don’t speak MPLS (especially LDP, some newer ones are looking at Segment Routing MPLS)

I used the word “typically” like 17 times there, because individual vendors and products definitely will vary, and it’s not like you can’t build a hub-spoke site with VxLAN on it. I just wanted to be clear you don’t have to pick “MPLS vs EVPN”, it’s MPLS vs VxLAN, and EVPN over either.

As to how EVPN differs from VPLS or pseudowires or VRF-based L3VPN…

It theoretically consolidates all of that into a single address-family, using route-type 2 for individual host entries, with their MAC address and potentially MAC-to-IP bindings. Route-type 5 carries L3 subnet routes like L3VPN used to. So if you used to run VPLS and also L3VPN, now you only need to manage BGP EVPN. You’ll manage RDs and RTs and BGP peers just as you did with L3VPN, but the output will look different as each route also includes a type and potentially MAC addresses. Lots more BGP routes in the table, as you’re doing host-routes now. This can be managed by filtering type-2s at various boundaries.

How you handle traffic engineering depends on which underlay you pick. VxLAN isn’t really meant to do TE as you describe, as it’s designed to be used primarily in a local datacenter (not focused on latency or subrate provider circuits) with ECMP links and ideally no congestion. MPLS has conventional MPLS TE and if you go Segment Routing there’s SRTE. Both rely on your underlying IGP, which could be manipulated to force preferred paths or ECMP, but likely not to differentiate “workstation vs camera” traffic as in your example.

3

Do you have a requirement for extending your L2 domain between buildings?

4

First of all, the thread subject do not make sense. EVPN is a control plane protocol based on BGP. MPLS is a data plane protocol. VXLAN is a new data plane protocol that has had massive interest recently as it has been implemented in relatively cheap chipsets like several from Broadcom.

EVPN is a control plane protocol that can use several data plane protocols like VXLAN and MPLS. One do often talk about EVPN/VXLAN or EVPN/MPLS to distinguish between these. So the migration you talk about is probably from L3VPN and VPLS or similar on top of MPLS to EVPN/VXLAN doing both L3VPN and L2VPN.

One point is that you can do EVPN on MPLS if you like to keep MPLS.

You seem to want to be able to change hardware platform to cheaper hardware. Few vendors focus on MPLS and EVPN/ VXLAN is implemented by more vendors, and this might make this cheaper.

EVPN/VXLAN is made to focus on Data Center. VXLAN is udp in ip in Ethernet. You mentioned Traffic Engineering. If you cannot put enough link bandwidth on the network and need Traffic Engineering I doubt EVPN/VXLAN is for you as there are no traffic engineering possibilities with VXLAN. The only thing you can do is tuning the IGP, the packets will follow the underlay best path or multi-path.

Another important thing to know is that there are significant more overhead with VXLAN than with MPLS. This can be important if link bandwidth is limited.

5

I think Arista advertises something like this?

https://www.arista.com/en/solutions/cognitive-campus

Not really sure though how you handle firewalling with that

6

Thanks for the reply! We are currently migrating our DC network from Cisco vPC architecture to spine and leaf with EVPN. We should have something ready in january I hope… We also have routers in place than can do both EVPN over MPLS and EVPN over VXLAN, so we could stitch in two cities for those older applications that like to have L2 connectivity over our current MPLS core network.

As we’re moving to Aruba 6300 in the access layer and they support EVPN too, we could probably migrate from MPLS core towards something else. Running just basic OSPF/BGP capable switches in the core and then do the EVPN segments from access layers towards the DC firewalls.

Those Aruba switches only support EVPN over VXLANs, so that’s why I was asking about this question. We’re not currently doing traffic engineering, but just having one active link currently and if that fails we’d have the traffic on the secondary link.

7

Some companies bringing in all the HVAC/camera/etc stuff usually insist on having L2… but usually they just work well with L3. Idea here is to run L3 underlay in the LAN, and then use EVPN to control VXLAN tunnels towards the datacenter where we could do policing and routing between different segments. Now we use VLAN towards the aggregation (PE) switches but with EVPN we wouldn’t have to have MPLS capable switches as aggregation switches and all the logic would be at the access layer.

We don’t have that big L2 domains, now we are building a new building and it’s going to be around 350 access switches in stacks of 4-5. But everything directly connected to aggregation so only two levels.

It might also make the core network simpler to manage, though not really sure about this yet. Currently we also do small PE routers at each branch that has multiple segments, so far we have something like 90 routers in our core OSPF area 0. Switching to SD-something would probably make things easier there.

8

EVPN/VXLAN is coming hard to access layer too. Cisco SD-access is VXLAN with their own stuff on top of that, Arista does something similar, I believe Extreme does and now Aruba does too. Everyone is doing EVPN controlled VXLAN to the access switch so it’s not just DC stuff anymore. Oh an Juniper has done this years ago.

I was cutting some corners here, true. When I compare our current MPLS network to EVPN stuff, I just think that the EVPN is over VXLAN tunnels and not MPLS.

9

We’re not currently doing traffic engineering, but just having one active link currently and if that fails we’d have the traffic on the secondary link.

So EVPN would be nice because you could now use both links active-active, without worrying about L2 loops.

Sounds like you’re well on your way, and that EVPN will be a good move. Changing your core from MPLS core to pure-IP core is a pretty big step though. That pretty much rules out doing most overlay options like EVPN or L3VPN/L2VPN/CEM across it. There’s IP tunneling options in a pinch, but if you actually need overlay services, removing MPLS feels like a step backwards…

10

Are you using MPLS VPNs today? It seems like a better fit for your use case than EVPN. EVPN has a similar MPLS VPN feature set with type-5 routes but it’s main use case is scalable extending broadcast domains.

11

Extreme push their SPB protocol in the LAN, a mac-in-mac fabric that is designed to replace spanning tree.

12

Yeah as the new Aruba switches are more expensive I’m basically looking for reasons to implement those now :slight_smile: And future proof our network, as I know they’re not going anywhere in the next 7 years or so…

We’re starting with dynamic segmentation (tunneling all the traffic to a controller) but as we have the option to do EVPN we could change to it without replacing hardware. And as our MPLS network isn’t that complicated, just moving bits in a VRF to DC, it wouldn’t be that difficult migration. We’ll probably start it by doing a “new-core” VRF there and putting everything in that, and then phase out the old MPLS core. If it seems that it’s the way to go :slight_smile:

13

Yes, our whole network is built on MPLS L3 VPNs. Every segment we have (something like 150 now for all of our customers and we’re still in the beginning…) is it’s own L3VPN.

Downside is that management might be bit more difficult, and we need MPLS capable stuff in our core. With EVPN we could for example use Aruba 8320 series switches instead. As we’re moving towards Aruba in the access it would be nice to have Aruba in the core too instead of the other vendor we have now.

We have some L2VPNs too between different cities and buildings and they are somewhat pain to manage. EVPN would be easier there.