Getting distracted by MPLS L3 VPN’s

It started of with my SD-WAN lab. After hours of studying and eventually adding a few vEdges to finalize the Control Plane bringup, I was ready for templating, creating different BFD topologies and all the fun SD-WAN provides.

But I just had one single WAN link from my DC all the way to the branch office. The whole WAN was just two routers and one link. Obviously this would work fine configuring the vEdge device and testing a few features, but it just wasn’t the real deal.

Let’s build a real MPLS WAN transport. Then we can also play around with increased jitter and packet loss within the WAN transport and see how my defined SLA responds to that. I might learn some new tricks in MPLS and MP-BGP.

:: Some basics

Before we start configuring some routers like a wild man, lets first define what we need and how to built it. From my experience I know that we can split MPLS into two types:

  • MPLS L2 VPN
  • MPLS L3 VPN

To make sure I can expand the topology and play around with “customers” and different transport paths for my SD-WAN lab I’ve chosen MPLS L3VPN. Not the easiest of the two.

The transport medium will contain the following separated functions:

:: CE Nodes 

Customer Edge nodes, CE nodes provide a means to interconnect a site to the MPLS network. It doesn’t have to be VRF aware. It can be static, but I will use OSPF as the routing protocol towards the PE routers.

:: PE Nodes 

Provider Edge nodes, this is where the encapsulation into MPLS happens and the VPN id are transported as an extra label over the transport. They also need to be VRF aware in order to extend the VPN across the MPLS network. Packets get a label (and VPN label) assigned to be forwarded over the P-network. We also use BGP extension to import and export VPN’s according to a defined Route Distinguisher and thus allowing sites or customer to ” talk” to each other or not.

:: P Nodes

Provider Core nodes make up the MPLS transport network and make forwarding decisions based on labels. You should run a routing protocol to create reachability in the P-Network. For large scaled network IS-IS would be the preferred choice. In my case I will use OSPF.

:: Topology

Below is the full SD-WAN lab. All nodes in blue will be the scope of our topology:

Note: I’ve added additional CE routers to this topology to play with advertised prefixes and VPN’s. When using this for SD-WAN the vEdges will act as the CE routers.

:: Approach

On both the CE routers I’ve installed Loopback9000 to advertise a test network prefix into the VPN and be transported across the MPLS P-network. After configuring the topology these two prefixes should be able to talk to each other.

I’ve created a small IP plan before configuring to make sure we don’t end up with silly IP connectivity issues:

RouterInterfaceIP RangeInterfaceRouterIP Validation
MPLS-PE-DCe0/0172.18.10.0/24e0/0MPLS-CE-DCOK
MPLS-PE-DCe0/1172.16.2.24/30e0/1MPLS-P3OK
MPLS-PE-SITEe0/0172.18.20.0/24e0/2MPLS-CE-SITEOK
MPLS-PE-SITEe0/1172.16.2.28/30e0/1MPLS-P4OK
MPLS-P3e0/0172.16.2.0/30e0/0MPLS-P4OK
MPLS-P3e0/3172.16.2.8/30e0/3MPLS-P1OK
MPLS-P4e0/2172.16.2.12/30e0/2MPLS-P1OK
MPLS-P4e0/3172.16.2.16/30e0/3MPLS-P2OK
MPLS-P2e0/0172.16.2.20/30e0/0MPLS-P1OK
MPLS-P2e0/2172.16.2.4/30e0/2MPLS-P3OK

The P-network will carry IP range 172.16.2.0/27 (8 p2p links). The PE to CE transports will be IP ranges 172.18.10.0/24 and 172.18.20.0/24

When doing a traceroute you can clearly identify CE-PE and P communications.

I’m not going into details about how to configure the p2p links. This should be a “no-brainer” when playing around with MPLS. Below a small list what to do in to create the base configuration:

  • Base configuration with ‘IP cef’ and ‘no ip domain-lookup’  🙂
  • Configure p2p links P-nodes
  • Configure Loopback interface P-nodes
  • Configure p2p links P-PE nodes
  • Configure Loopback interfaces PE-nodes
  • Configure p2p link CE-PE nodes
  • Configure Loopback interfaces CE-nodes
  • Check p2p IP connectivity
NodeFunction
MPLS-CE-DCOSPF handover to PE
MPLS-PE-DCOSPF/MPLS and MP-BGP
MPLS-CE-SITEOSPF handover to PE
MPLS-PE-SITEOSPF/MPLS and MP-BGP
MPLS-P1MPLS (LDP)
MPLS-P2MPLS (LDP)
MPLS-P3MPLS (LDP)
MPLS-P4MPLS (LDP)

:: Configure P-Network for MPLS and reachability with OSPF

Configurations on all P-nodes:

!
! >> 'MPLS IP' is turned on by default, but for sake of clarity included
mpls ip
mpls label protocol ldp
!
Interface range ethx/x
 mpls label protocol ldp
 mpls ip
!
! >> Force to use loopback in MPLS
mpls ldp router-id Loopback{#} force
!
!
router ospf 1
! >> Always advertise loopback interfaces for reachability
 network {loopback-address} 0.0.0.0 area 0
!
! >> Advertise P-network subnets
 network 172.16.2.0 0.0.0.31 area 0
!

Result:

Soon after configuring you will notice some notifications pop up:

<omitted>
*Jun 20 09:59:59.443: %SYS-5-CONFIG_I: Configured from console by console
*Jun 20 10:01:00.688: %OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on Ethernet0/0 from LOADING to FULL, Loading Done
*Jun 20 10:01:05.739: %LDP-5-NBRCHG: LDP Neighbor 4.4.4.4:0 (1) is UP
*Jun 20 10:02:54.921: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on Ethernet0/2 from LOADING to FULL, Loading Done
*Jun 20 10:03:00.442: %LDP-5-NBRCHG: LDP Neighbor 2.2.2.2:0 (2) is UP
*Jun 20 10:03:14.586: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.1 on Ethernet0/3 from LOADING to FULL, Loading Done
*Jun 20 10:03:20.111: %LDP-5-NBRCHG: LDP Neighbor 1.1.1.1:0 (3) is UP

Output:

======================================================================
MPLS-P3#show mpls ldp neighbor 
    Peer LDP Ident: 4.4.4.4:0; Local LDP Ident 3.3.3.3:0
TCP connection: 4.4.4.4.33304 - 3.3.3.3.646
State: Oper; Msgs sent/rcvd: 18/18; Downstream
Up time: 00:02:56
LDP discovery sources:
  Ethernet0/0, Src IP addr: 172.16.2.2
        Addresses bound to peer LDP Ident:
          172.16.2.2      172.16.2.30     172.16.2.14     172.16.2.17     
          4.4.4.4         
    Peer LDP Ident: 2.2.2.2:0; Local LDP Ident 3.3.3.3:0
TCP connection: 2.2.2.2.646 - 3.3.3.3.49129
State: Oper; Msgs sent/rcvd: 16/16; Downstream
Up time: 00:01:02
LDP discovery sources:
  Ethernet0/2, Src IP addr: 172.16.2.6
        Addresses bound to peer LDP Ident:
          172.16.2.21     172.16.2.6      172.16.2.18     2.2.2.2         
    Peer LDP Ident: 1.1.1.1:0; Local LDP Ident 3.3.3.3:0
TCP connection: 1.1.1.1.646 - 3.3.3.3.56828
State: Oper; Msgs sent/rcvd: 15/15; Downstream
Up time: 00:00:42
LDP discovery sources:
          Ethernet0/3, Src IP addr: 172.16.2.10
        Addresses bound to peer LDP Ident:
          172.16.2.22     172.16.2.13     172.16.2.10     1.1.1.1  
======================================================================
MPLS-P3#sh ip ospf neighbor 

Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1           1   FULL/BDR        00:00:33    172.16.2.10     Ethernet0/3
2.2.2.2           1   FULL/BDR        00:00:32    172.16.2.6      Ethernet0/2
4.4.4.4           1   FULL/DR         00:00:31    172.16.2.2      Ethernet0/0
======================================================================
MPLS-P3#sh ip route ospf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
O        1.1.1.1 [110/11] via 172.16.2.10, 00:01:39, Ethernet0/3
      2.0.0.0/32 is subnetted, 1 subnets
O        2.2.2.2 [110/11] via 172.16.2.6, 00:01:58, Ethernet0/2
      4.0.0.0/32 is subnetted, 1 subnets
O        4.4.4.4 [110/11] via 172.16.2.2, 00:03:53, Ethernet0/0
      172.16.0.0/16 is variably subnetted, 12 subnets, 2 masks
O        172.16.2.12/30 [110/20] via 172.16.2.10, 00:01:39, Ethernet0/3
                        [110/20] via 172.16.2.2, 00:03:53, Ethernet0/0
O        172.16.2.16/30 [110/20] via 172.16.2.6, 00:01:58, Ethernet0/2
                        [110/20] via 172.16.2.2, 00:03:53, Ethernet0/0
O        172.16.2.20/30 [110/20] via 172.16.2.10, 00:00:54, Ethernet0/3
                        [110/20] via 172.16.2.6, 00:01:58, Ethernet0/2
O        172.16.2.28/30 [110/20] via 172.16.2.2, 00:03:53, Ethernet0/0
======================================================================

:: Configure PE and CE nodes

PE nodes are Provider edge nodes. The configuration is slightly more complex as this type of device needs to insert, convert and transport non-vrf aware traffic in to a VRF-aware transport medium called the P-network.

With the PE nodes we also create an overlay with MP-BGP to communicate over the P-network. From PE to PE it looks like a direct logical connection. In order to identify different VPN’s we use Route distinguishers to identify them. With route-target we choose to import and export these to allow communications. Without route-targets no edge communications is shared until these are configured. Play around with them and see the results. For now I’ve included them to keep track of configurations and checks.

Let’s start inserting the required commands for nodes MPLS-PE-DC and MPLS-PE-SITE:

!
! 
! >> enable MPLS 
interface eth0/1
mpls ip
mpls label protocol ldp
!
!
!>> Create customer/transport VPN
vrf definition VPN0
! >> rd = 1:{SD-WAN site-id}
 rd 1:9000
 !
 address-family ipv4
! >> We choose to have both ways of communications of our site-id
  route-target both 1:9000
  route-target import 1:1
 exit-address-family
!
! >> OSPF for reachability in MPLS
router ospf 1
 router-id 5.5.5.5
 network 5.5.5.5 0.0.0.0 area 0
 network 172.16.2.0 0.0.0.31 area 0
!
!
! >> Implement MP-BGP neighbors across MPLS transport
router bgp 65535
 bgp log-neighbor-changes
 neighbor 6.6.6.6 remote-as 65535
 neighbor 6.6.6.6 update-source Loopback5
 !
 address-family vpnv4
  neighbor 6.6.6.6 activate
!
! >> Send BGP extentions in order to keep track of VPN targets in VPN-vrf tables
  neighbor 6.6.6.6 send-community extended
 exit-address-family
 !

Output:

After configuring both PE routers the following output should be seen that ensure that LDP, OSPF and BGP are up and running:

*Jun 20 10:51:55.203: %SYS-5-CONFIG_I: Configured from console by console
*Jun 20 10:51:58.377: %OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on Ethernet0/1 from LOADING to FULL, Loading Done
*Jun 20 10:52:04.998: %LDP-5-NBRCHG: LDP Neighbor 4.4.4.4:0 (1) is UP
*Jun 20 10:52:14.650: %BGP-5-ADJCHANGE: neighbor 5.5.5.5 Up 

Notice the order of output is stating that OSPF reachability should be UP before MPLS can start discovering neighbors. When this is done BGP can peer across the MPLS transport.

Checks:

MPLS-PE-DC#show ip ospf neigh

Neighbor ID     Pri   State           Dead Time   Address         Interface
3.3.3.3           1   FULL/DR         00:00:33    172.16.2.26     Ethernet0/1
======================================================================
MPLS-PE-DC#show mpls ldp neigh
    Peer LDP Ident: 3.3.3.3:0; Local LDP Ident 5.5.5.5:0
TCP connection: 3.3.3.3.646 - 5.5.5.5.49338
State: Oper; Msgs sent/rcvd: 29/28; Downstream
Up time: 00:10:11
LDP discovery sources:
  Ethernet0/1, Src IP addr: 172.16.2.26
        Addresses bound to peer LDP Ident:
          172.16.2.1      172.16.2.26     172.16.2.5      172.16.2.9      
          3.3.3.3  

=====================================================================     
MPLS-PE-DC# sh ip bgp summary
BGP router identifier 5.5.5.5, local AS number 65535
BGP table version is 1, main routing table version 1

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
6.6.6.6         4        65535      10      10        1    0    0 00:05:28        0

======================================================================
MPLS-PE-DC#show ip route 
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
O        1.1.1.1 [110/21] via 172.16.2.26, 00:27:11, Ethernet0/1
      2.0.0.0/32 is subnetted, 1 subnets
O        2.2.2.2 [110/21] via 172.16.2.26, 00:27:11, Ethernet0/1
      3.0.0.0/32 is subnetted, 1 subnets
O        3.3.3.3 [110/11] via 172.16.2.26, 00:27:11, Ethernet0/1
      4.0.0.0/32 is subnetted, 1 subnets
O        4.4.4.4 [110/21] via 172.16.2.26, 00:27:11, Ethernet0/1
      5.0.0.0/32 is subnetted, 1 subnets
C        5.5.5.5 is directly connected, Loopback5
      6.0.0.0/32 is subnetted, 1 subnets
O        6.6.6.6 [110/31] via 172.16.2.26, 00:05:49, Ethernet0/1
      172.16.0.0/16 is variably subnetted, 9 subnets, 2 masks
O        172.16.2.0/30 [110/20] via 172.16.2.26, 00:27:11, Ethernet0/1
O        172.16.2.4/30 [110/20] via 172.16.2.26, 00:27:11, Ethernet0/1
O        172.16.2.8/30 [110/20] via 172.16.2.26, 00:27:11, Ethernet0/1
O        172.16.2.12/30 [110/30] via 172.16.2.26, 00:27:11, Ethernet0/1
O        172.16.2.16/30 [110/30] via 172.16.2.26, 00:27:11, Ethernet0/1
O        172.16.2.20/30 [110/30] via 172.16.2.26, 00:27:11, Ethernet0/1
C        172.16.2.24/30 is directly connected, Ethernet0/1
L        172.16.2.25/32 is directly connected, Ethernet0/1
O        172.16.2.28/30 [110/30] via 172.16.2.26, 00:27:11, Ethernet0/1
      172.18.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.18.10.0/24 is directly connected, Ethernet0/0
L        172.18.10.1/32 is directly connected, Ethernet0/0
======================================================================
MPLS-PE-DC#show ip route vrf VPN0

Routing Table: VPN0
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

Notice that we do not have any BGP route or vrf VPN0 entries? That because we do not advertise or redistribute network/prefixes.

The CE routers need to be configured first and PE needs to redistribute OSPF into BGP and vice versa

:: Configure CE nodes (and applying some magic on PE nodes)

The only thing we need to do on CE nodes is enable OSPF and advertise local networks. I use OSPF process 100 to reference to the PE node configuration. OSPF process number are locally significant, so you could should any number you like.

ON CE nodes:

!
router ospf 100
Router-id {local lo interface}
 passive-interface Loopback{#}
!
! >> to keep it simple I advertise any network
 network 0.0.0.0 255.255.255.255 area 0
!
Interface Lo9000
! >> Convert Loopback OSPF type from host to subnet for some real life experience ;-)
ip ospf network point-to-point
!

And now for the magic to happen!

Redistribute OSPF and BGP routes into VPN0 on PE nodes. Before we start please note we use two OSPF processes on one router. One for internal communications (process 100) and the other one for MPLS reachability (process 1). For OSPF we need unique router-id’s and preferably ones that do not go down for physical reasons. So we implement an additional looback interface.

ON PE nodes:

!
!
! >> Create new Lo interface for new OSPF process 100
Int lo{#}
Ip address {ip-address} 255.255.255.255
!
! >> map CE router into vrf VPN0
Interface eth0/0
 vrf forwarding VPN0
ip address 172.18.10.1 255.255.255.0
!
!
router bgp 65535
!
 address-family ipv4 vrf VPN0
  redistribute ospf 100
 exit-address-family
!
!
router ospf 100 vrf VPN0
 router-id {loopback-interface-process-100}
 network 10.10.10.10 0.0.0.0 area 0
 network 172.18.10.0 0.0.0.255 area 0
 redistribute bgp 65535 subnets
!
End
!
!
>> OSPF can start complaining for a router-id before you even manage to config it. Adding one and clearing the process will re-run the RID selection and start the process properly.
!
Clear ip ospf 100 process
Yes

!

Verify on CE nodes:

MPLS-CE-DC#show ip route ospf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      7.0.0.0/32 is subnetted, 1 subnets
O IA     7.7.7.7 [110/21] via 172.18.10.1, 00:02:47, Ethernet0/0
      172.18.0.0/16 is variably subnetted, 3 subnets, 2 masks
O IA     172.18.20.0/24 [110/11] via 172.18.10.1, 00:02:47, Ethernet0/0
O IA  192.168.10.0/24 [110/21] via 172.18.10.1, 00:02:47, Ethernet0/0
======================================================================
MPLS-CE-SITE#sh ip route ospf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      8.0.0.0/32 is subnetted, 1 subnets
O IA     8.8.8.8 [110/21] via 172.18.20.1, 00:02:04, Ethernet0/2
      172.18.0.0/16 is variably subnetted, 3 subnets, 2 masks
O IA     172.18.10.0/24 [110/11] via 172.18.20.1, 00:10:50, Ethernet0/2
O IA  192.168.1.0/24 [110/21] via 172.18.20.1, 00:00:00, Ethernet0/2
======================================================================
MPLS-CE-DC#ping 192.168.10.1 
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.10.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/2/3 ms

MPLS-CE-SITE#ping 192.168.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms
======================================================================
MPLS-CE-DC#traceroute 192.168.10.1
Type escape sequence to abort.
Tracing the route to 192.168.10.1
VRF info: (vrf in name/id, vrf out name/id)
  1 172.18.10.1 1 msec 1 msec 1 msec
  2 172.16.2.26 [MPLS: Labels 23/30 Exp 0] 3 msec 2 msec 2 msec
  3 172.16.2.2 [MPLS: Labels 23/30 Exp 0] 2 msec 2 msec 2 msec
  4 172.18.20.1 [MPLS: Label 30 Exp 0] 1 msec 2 msec 1 msec
  5 172.18.20.2 2 msec *  3 msec

MPLS-CE-SITE#traceroute 192.168.1.1
Type escape sequence to abort.
Tracing the route to 192.168.1.1
VRF info: (vrf in name/id, vrf out name/id)
  1 172.18.20.1 1 msec 1 msec 1 msec
  2 172.16.2.30 [MPLS: Labels 16/30 Exp 0] 3 msec 2 msec 2 msec
  3 172.16.2.1 [MPLS: Labels 19/30 Exp 0] 2 msec 2 msec 1 msec
  4 172.18.10.1 [MPLS: Label 30 Exp 0] 1 msec 1 msec 2 msec
  5 172.18.10.2 2 msec *  2 msec

:: Resources

:: MPLS L3 VPN Config and Labs ::

Hope you find it useful. If you’re looking for answers to questions, it should be 42. Otherwise leave a comment! 😉

Leave a Reply