Introduction
EVPN Multihoming (MH) is a standards based replacement for MLAG to achieve standards based all-active server redundancy in datacenters and enterprise networks. Multihoming eliminates the need to have MLAG and unlike MLAG, does not require the group of devices used to create a multihoming group to have inter-connecting peer links.
EVPN multihoming is based on the concept of forming Ethernet Segments (ES). In the context of BGP EVPN, an Ethernet Segment is a set of links connecting a single end host or server with the EVPN VXLAN fabric. In the Figure below, Server1 is multihomed to the VXLAN fabric through links connecting it to VTEP1 and VTEP2, forming an Ethernet Segment. Similarly, Server2, Server3 and Server4 are connected to the fabric through links forming segment 2, segment 3 and segment 4 respectively.
Figure 1. EVPN Multihoming
Key Benefits
- Interconnecting switch links between TOR switches in a redundancy group are not required.
- There can be more than two TOR switches in the redundancy group. A maximum of 8 devices can form one redundancy group as opposed to a maximum of two devices in an MLAG group.
- Ease of management as it provides a single BGP-EVPN control plane.
- Allows multi-vendor interoperability because of standards based implementation.
Brief Mechanism
As shown in Figure 1, if VTEP1 loses direct connection to Server1, it will signal to VTEP2, VTEP3 and VTEP4 that Server1 is no longer accessible and to change their next-hop group for Server1 if there are alternative routes to this ES. Since VTEP2 also has a route to this ES, VTEP3 and VTEP4 will change their next hop groups for Server1 MAC address to go through VTEP2 instead of VTEP1.
EVPN multihoming uses EVPN Type-1, Type-2 and Type-4 routes to discover these Ethernet Segments and deliver traffic to and from these segments. Each Ethernet Segment is identified by a unique ID called Ethernet Segment ID (ESI). ESI is unique across the entire EVPN VXLAN domain on all VTEPs. To configure ESI, we first need to configure the Ethernet Segment System MAC and a local ES-ID. The local ES system MAC and local ES ID are used to automatically generate the ESI. Different ES System MACs can be configured for different Ethernet Segments but the ES System MAC configured on interfaces connecting to the same server or host must be same. In the sections below we will briefly explore EVPN Type-1Auto Discovery and EVPN Type-4 Ethernet Segment Routes.
EVPN Type-1 Ethernet Auto Discovery (A-D) Route
EVPN Type-1 routes are used for fast convergence and aliasing for multihomed sites or end hosts. Fast convergence is important in networks with large number of hosts. When a link failure occurs, devices can signal to upstream PE devices or other VTEPs in the network to change their next-hop groups for the MAC addresses associated with that particular Ethernet Segment. Aliasing is used for load balancing on multiple outgoing links.
Route Distinguisher (RD) (8 octets) |
Ethernet Segment Identifier (10 octets) |
Ethernet Tag ID (4 octets) |
MPLS Label (3 octets) |
Table 1. EVPN Type-1 Route NLRI
Type-1 routes are only originated for multihomed devices, i.e. these routes are sent only when the Ethernet Segment ID is set to a non-zero value.
EVPN Type-4 Ethernet Segment Route
Type-4 routes are used for electing the Designated Forwarder (DF) in EVPN multihoming scenarios. The DF is responsible for sending broadcast, unknown unicast and multicast (BUM) traffic to multihomed hosts or servers. If a server is multihomed to two upstream devices in the EVPN fabric, only one would be elected as the DF for sending BUM traffic to the multihomed server on this ES.
The EVPN NLRI for Type-4 Ethernet Segment Route consists of the following four fields. For multihomed hosts, the Ethernet Segment Identifier (ESI) must be set to a non-zero value. A value of zero for the ESI indicates a single homed device.
RD (8 Octets) |
Ethernet Segment Identifier (10 Octets) |
IP Address Length (1 octet) |
Originating Router's IP Address (4 or 16 octets) |
Table 2. EVPN Type-4 NLRI
EVPN multihoming works by utilizing EVPN Type-1, Type-2 and Type-4 routes. Three types of EVPN routes are used to fulfill three distinct EVPN multihoming requirements.
The PICOS EVPN multihoming can be summarized in the following three steps
Step1:
EVPN Type-2 routes are used for remote learning of end hosts. As shown in Figure 1 for example, Server1 is multihomed to VTEP1 and VTEP2. Both VTEP1 and VTEP2 will first locally learn Server1’s MAC/IP address and then advertise it to VTEP3 and VTEP4 using EVPN Type-2 routes (remote learning).
Step2:
With EVPN multihoming enabled on switch ports connecting Server1 to VTEP1 and VTEP2, both devices will configure the same Ethernet Subnet Identifier (ESI) on the two links and advertise this information to all the other VTEPs using Type-4 routes. This will indicate to remote VTEPs that Server1 is multihomed to the EVPN fabric and is accessible through both VTEP1 and VTEP2.
Step3:
For fast convergence and aliasing, VTEP1 and VTEP2 will advertise Type-1 routes. This type of advertisement is sent in the event of a link failure. If the link connecting VTEP1 and Server1 goes down, VTEP1 will advertise Type-1 route indicating to all the other VTEPs that it has lost connection with Server1 (Ethernet Segment). If VTEP1 was elected the designated forward (DF) for this segment, VTEP2 will then assume the role of DF for Server1 whereas VTEP3 and VTEP4 will adjust their next-hop groups for Server1 accordingly.
Note
It is strongly suggested to remove any MLAG related configuration from the overlay devices (VTEPs) if you plan on deploying EVPN MH in your network. The two technologies are designed to address almost similar requirements of redundancy but MLAG is a layer2 technique whereas EVPN MH is deployed in the overlay and is a layer3 technique. The two technologies hence do not mix well and users are urged to stick to EVPN MH for its standards based implementation and interoperability benefits as opposed to MLAG especially in the overlay network design.
Designated Forwarder Election
For BGP EVPN, a Designated Forwarder (DF) is responsible for handling Broadcast, Unknown Unicast and Multicast (BUM) traffic for Ethernet Segments. As shown in Figure 1, Server1 constitutes an Ethernet Segment as it is multihomed to two VTEPs in the EVPN overlay. Only one of the two VTEPs has to be elected a DF to deliver BUM traffic to Server1 from remote VTEPs (VTEP3 and VTEP4). The administrator needs to configure EVPN multihoming preference for the Ethernet Segment. A higher preference value wins the election process and assumes the role of the DF. If the preference on both the VTEPs is same then the VTEP with the smallest IP wins the election. It’s important to note that the interface connecting the Ethernet Segment must be an Aggregate-Ethernet interface.
For every Ethernet Segment there must be only one DF.
The following command can be used to set the ES DF preference.
set interface aggregate-ethernet <interface> evpn mh es-df-pref <preference>
VTEP Uplink Status Tracking
Overlay VTEPs are capable of tracking the state of uplinks connecting the overlay leaf devices with underlay spine switches. This feature is automatically enabled when the user enables the EVPN MH. All the downstream links connecting the MH hosts are put in protocol down state when all the uplinks go down. In Figure 1, all the links connecting the underlay spine switches must go down before the downstream links are put in protocol down state. The links will not be put in protocol down state if there is only one link up and active connecting the overlay devices with the underlay.
EVPN MH Split Horizon
Split-horizon technique is used to stop BUM traffic from forming a loop in the EVPN VXLAN network that employs multihoming. Using split-horizon, BUM traffic from the same Ethernet Segment will be dropped by the peer device. As can been seen in Figure 2, Host1 is multihomed into the EVPN VXLAN fabric and connected to both VTEP1 and VTEP2 forming Ethernet Segment ES1.
In the example below, if Host1 sends some BUM traffic to DF and then it is forwarded to the spine switch and then relayed back to VTEP2, that traffic will be discarded at VTEP2 since VTEP2 has an interface configured for the same Ethernet Segment. Split horizon mechanism ensures BUM traffic can’t loop back to Host1 if Host1 originates this traffic.
Figure 2. EVPN MH Split-Horizon
Key points to note for EVPN MH Split Horizon and BUM traffic duplication:
- Only one DF is elected in one ESI (one LAG), so partial connection is not allowed for EVPN MH access connection.
- Non-DF node will drop the BUM traffic from other nodes, including peer of the same ES. BUM traffic from remote VTEPs is dropped by non-DF nodes to avoid duplication.
Note
Note: Single homed ports connecting a single device with the EVPN VXLAN fabric are not yet supported on VTEP devices. If a server is dual homed to two VTEPs and the connecting link to one of the VTEP fails, the server will still be considered a dual homed device even if there is only one active link connecting it to the fabric.
EVPN MH Basic Configuration
To configure EVPN multihoming on a LAG interface, the user needs to configure an Ethernet Segment identifier (ES-ID) and an Ethernet Segment System MAC. A type-3 ESI value is automatically generated by using these two parameters. The resulting 10-byte ESI value has the following format, where the XXs denote the 6-byte ES System MAC and the YYs denote the 3-byte local ES ID.
03 : XX : XX : XX : XX : XX : XX : YY : YY : YY
For each Ethernet segment, both the ES System MAC and the ES ID must be the same on all the switches forming the EVPN MH redundancy group. However, switches can be configured with different system MACs for different Ethernet segments or share the same system MAC for all Ethernet segments.
Aliasing
The aliasing feature provides remote VTEP devices the ability to load balance Layer-2 unicast traffic through other VTEPs that have configured the same ES (Ethernet Segment). VTEP devices in the EVPN multihoming redundancy group always advertise the communicated information of the same ES to all remote VTEPs via MP-BGP protocol. If one of the VTEP device in an ES, say VTEP1 in the following figure, learns the MAC/ARP information and advertises it to the remote VTEP, say VTEP3. Then, VTEP3 adds not only the IP address of VTEP1 as the next hop, but also adds the IP addresses of other VTEPs in the EVPN multihoming redundancy group as the next hop, to automatically form equivalent paths between different VTEPs.
EVPN multihoming aliasing function uses the ECMP hash fields "ingress-interface","ip-destination", "ip-source," "port-destination," "port-source," and "vlan" for equivalent paths calculation. By default, all these fields are enabled. Users can run the following CLI commands to enable or disable the ECMP hash fields: set interface ecmp hash-mapping field ingress-interface disable set interface ecmp hash-mapping field ip-destination disable set interface ecmp hash-mapping field ip-protocol disable set interface ecmp hash-mapping field ip-source disable set interface ecmp hash-mapping field port-destination disable set interface ecmp hash-mapping field port-source disable set interface ecmp hash-mapping field vlan disable |
On the remote VTEP, run the following EVPN MAC show commands to view the next hop. It shows the next hop interface is vplag, including all VTEPs in remote ES.
admin@Xorplus# run show mac-address table Total entries in switching table: 1 Static entries in switching table: 0 Dynamic entries in switching table: 1 VLAN MAC address Type Age Interfaces User ---- ----------------- --------- ---- ---------------- ---------- N/A 00:0c:29:77:8b:15 Dynamic 300 vplag-0x1 xorp admin@Xorplus# run show vxlan address-table VNID MAC address Type Interface VTEP ----------- ----------------- ------- ---------------- --------------- 10020 00:0c:29:77:8b:15 Dynamic 10.226.14.253,10.226.14.254 Entries in access port: 0 Entries in network port: 1
Switch Configuration Example and Topology
In PICOS EVPN multihoming, there are two ways to forward BUM messages. You can choose one of these to apply to your EVPN fabric.
- Head End Replication. This is the default way which needs no PIM configurations. The BUM message will be encapsulated and sent to all outgoing ports VTEPs of the same VNI by the access VTEP.
- BUM tunnel. All VTEPs of the same VNI join the same multicast group, and use multicast routing protocol PIM to establish a multicast forwarding table entry for the multicast group. This method can reduce the traffic flooding caused by the head end replication of the BUM messages. This way needs to configure PIM protocol.
EVPN Multihoming with Head End Replication
For this example configuration we have the following topology comprising of two spine switches, four leaf switches and four servers that are multihomed to this network. Two servers are multihomed to Leaf1 and Leaf2 switches whereas the other two servers are attached to Leaf3 and Leaf4 switches.
The four leaf switches form two MH redundancy groups for the four ES IDs. Two ES ID’s are configured under Leaf1 and Leaf2 whereas, another two ES ID’s are configured under Leaf3 and Leaf4. Each leaf switch has one physical interface added to a LAG interface to connect to the downstream server.
The two ES IDs are configured on each peer connected to the two downstream multihomed servers. On Leaf1, two physical ports connect to the multihomed servers; both the physical ports are first added to LAG interfaces. The LAG interface connected to Server1 is “ae2” whereas the interface connecting Server2 is “ae1”. For EVPN MH, it is a mandatory that the ports connecting the multihomed hosts must be LAG interfaces. Similarly, on all leaf switches there are two LAG interfaces and two ES IDs configured.
To implement a seamless migration of servers without changing network settings or disrupting the traffic forwarding, configure the same anycast gateway IP and the same virtual MAC (router MAC) on all four leaf switches to enable the anycast gateway function.
The following configuration examples use the topology illustrated below and configure EVPN multihoming with head end replication.
Figure 3. EVPN MH Example Topology
Note
Note: The physical ports connecting the multihomed servers or hosts must be added to a LAG interface and then configured for EVPN MH. Physical interfaces such as ge-1/1/1 cannot be directly used for MH configuration.
Leaf1 Configuration
Step 1. Configure LAG interfaces, set the LAG interface MTU, configure ES ID, configure the LAG interface VLAN membership and the LAG interface trunk mode. MTU configuration is optional, it can be configured when the MTU size needs to be of a particular size based on the customer use case environment. The LACP configuration is also optional.
admin@Leaf1# set interface aggregate-ethernet ae1 aggregated-ether-options lacp enable true admin@Leaf1# set interface aggregate-ethernet ae2 aggregated-ether-options lacp enable true admin@Leaf1# set interface aggregate-ethernet ae1 family ethernet-switching port-mode trunk admin@Leaf1# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 30 admin@Leaf1# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 20 admin@Leaf1# set interface aggregate-ethernet ae2 family ethernet-switching port-mode trunk admin@Leaf1# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 30 admin@Leaf1# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 20 admin@Leaf1# set interface aggregate-ethernet ae1 evpn mh es-id 4 admin@Leaf1# set interface aggregate-ethernet ae2 evpn mh es-id 2
Step2. Configure ES system MAC, it can be the same or different for different ethernet segments. Configure the physical interfaces and VLAN membership and the ports trunk mode. Also add physical interfaces to LAG interfaces.
admin@Leaf1# set interface aggregate-ethernet ae1 evpn mh es-sys-mac 00:22:22:22:22:22 admin@Leaf1# set interface aggregate-ethernet ae2 evpn mh es-sys-mac 00:22:22:22:22:22 admin@Leaf1# set interface gigabit-ethernet ge-1/1/19 mtu 3000 admin@Leaf1# set interface gigabit-ethernet ge-1/1/19 family ethernet-switching native-vlan-id 4014 admin@Leaf1# set interface gigabit-ethernet ge-1/1/23 ether-options 802.3ad ae2 admin@Leaf1# set interface gigabit-ethernet ge-1/1/23 family ethernet-switching admin@Leaf1# set interface gigabit-ethernet ge-1/1/24 disable true admin@Leaf1# set interface gigabit-ethernet ge-1/1/27 mtu 3000 admin@Leaf1# set interface gigabit-ethernet ge-1/1/27 family ethernet-switching native-vlan-id 4012 admin@Leaf1# set interface gigabit-ethernet ge-1/1/37 ether-options 802.3ad ae1
Step3. Configure VRF and enable IP routing.
admin@Leaf1# set ip routing enable true admin@Leaf1# set ip vrf vrf1
Step4. Configure loopback interfaces and assign IP addresses.
admin@Leaf1# set l3-interface loopback lo address 10.226.14.254 prefix-length 32 admin@Leaf1# set l3-interface loopback lo address 10.226.14.207 prefix-length 32 admin@Leaf1# set l3-interface loopback vrf1 address 17.17.17.17 prefix-length 32
Step5. Configure VLAN interfaces and assign IP addresses, configure anycast gateway.
admin@Leaf1# set l3-interface vlan-interface vlan4014 address 10.226.15.65 prefix-length 30 admin@Leaf1# set l3-interface vlan-interface vlan4012 address 10.226.15.29 prefix-length 30 admin@Leaf1# set l3-interface vlan-interface vlan4094 address 10.226.13.26 prefix-length 30 admin@Leaf1# set l3-interface vlan-interface vlan99 vrf vrf1 admin@Leaf1# set l3-interface vlan-interface vlan20 vrf vrf1 admin@Leaf1# set l3-interface vlan-interface vlan20 address 192.168.0.1 prefix-length 24 admin@Leaf1# set l3-interface vlan-interface vlan20 address 2002:0:0:1::1 prefix-length 64 admin@Leaf1# set l3-interface vlan-interface vlan30 vrf vrf1 admin@Leaf1# set l3-interface vlan-interface vlan30 address 172.168.0.1 prefix-length 24 admin@Leaf1# set l3-interface vlan-interface vlan30 address 2003:0:0:1::1 prefix-length 64 admin@Leaf1# set l3-interface vlan-interface vlan20 anycast address 192.168.0.20 prefix-length 24 admin@Leaf1# set l3-interface vlan-interface vlan20 anycast address 2002:0:0:1::20 prefix-length 64 admin@Leaf1# set l3-interface vlan-interface vlan30 anycast address 172.168.0.30 prefix-length 24 admin@Leaf1# set l3-interface vlan-interface vlan30 anycast address 2003:0:0:1::30 prefix-length 64 admin@Leaf1# set l3-interface vlan-interface vlan20 anycast mac 00:00:20:00:00:FE admin@Leaf1# set l3-interface vlan-interface vlan30 anycast mac 00:00:30:00:00:FE
Step6. Configure BGP and EVPN.
admin@Leaf1# set protocols bgp local-as 65001 admin@Leaf1# set protocols bgp router-id 10.226.14.207 admin@Leaf1# set protocols bgp neighbor 10.226.14.48 remote-as internal admin@Leaf1# set protocols bgp neighbor 10.226.14.48 update-source 10.226.14.207 admin@Leaf1# set protocols bgp neighbor 10.226.14.48 evpn activate true admin@Leaf1# set protocols bgp neighbor 10.226.14.208 remote-as internal admin@Leaf1# set protocols bgp neighbor 10.226.14.208 update-source 10.226.14.207 admin@Leaf1# set protocols bgp neighbor 10.226.14.208 evpn activate true admin@Leaf1# set protocols bgp neighbor 10.226.14.24 remote-as internal admin@Leaf1# set protocols bgp neighbor 10.226.14.24 update-source 10.226.14.207 admin@Leaf1# set protocols bgp neighbor 10.226.14.24 evpn activate true admin@Leaf1# set protocols bgp ipv4-unicast admin@Leaf1# set protocols bgp evpn advertise-all-vni admin@Leaf1# set protocols bgp evpn advertise ipv4-unicast admin@Leaf1# set protocols bgp evpn advertise ipv6-unicast admin@Leaf1# set protocols bgp evpn advertise-svi-ip admin@Leaf1# set protocols bgp vrf vrf1 local-as 65001 admin@Leaf1# set protocols bgp vrf vrf1 router-id 17.17.17.17 admin@Leaf1# set protocols bgp vrf vrf1 evpn advertise ipv4-unicast admin@Leaf1# set protocols bgp vrf vrf1 evpn advertise ipv6-unicast admin@Leaf1# set protocols evpn mh admin@Leaf1# set protocols lldp enable true
Step7. Configure OSPF.
admin@Leaf1# set protocols ospf router-id 17.17.17.17 admin@Leaf1# set protocols ospf network 10.226.15.64/30 area 0.0.0.0 admin@Leaf1# set protocols ospf network 10.226.15.28/30 area 0.0.0.0 admin@Leaf1# set protocols ospf network 10.226.14.254/32 area 0.0.0.0 admin@Leaf1# set protocols ospf network 10.226.14.207/32 area 0.0.0.0
Step8. Configure hostname, inband mode and VLANs.
admin@Leaf1# set system hostname Leaf1 admin@Leaf1# set system inband enable true admin@Leaf1# set vlans vlan-id 20 l3-interface vlan20 admin@Leaf1# set vlans vlan-id 30 l3-interface vlan30 admin@Leaf1# set vlans vlan-id 99 l3-interface vlan99 admin@Leaf1# set vlans vlan-id 4012 l3-interface vlan4012 admin@Leaf1# set vlans vlan-id 4014 l3-interface vlan4014 admin@Leaf1# set vlans vlan-id 4094 l3-interface vlan4094
Step9. Configure VXLANs.
admin@Leaf1# set vxlans source-interface lo address 10.226.14.254 admin@Leaf1# set vxlans vni 10030 decapsulation mode service-vlan-per-port admin@Leaf1# set vxlans vni 10030 vlan 30 admin@Leaf1# set vxlans vni 10030 arp-nd-suppress disable false admin@Leaf1# set vxlans vni 10020 decapsulation mode service-vlan-per-port admin@Leaf1# set vxlans vni 10020 vlan 20 admin@Leaf1# set vxlans vni 10020 arp-nd-suppress disable false admin@Leaf1# set vxlans vni 99 vlan 99 admin@Leaf1# set vxlans vrf vrf1 l3-vni 99 admin@Leaf1# commit
Leaf2 Configuration
admin@Leaf2# set interface aggregate-ethernet ae1 aggregated-ether-options lacp enable true admin@Leaf2# set interface aggregate-ethernet ae2 aggregated-ether-options lacp enable true admin@Leaf2# set interface aggregate-ethernet ae1 family ethernet-switching port-mode trunk admin@Leaf2# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 30 admin@Leaf2# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 20 admin@Leaf2# set interface aggregate-ethernet ae1 evpn mh es-id 4 admin@Leaf2# set interface aggregate-ethernet ae1 evpn mh es-sys-mac 00:22:22:22:22:22 admin@Leaf2# set interface aggregate-ethernet ae2 family ethernet-switching port-mode "trunk" admin@Leaf2# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 30 admin@Leaf2# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 20 admin@Leaf2# set interface aggregate-ethernet ae2 evpn mh es-id 2 admin@Leaf2# set interface aggregate-ethernet ae2 evpn mh es-sys-mac 00:22:22:22:22:22 admin@Leaf2# set interface gigabit-ethernet te-1/1/17 mtu 3000 admin@Leaf2# set interface gigabit-ethernet te-1/1/17 family ethernet-switching native-vlan-id 4014 admin@Leaf2# set interface gigabit-ethernet te-1/1/19 mtu 3000 admin@Leaf2# set interface gigabit-ethernet te-1/1/19 family ethernet-switching native-vlan-id 4013 admin@Leaf2# set interface gigabit-ethernet te-1/1/21 ether-options 802.3ad ae2 admin@Leaf2# set interface gigabit-ethernet te-1/1/23 ether-options 802.3ad ae1 admin@Leaf2# set interface ethernet-switching-options admin@Leaf2# set ip routing enable true admin@Leaf2# set ip vrf vrf1 admin@Leaf2# set l3-interface loopback lo address 10.226.14.253 prefix-length 32 admin@Leaf2# set l3-interface loopback vrf1 address 26.26.26.26 prefix-length 32 admin@Leaf2# set l3-interface vlan-interface vlan4013 address 10.226.15.25 prefix-length 30 admin@Leaf2# set l3-interface vlan-interface vlan4014 address 10.226.15.81 prefix-length 30 admin@Leaf2# set l3-interface vlan-interface vlan4094 address 10.226.13.25 prefix-length 30 admin@Leaf2# set l3-interface vlan-interface vlan99 vrf vrf1 admin@Leaf2# set l3-interface vlan-interface vlan20 vrf vrf1 admin@Leaf2# set l3-interface vlan-interface vlan20 address 192.168.0.2 prefix-length 24 admin@Leaf2# set l3-interface vlan-interface vlan20 address 2002:0:0:1::2 prefix-length 64 admin@Leaf2# set l3-interface vlan-interface vlan30 vrf vrf1 admin@Leaf2# set l3-interface vlan-interface vlan30 address 172.168.0.2 prefix-length 24 admin@Leaf2# set l3-interface vlan-interface vlan30 address 2003:0:0:1::2 prefix-length 64 admin@Leaf2# set l3-interface vlan-interface vlan20 anycast address 192.168.0.20 prefix-length 24 admin@Leaf2# set l3-interface vlan-interface vlan20 anycast address 2002:0:0:1::20 prefix-length 64 admin@Leaf2# set l3-interface vlan-interface vlan30 anycast address 172.168.0.30 prefix-length 24 admin@Leaf2# set l3-interface vlan-interface vlan30 anycast address 2003:0:0:1::30 prefix-length 64 admin@Leaf2# set l3-interface vlan-interface vlan20 anycast mac 00:00:20:00:00:FE admin@Leaf2# set l3-interface vlan-interface vlan30 anycast mac 00:00:30:00:00:FE admin@Leaf2# set protocols bgp local-as 65001 admin@Leaf2# set protocols bgp router-id 10.226.14.208 admin@Leaf2# set protocols bgp neighbor 10.226.14.48 remote-as internal admin@Leaf2# set protocols bgp neighbor 10.226.14.48 update-source 10.226.14.208 admin@Leaf2# set protocols bgp neighbor 10.226.14.48 evpn activate true admin@Leaf2# set protocols bgp neighbor 10.226.14.207 remote-as internal admin@Leaf2# set protocols bgp neighbor 10.226.14.207 update-source 10.226.14.208 admin@Leaf2# set protocols bgp neighbor 10.226.14.207 evpn activate true admin@Leaf2# set protocols bgp neighbor 10.226.14.24 remote-as internal admin@Leaf2# set protocols bgp neighbor 10.226.14.24 update-source 10.226.14.208 admin@Leaf2# set protocols bgp neighbor 10.226.14.24 evpn activate true admin@Leaf2# set protocols bgp ipv4-unicast admin@Leaf2# set protocols bgp evpn advertise-all-vni admin@Leaf2# set protocols bgp evpn advertise ipv4-unicast admin@Leaf2# set protocols bgp evpn advertise ipv6-unicast admin@Leaf2# set protocols bgp evpn advertise-svi-ip admin@Leaf2# set protocols bgp vrf vrf1 local-as 65001 admin@Leaf2# set protocols bgp vrf vrf1 router-id 26.26.26.26 admin@Leaf2# set protocols bgp vrf vrf1 evpn advertise ipv4-unicast admin@Leaf2# set protocols bgp vrf vrf1 evpn advertise ipv6-unicast admin@Leaf2# set protocols evpn mh admin@Leaf2# set protocols lldp enable true admin@Leaf2# set protocols ospf router-id 110.110.110.110 admin@Leaf2# set protocols ospf network 10.226.15.24/30 area 0.0.0.0 admin@Leaf2# set protocols ospf network 10.226.15.80/30 area 0.0.0.0 admin@Leaf2# set protocols ospf network 10.226.14.208/32 area 0.0.0.0 admin@Leaf2# set protocols ospf network 10.226.14.253/32 area 0.0.0.0 admin@Leaf2# set vlans vlan-id 20 l3-interface vlan20 admin@Leaf2# set vlans vlan-id 30 l3-interface vlan30 admin@Leaf2# set vlans vlan-id 99 l3-interface vlan99 admin@Leaf2# set vlans vlan-id 4013 l3-interface vlan4013 admin@Leaf2# set vlans vlan-id 4014 l3-interface vlan4014 admin@Leaf2# set vlans vlan-id 4094 l3-interface vlan4094 admin@Leaf2# set vxlans source-interface lo address 10.226.14.253 admin@Leaf2# set vxlans vni 10030 decapsulation mode service-vlan-per-port admin@Leaf2# set vxlans vni 10030 vlan 30 admin@Leaf2# set vxlans vni 10030 arp-nd-suppress disable false admin@Leaf2# set vxlans vni 10020 decapsulation mode service-vlan-per-port admin@Leaf2# set vxlans vni 10020 vlan 20 admin@Leaf2# set vxlans vni 10020 arp-nd-suppress disable false admin@Leaf2# set vxlans vni 99 vlan 99 admin@Leaf2# set vxlans vrf vrf1 l3-vni 99 admin@Leaf2# commit
Leaf3 Configuration
admin@Leaf3# set interface aggregate-ethernet ae1 aggregated-ether-options lacp enable true admin@Leaf3# set interface aggregate-ethernet ae2 aggregated-ether-options lacp enable true admin@Leaf3# set interface aggregate-ethernet ae1 family ethernet-switching port-mode trunk admin@Leaf3# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 30 admin@Leaf3# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 20 admin@Leaf3# set interface aggregate-ethernet ae1 evpn mh es-id 3 admin@Leaf3# set interface aggregate-ethernet ae1 evpn mh es-sys-mac 00:00:00:22:22:22 admin@Leaf3# set interface aggregate-ethernet ae2 family ethernet-switching port-mode trunk admin@Leaf3# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 30 admin@Leaf3# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 20 admin@Leaf3# set interface aggregate-ethernet ae2 evpn mh es-id 1 admin@Leaf3# set interface aggregate-ethernet ae2 evpn mh es-sys-mac 00:00:00:22:22:22 admin@Leaf3# set interface gigabit-ethernet ge-1/1/3 mtu 3000 admin@Leaf3# set interface gigabit-ethernet ge-1/1/3 family ethernet-switching native-vlan-id 3011 admin@Leaf3# set interface gigabit-ethernet ge-1/1/5 mtu 3000 admin@Leaf3# set interface gigabit-ethernet ge-1/1/5 family ethernet-switching native-vlan-id 3012 admin@Leaf3# set interface gigabit-ethernet ge-1/1/21 ether-options 802.3ad ae1 admin@Leaf3# set interface gigabit-ethernet ge-1/1/22 ether-options 802.3ad ae2 admin@Leaf3# set interface ethernet-switching-options admin@Leaf3# set ip routing enable true admin@Leaf3# set ip vrf vrf1 admin@Leaf3# set l3-interface loopback lo address 10.226.14.24 prefix-length 32 admin@Leaf3# set l3-interface loopback lo address 10.226.14.201 prefix-length 32 admin@Leaf3# set l3-interface vlan-interface vlan3011 address 10.226.0.98 prefix-length 30 admin@Leaf3# set l3-interface vlan-interface vlan3012 address 10.226.0.94 prefix-length 30 admin@Leaf3# set l3-interface vlan-interface vlan99 vrf vrf1 admin@Leaf3# set l3-interface vlan-interface vlan20 vrf vrf1 admin@Leaf3# set l3-interface vlan-interface vlan20 address 192.168.0.3 prefix-length 24 admin@Leaf3# set l3-interface vlan-interface vlan20 address 2002:0:0:1::3 prefix-length 64 admin@Leaf3# set l3-interface vlan-interface vlan30 vrf vrf1 admin@Leaf3# set l3-interface vlan-interface vlan30 address 172.168.0.3 prefix-length 24 admin@Leaf3# set l3-interface vlan-interface vlan30 address 2003:0:0:1::3 prefix-length 64 admin@Leaf3# set l3-interface vlan-interface vlan20 anycast address 192.168.0.20 prefix-length 24 admin@Leaf3# set l3-interface vlan-interface vlan20 anycast address 2002:0:0:1::20 prefix-length 64 admin@Leaf3# set l3-interface vlan-interface vlan30 anycast address 172.168.0.30 prefix-length 24 admin@Leaf3# set l3-interface vlan-interface vlan30 anycast address 2003:0:0:1::30 prefix-length 64 admin@Leaf3# set l3-interface vlan-interface vlan20 anycast mac 00:00:20:00:00:FE admin@Leaf3# set l3-interface vlan-interface vlan30 anycast mac 00:00:30:00:00:FE admin@Leaf3# set protocols bgp local-as 65001 admin@Leaf3# set protocols bgp router-id 10.226.14.24 admin@Leaf3# set protocols bgp neighbor 10.226.14.207 remote-as internal admin@Leaf3# set protocols bgp neighbor 10.226.14.207 timers delayopen 60 admin@Leaf3# set protocols bgp neighbor 10.226.14.207 update-source "10.226.14.24" admin@Leaf3# set protocols bgp neighbor 10.226.14.207 evpn activate true admin@Leaf3# set protocols bgp neighbor 10.226.14.208 remote-as "internal" admin@Leaf3# set protocols bgp neighbor 10.226.14.208 timers delayopen 60 admin@Leaf3# set protocols bgp neighbor 10.226.14.208 update-source 10.226.14.24 admin@Leaf3# set protocols bgp neighbor 10.226.14.208 evpn activate true admin@Leaf3# set protocols bgp neighbor 10.226.14.48 remote-as internal admin@Leaf3# set protocols bgp neighbor 10.226.14.48 timers delayopen 60 admin@Leaf3# set protocols bgp neighbor 10.226.14.48 update-source 10.226.14.24 admin@Leaf3# set protocols bgp neighbor 10.226.14.48 evpn activate true admin@Leaf3# set protocols bgp ipv4-unicast admin@Leaf3# set protocols bgp evpn advertise-all-vni admin@Leaf3# set protocols bgp evpn advertise ipv4-unicast admin@Leaf3# set protocols bgp evpn advertise ipv6-unicast admin@Leaf3# set protocols bgp evpn advertise-svi-ip admin@Leaf3# set protocols lldp enable true admin@Leaf3# set protocols ospf router-id 24.24.24.24 admin@Leaf3# set protocols ospf network 10.226.0.96/30 area 0.0.0.0 admin@Leaf3# set protocols ospf network 10.226.0.92/30 area 0.0.0.0 admin@Leaf3# set protocols ospf network 10.226.14.201/32 area 0.0.0.0 admin@Leaf3# set protocols ospf network 10.226.14.24/32 area 0.0.0.0 admin@Leaf3# set vlans vlan-id 20 l3-interface vlan20 admin@Leaf3# set vlans vlan-id 30 l3-interface vlan30 admin@Leaf3# set vlans vlan-id 99 l3-interface vlan99 admin@Leaf3# set vlans vlan-id 3012 l3-interface vlan3012 admin@Leaf3# set vlans vlan-id 3011 l3-interface vlan3011 admin@Leaf3# set vxlans source-interface lo address 10.226.14.201 admin@Leaf3# set vxlans vni 10020 decapsulation mode service-vlan-per-port admin@Leaf3# set vxlans vni 10020 vlan 20 admin@Leaf3# set vxlans vni 10020 arp-nd-suppress disable false admin@Leaf3# set vxlans vni 10030 decapsulation mode service-vlan-per-port admin@Leaf3# set vxlans vni 10030 vlan 30 admin@Leaf3# set vxlans vni 10030 arp-nd-suppress disable false admin@Leaf3# set vxlans vni 99 vlan 99 admin@Leaf3# set vxlans traceoptions flag all disable false admin@Leaf3# set vxlans vrf vrf1 l3-vni 99 admin@Leaf3# commit
Leaf4 Configuration
admin@Leaf4# set interface aggregate-ethernet ae1 aggregated-ether-options lacp enable true admin@Leaf4# set interface aggregate-ethernet ae2 aggregated-ether-options lacp enable true admin@Leaf4# set interface aggregate-ethernet ae1 family ethernet-switching port-mode trunk admin@Leaf4# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 30 admin@Leaf4# set interface aggregate-ethernet ae1 family ethernet-switching vlan members 20 admin@Leaf4# set interface aggregate-ethernet ae1 evpn mh es-id 1 admin@Leaf4# set interface aggregate-ethernet ae1 evpn mh es-sys-mac 00:00:00:22:22:22 admin@Leaf4# set interface aggregate-ethernet ae2 mtu 3000 admin@Leaf4# set interface aggregate-ethernet ae2 family ethernet-switching port-mode trunk admin@Leaf4# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 30 admin@Leaf4# set interface aggregate-ethernet ae2 family ethernet-switching vlan members 20 admin@Leaf4# set interface aggregate-ethernet ae2 evpn mh es-id 3 admin@Leaf4# set interface aggregate-ethernet ae2 evpn mh es-sys-mac 00:00:00:22:22:22 admin@Leaf4# set interface gigabit-ethernet ge-1/1/3 mtu 3000 admin@Leaf4# set interface gigabit-ethernet ge-1/1/3 family ethernet-switching native-vlan-id 3011 admin@Leaf4# set interface gigabit-ethernet ge-1/1/5 mtu 3000 admin@Leaf4# set interface gigabit-ethernet ge-1/1/5 family ethernet-switching native-vlan-id 3012 admin@Leaf4# set interface gigabit-ethernet ge-1/1/21 ether-options 802.3ad ae1 admin@Leaf4# set interface gigabit-ethernet ge-1/1/22 ether-options 802.3ad ae2 admin@Leaf4# set ip routing enable true admin@Leaf4# set ip vrf vrf1 admin@Leaf4# set l3-interface loopback lo address 10.226.14.202 prefix-length 32 admin@Leaf4# set l3-interface loopback lo address 10.226.14.48 prefix-length 32 admin@Leaf4# set l3-interface loopback vrf1 address 48.48.48.48 prefix-length 32 admin@Leaf4# set l3-interface vlan-interface vlan3011 address 10.226.15.70 prefix-length 30 admin@Leaf4# set l3-interface vlan-interface vlan3012 address 10.226.15.78 prefix-length 30 admin@Leaf4# set l3-interface vlan-interface vlan99 vrf vrf1 admin@Leaf4# set l3-interface vlan-interface vlan20 vrf vrf1 admin@Leaf4# set l3-interface vlan-interface vlan20 address 192.168.0.4 prefix-length 24 admin@Leaf4# set l3-interface vlan-interface vlan20 address 2002:0:0:1::4 prefix-length 64 admin@Leaf4# set l3-interface vlan-interface vlan30 vrf vrf1 admin@Leaf4# set l3-interface vlan-interface vlan30 address 172.168.0.4 prefix-length 24 admin@Leaf4# set l3-interface vlan-interface vlan30 address 2003:0:0:1::4 prefix-length 64 admin@Leaf4# set l3-interface vlan-interface vlan20 anycast address 192.168.0.20 prefix-length 24 admin@Leaf4# set l3-interface vlan-interface vlan20 anycast address 2002:0:0:1::20 prefix-length 64 admin@Leaf4# set l3-interface vlan-interface vlan30 anycast address 172.168.0.30 prefix-length 24 admin@Leaf4# set l3-interface vlan-interface vlan30 anycast address 2003:0:0:1::30 prefix-length 64 admin@Leaf4# set l3-interface vlan-interface vlan20 anycast mac 00:00:20:00:00:FE admin@Leaf4# set l3-interface vlan-interface vlan30 anycast mac 00:00:30:00:00:FE admin@Leaf4# set protocols bgp local-as 65001 admin@Leaf4# set protocols bgp router-id 10.226.14.48 admin@Leaf4# set protocols bgp neighbor 10.226.14.207 remote-as internal admin@Leaf4# set protocols bgp neighbor 10.226.14.207 update-source 10.226.14.48 admin@Leaf4# set protocols bgp neighbor 10.226.14.207 evpn activate true admin@Leaf4# set protocols bgp neighbor 10.226.14.208 remote-as internal admin@Leaf4# set protocols bgp neighbor 10.226.14.208 update-source 10.226.14.48 admin@Leaf4# set protocols bgp neighbor 10.226.14.208 evpn activate true admin@Leaf4# set protocols bgp neighbor 10.226.14.24 remote-as internal admin@Leaf4# set protocols bgp neighbor 10.226.14.24 update-source 10.226.14.48 admin@Leaf4# set protocols bgp neighbor 10.226.14.24 evpn activate true admin@Leaf4# set protocols bgp ipv4-unicast admin@Leaf4# set protocols bgp evpn advertise-all-vni admin@Leaf4# set protocols bgp evpn advertise ipv4-unicast admin@Leaf4# set protocols bgp evpn advertise ipv6-unicast admin@Leaf4# set protocols bgp evpn advertise-svi-ip admin@Leaf4# set protocols bgp vrf vrf1 local-as 65001 admin@Leaf4# set protocols bgp vrf vrf1 router-id 48.48.48.48 admin@Leaf4# set protocols bgp vrf vrf1 evpn advertise ipv4-unicast admin@Leaf4# set protocols bgp vrf vrf1 evpn advertise ipv6-unicast admin@Leaf4# set protocols evpn mh startup-delay 60 admin@Leaf4# set protocols lldp enable true admin@Leaf4# set protocols ospf router-id 48.48.48.48 admin@Leaf4# set protocols ospf network 10.226.14.202/32 area 0.0.0.0 admin@Leaf4# set protocols ospf network 10.226.14.48/32 area 0.0.0.0 admin@Leaf4# set protocols ospf network 10.226.15.68/30 area 0.0.0.0 admin@Leaf4# set protocols ospf network 10.226.15.76/30 area 0.0.0.0 admin@Leaf4# set vlans vlan-id 20 l3-interface vlan20 admin@Leaf4# set vlans vlan-id 30 l3-interface vlan30 admin@Leaf4# set vlans vlan-id 99 l3-interface vlan99 admin@Leaf4# set vlans vlan-id 3011 l3-interface vlan3011 admin@Leaf4# set vlans vlan-id 3012 l3-interface vlan3012 admin@Leaf4# set vxlans source-interface lo address 10.226.14.202 admin@Leaf4# set vxlans vni 10020 decapsulation mode service-vlan-per-port admin@Leaf4# set vxlans vni 10020 vlan 20 admin@Leaf4# set vxlans vni 10020 arp-nd-suppress disable false admin@Leaf4# set vxlans vni 10030 decapsulation mode service-vlan-per-port admin@Leaf4# set vxlans vni 10030 vlan 30 admin@Leaf4# set vxlans vni 10030 arp-nd-suppress disable false admin@Leaf4# set vxlans vni 99 vlan 99 admin@Leaf4# set vxlans vrf vrf1 l3-vni 99 admin@Leaf4# commit
Spine1 Configuration
admin@Spine1# set interface gigabit-ethernet te-1/1/35 mtu 2000 admin@Spine1# set interface gigabit-ethernet te-1/1/35 family ethernet-switching native-vlan-id 4013 admin@Spine1# set interface gigabit-ethernet te-1/1/41 mtu 2000 admin@Spine1# set interface gigabit-ethernet te-1/1/41 family ethernet-switching native-vlan-id 4014 admin@Spine1# set interface gigabit-ethernet te-1/1/44 mtu 2000 admin@Spine1# set interface gigabit-ethernet te-1/1/44 family ethernet-switching native-vlan-id 4016 admin@Spine1# set interface gigabit-ethernet te-1/1/45 family ethernet-switching native-vlan-id 4015 admin@Spine1# set ip routing enable true admin@Spine1# set l3-interface loopback lo address 185.185.185.185 prefix-length 32 admin@Spine1# set l3-interface vlan-interface vlan4014 address 10.226.15.66 prefix-length 30 admin@Spine1# set l3-interface vlan-interface vlan4013 address 10.226.15.26 prefix-length 30 admin@Spine1# set l3-interface vlan-interface vlan4015 address 10.226.15.69 prefix-length 30 admin@Spine1# set l3-interface vlan-interface vlan4016 address 10.226.0.97 prefix-length 30 admin@Spine1# set protocols spanning-tree enable false admin@Spine1# set protocols ospf router-id 185.185.185.185 admin@Spine1# set protocols ospf network 10.226.15.64/30 area 0.0.0.0 admin@Spine1# set protocols ospf network 10.226.15.24/30 area 0.0.0.0 admin@Spine1# set protocols ospf network 10.226.15.68/30 area 0.0.0.0 admin@Spine1# set protocols ospf network 10.226.0.96/30 area 0.0.0.0 admin@Spine1# set protocols ospf network 185.185.185.185/32 area 0.0.0.0 admin@Spine1# set vlans vlan-id 4013 l3-interface vlan4013 admin@Spine1# set vlans vlan-id 4014 l3-interface vlan4014 admin@Spine1# set vlans vlan-id 4015 l3-interface vlan4015 admin@Spine1# set vlans vlan-id 4016 l3-interface vlan4016 admin@Spine1# commit
Spine2 Configuration
admin@Spine2# set interface gigabit-ethernet te-1/1/27 mtu 2000 admin@Spine2# set interface gigabit-ethernet te-1/1/27 family ethernet-switching native-vlan-id 4012 admin@Spine2# set interface gigabit-ethernet te-1/1/27 speed 1000 admin@Spine2# set interface gigabit-ethernet te-1/1/37 mtu 2000 admin@Spine2# set interface gigabit-ethernet te-1/1/37 family ethernet-switching native-vlan-id 4014 admin@Spine2# set interface gigabit-ethernet te-1/1/37 speed 1000 admin@Spine2# set interface gigabit-ethernet te-1/1/44 speed 1000 admin@Spine2# set interface gigabit-ethernet te-1/1/44 family ethernet-switching native-vlan-id 4015 admin@Spine2# set interface gigabit-ethernet te-1/1/45 family ethernet-switching native-vlan-id 3012 admin@Spine2# set interface gigabit-ethernet te-1/1/45 speed 1000 admin@Spine2# set ip routing enable true admin@Spine2# set l3-interface loopback lo address 202.202.202.202 prefix-length 32 admin@Spine2# set l3-interface vlan-interface vlan4012 address 10.226.15.30 prefix-length 30 admin@Spine2# set l3-interface vlan-interface vlan4014 address 10.226.15.82 prefix-length 30 admin@Spine2# set l3-interface vlan-interface vlan4015 address 10.226.0.93 prefix-length 30 admin@Spine2# set l3-interface vlan-interface vlan3012 address 10.226.15.77 prefix-length 30 admin@Spine2# set protocols lldp enable true admin@Spine2# set protocols spanning-tree enable false admin@Spine2# set protocols ospf router-id 202.202.202.202 admin@Spine2# set protocols ospf network 10.226.0.92/30 area 0.0.0.0 admin@Spine2# set protocols ospf network 10.226.15.28/30 area 0.0.0.0 admin@Spine2# set protocols ospf network 10.226.15.80/30 area 0.0.0.0 admin@Spine2# set protocols ospf network 10.226.15.76/30 area 0.0.0.0 admin@Spine2# set protocols ospf network 202.202.202.202/32 area 0.0.0.0 admin@Spine2# set vlans vlan-id 4012 l3-interface vlan4012 admin@Spine2# set vlans vlan-id 4014 l3-interface vlan4014 admin@Spine2# set vlans vlan-id 4015 l3-interface vlan4015 admin@Spine2# set vlans vlan-id 3012 l3-interface vlan3012 admin@Spine2# commit
Verify Configuration
The above configuration uses the same default ES-DF preferences on all the three Leaf switches. The resulting effect is that the leaf switch with the smallest VTEP IP is selected as the DF. In the first EVPN MH redundancy group, Leaf2 is elected as the DF since it has the smallest VTEP IP in the group. Similarly in the second redundancy group, Leaf3 has the smallest VTEP IP and is elected the DF.
admin@Leaf1# run show evpn es Type: B bypass, L local, R remote, N non-DF ESI Type ES-IF VTEPs 03:00:22:22:22:22:22:00:00:04 LRN ae1 10.226.14.253 03:00:22:22:22:22:22:00:00:02 LRN ae2 10.226.14.253 admin@Leaf2# run show evpn es Type: B bypass, L local, R remote, N non-DF ESI Type ES-IF VTEPs 03:00:22:22:22:22:22:00:00:04 LR ae1 10.226.14.254 03:00:22:22:22:22:22:00:00:02 LR ae2 10.226.14.254 admin@Leaf3# run show evpn es Type: B bypass, L local, R remote, N non-DF ESI Type ES-IF VTEPs 03:00:00:00:22:22:22:00:00:03 LR ae1 10.226.14.202 03:00:00:00:22:22:22:00:00:01 LR ae2 10.226.14.202 admin@Leaf4# run show evpn es Type: B bypass, L local, R remote, N non-DF ESI Type ES-IF VTEPs 03:00:00:00:22:22:22:00:00:01 LRN ae1 10.226.14.201 03:00:00:00:22:22:22:00:00:03 LRN ae2 10.226.14.201
Now let’s change the ES-DF preference on Leaf1 to 65535 and see the output again. This time Leaf1 assumes the role of DF.
admin@Leaf1# set interface aggregate-ethernet ae1 evpn mh es-df-pref 65535 admin@Leaf1# set interface aggregate-ethernet ae2 evpn mh es-df-pref 65535 admin@Leaf1# commit Commit OK. Save done admin@Leaf1# run show evpn es Type: B bypass, L local, R remote, N non-DF ESI Type ES-IF VTEPs 03:00:22:22:22:22:22:00:00:04 LR ae1 10.226.14.253 03:00:22:22:22:22:22:00:00:02 LR ae2 10.226.14.253 admin@Leaf2# run show evpn es Type: B bypass, L local, R remote, N non-DF ESI Type ES-IF VTEPs 03:00:22:22:22:22:22:00:00:04 LRN ae1 10.226.14.254 03:00:22:22:22:22:22:00:00:02 LRN ae2 10.226.14.254
To check the EVPN ES detail information, run the following show command. This command shows important information such as ESI, Type, interface, state, DF status and the VTEPs etc.
admin@Leaf3# run show evpn es detail
ESI: 03:00:00:00:22:22:22:00:00:03
Type: Local,Remote
Interface: ae1
State: up
Bridge port: yes
Ready for BGP: yes
VNI Count: 2
MAC Count: 0
DF status: df
DF preference: 32767
Nexthop group: 536870913
VTEPs:
10.226.14.202 df_alg: preference df_pref: 32767 nh: 268435459
ESI: 03:00:00:00:22:22:22:00:00:01
Type: Local,Remote
Interface: ae2
State: up
Bridge port: yes
Ready for BGP: yes
VNI Count: 2
MAC Count: 0
DF status: df
DF preference: 32767
Nexthop group: 536870914
VTEPs:
10.226.14.202 df_alg: preference df_pref: 32767 nh: 268435459
Check the VXLAN address table with the following show command.
admin@Leaf3# run show vxlan address-table VNID MAC address Type Interface VTEP ----------- ----------------- ------- ---------------- --------------- 99 04:f8:f8:20:67:7b Dynamic 10.226.14.254 99 18:5a:58:37:55:e1 Dynamic 10.226.14.253 10020 00:1e:c9:bb:c0:3c Dynamic ge-1/1/19 10020 04:f8:f8:20:67:7b Dynamic 10.226.14.254 10020 18:5a:58:37:55:e1 Dynamic 10.226.14.253 10030 04:f8:f8:20:67:7b Dynamic 10.226.14.254 10030 18:5a:58:37:55:e1 Dynamic 10.226.14.253 10030 22:11:11:00:00:00 Dynamic ae1 Entries in access port: 2 Entries in network port: 7
To get the tunnel information within a VNI, run the following show command.
admin@Leaf3# run show vxlan tunnel vni 10030 Total number of tunnels: 2 VNI 10030, Encap:service-vlan-delete, Decap:service-vlan-per-port src addr:10.226.14.202, dst addr:10.226.14.253, state:UP traffic type:all Vtep type:EVPN nexthops:10.226.15.69 10.226.15.77 output ports:ge-1/1/5 ge-1/1/3 src addr:10.226.14.202, dst addr:10.226.14.254, state:UP traffic type:all Vtep type:EVPN nexthops:10.226.15.69 10.226.15.77 output ports:ge-1/1/5 ge-1/1/3
To get more info on interfaces belonging to a particular VNI, run the following show command.
admin@Leaf3# run show vxlan vni 10030 Interface ID Type Egress Vlan ID ----------- ----------- ----------- ------- -------- ae1 0x80000005 Access 100012 30 ae2 0x80000006 Access 100013 30 ge-1/1/23 0x80000007 Access 100014 30 ge-1/1/5 0x80000008 Network(UC) 200258 ge-1/1/5 0x8000000a Network(MC) 100020 ge-1/1/5 0x8000000b Network(UC) 200258 ge-1/1/5 0x8000000d Network(MC) 100022 ge-1/1/3 0x8000000f Network(MC) 100024 ge-1/1/3 0x80000010 Network(UC) 100017
Show the VXLAN MAC address information on Leaf1. Port ge-1/1/7 is single homed host connected to a VXLAN access port in VNID 10030 with MAC address 22:00:00:00:00:00.
admin@Leaf1# run show vxlan address-table VNID MAC address Type Interface VTEP ----------- ----------------- ------- ---------------- --------------- 99 18:5a:58:37:55:e1 Dynamic 10.226.14.253 10020 18:5a:58:37:55:e1 Dynamic 10.226.14.253 10030 18:5a:58:37:55:e1 Dynamic 10.226.14.253 10030 22:00:00:00:00:00 Dynamic ge-1/1/7 Entries in access port: 1 Entries in network port: 3
Show the BGP EVPN route information.
admin@Leaf1# run show bgp evpn route
BGP table version is 19, local router ID is 10.226.14.207
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.226.14.24:3
*>i[2]:[0]:[48]:[00:00:20:20:00:fe]:[32]:[192.168.0.1]
10.226.14.201 100 0 i
RT:65001:99 RT:65001:10020 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[2]:[0]:[48]:[00:00:20:20:00:fe]:[128]:[2002:0:0:1::1]
10.226.14.201 100 0 i
RT:65001:99 RT:65001:10020 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[2]:[0]:[48]:[00:00:20:20:00:fe]:[128]:[fe80::529a:4c20:1e6:7e71]
10.226.14.201 100 0 i
RT:65001:10020 ET:8
*>i[2]:[0]:[48]:[00:0c:29:77:8b:15]
10.226.14.201 100 0 i
RT:65001:10020 ET:8
*>i[2]:[0]:[48]:[00:0c:29:77:8b:15]:[128]:[fe80::20c:29ff:fe77:8b15]
10.226.14.201 100 0 i
RT:65001:10020 ET:8
Route Distinguisher: 10.226.14.24:4
*>i[2]:[0]:[48]:[00:00:30:30:00:fe]:[32]:[172.168.0.1]
10.226.14.201 100 0 i
RT:65001:99 RT:65001:10030 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[2]:[0]:[48]:[00:00:30:30:00:fe]:[128]:[2003:0:0:1::1]
10.226.14.201 100 0 i
RT:65001:99 RT:65001:10030 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[2]:[0]:[48]:[00:00:30:30:00:fe]:[128]:[fe80::529a:4c20:2e6:7e71]
10.226.14.201 100 0 i
RT:65001:10030 ET:8
*>i[2]:[0]:[48]:[00:0c:29:77:8b:15]
10.226.14.201 100 0 i
RT:65001:10030 ET:8
Route Distinguisher: 10.226.14.48:3
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:01]:[32]:[0.0.0.0]
10.226.14.202 100 0 i
RT:65001:10020 ET:8
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:02]:[32]:[0.0.0.0]
10.226.14.202 100 0 i
RT:65001:10020 ET:8
Route Distinguisher: 10.226.14.48:4
*>i[1]:[4294967295]:[03:00:22:22:22:22:22:00:00:01]:[32]:[0.0.0.0]
10.226.14.202 100 0 i
RT:65001:10020 RT:65001:10030 ET:8 ESI-label-Rt:AA
*>i[4]:[03:00:22:22:22:22:22:00:00:01]:[32]:[10.226.14.202]
10.226.14.202 100 0 i
ET:8 ES-Import-Rt:00:22:22:22:22:22 DF: (alg: 2, pref: 32767)
Route Distinguisher: 10.226.14.48:5
*>i[1]:[4294967295]:[03:00:22:22:22:22:22:00:00:02]:[32]:[0.0.0.0]
10.226.14.202 100 0 i
RT:65001:10020 RT:65001:10030 ET:8 ESI-label-Rt:AA
*>i[4]:[03:00:22:22:22:22:22:00:00:02]:[32]:[10.226.14.202]
10.226.14.202 100 0 i
ET:8 ES-Import-Rt:00:22:22:22:22:22 DF: (alg: 2, pref: 32767)
Route Distinguisher: 10.226.14.48:6
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:01]:[32]:[0.0.0.0]
10.226.14.202 100 0 i
RT:65001:10030 ET:8
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:02]:[32]:[0.0.0.0]
10.226.14.202 100 0 i
RT:65001:10030 ET:8
*>i[2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[32]:[172.168.0.42]
10.226.14.202 100 0 i
ESI:03:00:22:22:22:22:22:00:00:01 VNI: 0
RT:65001:99 RT:65001:10030 ET:8 Rmac:18:5a:58:37:64:61
*>i[2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[128]:[2003:0:0:1::42]
10.226.14.202 100 0 i
ESI:03:00:22:22:22:22:22:00:00:01 VNI: 0
RT:65001:99 RT:65001:10030 ET:8 Rmac:18:5a:58:37:64:61
*>i[2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[128]:[fe80::21e:c9ff:febb:c03c]
10.226.14.202 100 0 i
ESI:03:00:22:22:22:22:22:00:00:01 VNI: 0
RT:65001:10030 ET:8
Route Distinguisher: 10.226.14.207:3
*> [1]:[0]:[03:00:22:22:22:22:22:00:00:01]:[128]:[::]
10.226.14.254 32768 i
ET:8 RT:65001:10020
*> [1]:[0]:[03:00:22:22:22:22:22:00:00:02]:[128]:[::]
10.226.14.254 32768 i
ET:8 RT:65001:10020
*> [2]:[0]:[48]:[00:00:20:00:00:fe]
10.226.14.254 32768 i
ET:8 RT:65001:10020
*> [2]:[0]:[48]:[00:00:20:00:00:fe]:[32]:[192.168.0.2]
10.226.14.254 32768 i
ET:8 RT:65001:10020 RT:65001:99 Rmac:04:f8:f8:20:67:7b
*> [2]:[0]:[48]:[00:00:20:00:00:fe]:[128]:[2002:0:0:1::2]
10.226.14.254 32768 i
ET:8 RT:65001:10020 RT:65001:99 Rmac:04:f8:f8:20:67:7b
*> [2]:[0]:[48]:[00:00:20:00:00:fe]:[128]:[fe80::6f8:f810:120:677b]
10.226.14.254 32768 i
ET:8 RT:65001:10020
Route Distinguisher: 10.226.14.207:4
*> [1]:[4294967295]:[03:00:22:22:22:22:22:00:00:01]:[128]:[::]
10.226.14.254 32768 i
ET:8 ESI-label-Rt:AA RT:65001:10020 RT:65001:10030
*> [4]:[03:00:22:22:22:22:22:00:00:01]:[32]:[10.226.14.254]
10.226.14.254 32768 i
ET:8 ES-Import-Rt:00:22:22:22:22:22 DF: (alg: 2, pref: 1)
Route Distinguisher: 10.226.14.207:5
*> [1]:[4294967295]:[03:00:22:22:22:22:22:00:00:02]:[128]:[::]
10.226.14.254 32768 i
ET:8 ESI-label-Rt:AA RT:65001:10020 RT:65001:10030
*> [4]:[03:00:22:22:22:22:22:00:00:02]:[32]:[10.226.14.254]
10.226.14.254 32768 i
ET:8 ES-Import-Rt:00:22:22:22:22:22 DF: (alg: 2, pref: 1)
Route Distinguisher: 10.226.14.207:6
*> [1]:[0]:[03:00:22:22:22:22:22:00:00:01]:[128]:[::]
10.226.14.254 32768 i
ET:8 RT:65001:10030
*> [1]:[0]:[03:00:22:22:22:22:22:00:00:02]:[128]:[::]
10.226.14.254 32768 i
ET:8 RT:65001:10030
*> [2]:[0]:[48]:[00:00:30:00:00:fe]
10.226.14.254 32768 i
ET:8 RT:65001:10030
*> [2]:[0]:[48]:[00:00:30:00:00:fe]:[32]:[192.168.30.2]
10.226.14.254 32768 i
ET:8 RT:65001:10030 RT:65001:99 Rmac:04:f8:f8:20:67:7b
*> [2]:[0]:[48]:[00:00:30:00:00:fe]:[128]:[2003:0:0:1::2]
10.226.14.254 32768 i
ET:8 RT:65001:10030 RT:65001:99 Rmac:04:f8:f8:20:67:7b
*> [2]:[0]:[48]:[00:00:30:00:00:fe]:[128]:[fe80::6f8:f810:220:677b]
10.226.14.254 32768 i
ET:8 RT:65001:10030
*> [2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[32]:[192.168.30.42]
10.226.14.254 32768 i
ESI:03:00:22:22:22:22:22:00:00:01
ET:8 RT:65001:10030 RT:65001:99 Rmac:04:f8:f8:20:67:7b ND:Proxy
*> [2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[128]:[2003:0:0:1::42]
10.226.14.254 32768 i
ESI:03:00:22:22:22:22:22:00:00:01
ET:8 RT:65001:10030 RT:65001:99 Rmac:04:f8:f8:20:67:7b ND:Proxy
*> [2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[128]:[fe80::21e:c9ff:febb:c03c]
10.226.14.254 32768 i
ESI:03:00:22:22:22:22:22:00:00:01
ET:8 RT:65001:10030 ND:Proxy
Route Distinguisher: 10.226.14.208:3
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:01]:[32]:[0.0.0.0]
10.226.14.253 100 0 i
RT:65001:10020 ET:8
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:02]:[32]:[0.0.0.0]
10.226.14.253 100 0 i
RT:65001:10020 ET:8
Route Distinguisher: 10.226.14.208:4
*>i[1]:[4294967295]:[03:00:22:22:22:22:22:00:00:01]:[32]:[0.0.0.0]
10.226.14.253 100 0 i
RT:65001:10020 RT:65001:10030 ET:8 ESI-label-Rt:AA
*>i[4]:[03:00:22:22:22:22:22:00:00:01]:[32]:[10.226.14.253]
10.226.14.253 100 0 i
ET:8 ES-Import-Rt:00:22:22:22:22:22 DF: (alg: 2, pref: 32767)
Route Distinguisher: 10.226.14.208:5
*>i[1]:[4294967295]:[03:00:22:22:22:22:22:00:00:02]:[32]:[0.0.0.0]
10.226.14.253 100 0 i
RT:65001:10020 RT:65001:10030 ET:8 ESI-label-Rt:AA
*>i[4]:[03:00:22:22:22:22:22:00:00:02]:[32]:[10.226.14.253]
10.226.14.253 100 0 i
ET:8 ES-Import-Rt:00:22:22:22:22:22 DF: (alg: 2, pref: 32767)
Route Distinguisher: 10.226.14.208:6
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:01]:[32]:[0.0.0.0]
10.226.14.253 100 0 i
RT:65001:10030 ET:8
*>i[1]:[0]:[03:00:22:22:22:22:22:00:00:02]:[32]:[0.0.0.0]
10.226.14.253 100 0 i
RT:65001:10030 ET:8
*>i[2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[32]:[192.168.30.42]
10.226.14.253 100 0 i
ESI:03:00:22:22:22:22:22:00:00:01 VNI: 0
RT:65001:99 RT:65001:10030 ET:8 Rmac:18:5a:58:37:55:e1 ND:Proxy
*>i[2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[128]:[2003:0:0:1::42]
10.226.14.253 100 0 i
ESI:03:00:22:22:22:22:22:00:00:01 VNI: 0
RT:65001:99 RT:65001:10030 ET:8 Rmac:18:5a:58:37:55:e1 ND:Proxy
*>i[2]:[0]:[48]:[00:1e:c9:bb:c0:3c]:[128]:[fe80::21e:c9ff:febb:c03c]
10.226.14.253 100 0 i
ESI:03:00:22:22:22:22:22:00:00:01 VNI: 0
RT:65001:10030 ET:8 ND:Proxy
Route Distinguisher: 24.24.24.24:2
*>i[5]:[0]:[24]:[172.16.0.0]
10.226.14.201 0 100 0 ?
RT:65001:99 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[5]:[0]:[24]:[192.168.20.0]
10.226.14.201 0 100 0 ?
RT:65001:99 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[5]:[0]:[24]:[192.168.30.0]
10.226.14.201 0 100 0 ?
RT:65001:99 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[5]:[0]:[64]:[2002:0:0:1::]
10.226.14.201 0 100 0 ?
RT:65001:99 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[5]:[0]:[64]:[2003:0:0:1::]
10.226.14.201 0 100 0 ?
RT:65001:99 ET:8 Rmac:50:9a:4c:e6:7e:71
*>i[5]:[0]:[64]:[2006:0:0:1::]
10.226.14.201 0 100 0 ?
RT:65001:99 ET:8 Rmac:50:9a:4c:e6:7e:71
Displayed 56 prefixes (56 paths)
EVPN Multihoming with BUM Tunnel
For the topology and figure in the last section, to enable EVPN multihoming with BUM tunnel, besides above configurations of head end replication, we need to add the following configurations:
- Configure PIM on each device, and configure one of the devices as static RP (Spine2 in this example).
- Configure the same VNI to join the same multicast group.
Leaf1 Configuration
Step 1. Configure PIM and configure Spine2 as static RP.
admin@Leaf1# set protocols pim rp 202.202.202.202 group 224.1.1.1/32 admin@Leaf1# set protocols pim rp 202.202.202.202 group 230.1.1.1/32 admin@Leaf1# set protocols pim interface lo admin@Leaf1# set protocols pim interface vlan4014 admin@Leaf1# set protocols pim interface vlan4012
Step 2. Configure the same VNI to join the same multicast group.
NOTE:
PICOS supports only one VXLAN segment mapping to one IP multicast group, which is the way to provide the optimal multicast forwarding. That is, to have multiple VXLAN segments share a single IP multicast group in the core network is not supported.
admin@Leaf1# set vxlans vni 10020 mcast-group 224.1.1.1 admin@Leaf1# set vxlans vni 10030 mcast-group 230.1.1.1 admin@Leaf1# commit
Leaf2 Configuration
admin@Leaf2# set protocols pim rp 202.202.202.202 group 224.1.1.1/32 admin@Leaf2# set protocols pim rp 202.202.202.202 group 230.1.1.1/32 admin@Leaf2# set protocols pim interface lo admin@Leaf2# set protocols pim interface vlan4014 admin@Leaf2# set protocols pim interface vlan4013 admin@Leaf2# set vxlans vni 10020 mcast-group 224.1.1.1 admin@Leaf2# set vxlans vni 10030 mcast-group 230.1.1.1 admin@Leaf2# commit
Leaf3 Configuration
admin@Leaf3# set protocols pim rp 202.202.202.202 group 224.1.1.1/32 admin@Leaf3# set protocols pim rp 202.202.202.202 group 230.1.1.1/32 admin@Leaf3# set protocols pim interface lo admin@Leaf3# set protocols pim interface vlan4016 admin@Leaf3# set protocols pim interface vlan3012 admin@Leaf3# set vxlans vni 10020 mcast-group 224.1.1.1 admin@Leaf3# set vxlans vni 10030 mcast-group 230.1.1.1 admin@Leaf3# commit
Leaf4 Configuration
admin@Leaf4# set protocols pim rp 202.202.202.202 group 224.1.1.1/32 admin@Leaf4# set protocols pim rp 202.202.202.202 group 230.1.1.1/32 admin@Leaf4# set protocols pim interface lo admin@Leaf4# set protocols pim interface vlan3011 admin@Leaf4# set protocols pim interface vlan3012 admin@Leaf4# set vxlans vni 10020 mcast-group 224.1.1.1 admin@Leaf4# set vxlans vni 10030 mcast-group 230.1.1.1 admin@Leaf4# commit
Spine1 Configuration
admin@Spine1# set protocols pim rp 202.202.202.202 group 224.1.1.1/32 admin@Spine1# set protocols pim rp 202.202.202.202 group 230.1.1.1/32 admin@Spine1# set protocols pim interface lo admin@Spine1# set protocols pim interface vlan4011 admin@Spine1# set protocols pim interface vlan4092 admin@Spine1# set protocols pim interface vlan4014 admin@Spine1# set protocols pim interface vlan4013 admin@Spine1# set protocols pim interface vlan4015 admin@Spine1# set protocols pim interface vlan4016 admin@Spine1# commit
Spine2 Configuration
admin@Spine2# set protocols pim rp 202.202.202.202 group 224.1.1.1/32 admin@Spine2# set protocols pim rp 202.202.202.202 group 230.1.1.1/32 admin@Spine2# set protocols pim interface lo admin@Spine2# set protocols pim interface vlan4011 admin@Spine2# set protocols pim interface vlan4092 admin@Spine2# set protocols pim interface vlan4012 admin@Spine2# set protocols pim interface vlan4014 admin@Spine2# set protocols pim interface vlan4015 admin@Spine2# commit
Verify Configuration
In addition to the above show command to verify the results, you can also use the run show vxlan mcast-tunnel vni command to view the establishment of the BUM tunnel.
The following example shows the result on Leaf 3:
admin@Leaf3# run show vxlan mcast-tunnel vni 10020 VNI Src-addr Mcast-addr Out-Vlan Out-Interface Tunnel-ID Egress-ID -------- --------------- --------------- -------- --------------- ------------ ------------ 10020 10.226.14.201 224.1.1.1 3012 ge-1/1/5 0x4c000000 0 admin@Leaf3# run show vxlan mcast-tunnel vni 10030 VNI Src-addr Mcast-addr Out-Vlan Out-Interface Tunnel-ID Egress-ID -------- --------------- --------------- -------- --------------- ------------ ------------ 10030 10.226.14.201 230.1.1.1 3012 ge-1/1/5 0x4c000001 0