Web-Scale Data Centre Networking with VXLAN/EVPN and Full Automation

At a Glance

Detail	Value
Industry	ISP
Environment	Greenfield data centre, 3 pods x 64 rack units
Timeline	8 weeks design, 6 weeks deployment
Key Results	95% reduction in manual network changes, near-zero-touch operations

The Challenge

A rapidly growing ISP was moving into a new data centre facility. Their existing network, a classic spanning-tree based design, was already showing signs of strain under the load of increasing east-west traffic between distributed application tiers. Manual configuration changes were error-prone. Even simple VLAN additions required weekend maintenance windows.

The leadership team wanted a clean break. A network that could scale horizontally, support multi-tenancy out of the box, and be fully programmable via infrastructure-as-code.

The Solution: VXLAN/EVPN Fabric

We designed and deployed a VXLAN/EVPN-based IP fabric using leaf-spine architecture. Unlike traditional spanning-tree designs, VXLAN/EVPN gives you:

Fabric-Level Multi-Tenancy: Each customer environment got an isolated VRF with its own IP addressing and routing policies, all running over a shared physical underlay.

Active-Active Anycast Gateway: Every leaf switch served as a default gateway for connected hosts. No need for HSRP or VRRP, and workloads could move seamlessly across racks.

Automation Architecture

The automation stack used Ansible as the single control plane, with Cumulus Linux native tooling handling device-level configuration:

Layer 1: Inventory and Topology Definition

A YAML inventory defined the entire fabric topology. Spine and leaf roles, BGP ASN assignments using unnumbered BGP (RFC 5549), rack locations, and tenant VRFs. Each switch’s role determined which Jinja2 templates would render its configuration:

leaf_spine_fabric:
  spines: [spine01, spine02]
  leaves:
    - name: leaf101
      role: leaf
      asn: 65010
      rack: R01
      tenants: [prod, staging]
      mlag_peer: leaf102
    - name: leaf102
      role: leaf
      asn: 65020
      rack: R01
      tenants: [prod, dev]
      mlag_peer: leaf101

Layer 2: Configuration Management with Ansible and Cumulus Linux

Ansible applied configurations using the nclu module, which issued NCLU (Network Command Line Utility) commands idempotently against the Cumulus Linux switches. For the underlay, Jinja2 templates rendered /etc/network/interfaces.d/ with ifupdown2 syntax. For the overlay, FRR (Free Range Routing) configuration templates rendered BGP EVPN address-family sections, enabling type-2 (MAC/IP advertisement) and type-3 (IMET) route handling across the fabric.

Layer 3: Zero-Touch Provisioning and Validation

Switches shipped with Cumulus Linux and were configured via ZTP: DHCP options pointed each switch to a provisioning server that served a bootstrap script and the initial Ansible hostname. Once online, automated validation ran net show commands (net show bgp summary, net show evpn vni, net show interface status) to confirm fabric convergence. Results fed into the monitoring pipeline for ongoing compliance.

The Results

• 95% reduction in manual network configuration changes
• 3x increase in deployment velocity for new tenant environments
• Zero network-related outages in the first 12 months of operation
• $240k/year savings in operational overhead
• 99.99% fabric uptime since cutover

Key Takeaways

1. VXLAN/EVPN is the de facto standard for modern data centre fabrics. It’s mature, well-supported, and it eliminates the limitations of spanning-tree.
2. Automate from day one. Manual processes built during deployment become hard to remove later. We treated the automation as the primary deliverable, with the switches as ephemeral endpoints.
3. Test in production, safely. BGP route propagation and EVPN type-2 and type-3 route handling were validated using traffic generators in a pre-production pod before touching production.
4. Document the intent, not the configuration. With IaC, the YAML inventory and playbooks are the documentation. We maintained no separate network diagrams. The live source of truth was in Git.