SD-WAN - Cisco Live

#CLUS

Khalid RazaDistinguished Engineer

BRKRST-2095

SD-WAN

Routing Design and Migrations

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS

• SD-WAN Architecture

• Enterprise Requirements

• Control and Data plane separation

• Flexible Control Plane

• OMP

• Existing Control plane architectures/design

• Concepts

• Routing and attributes

Agenda

BRKRST-2095 3


• Site migrations

• Data center migration

• Interoperability of underlay/overlay

• Routing migrations inside DC

• Convergence

• MPLS network

• Overlay convergence (MPLS, Internet)

• Data center convergence

• High Availability

Agenda

BRKRST-2095 4

Questions? Use Cisco Webex Teams to chat with the speaker after the session

Find this session in the Cisco Live Mobile App

Click “Join the Discussion”

Install Webex Teams or go directly to the team space

Enter messages/questions in the team space

How

Webex Teams will be moderated by the speaker until June 16, 2019.

1

2

3

4


Cisco Webex Teams

cs.co/ciscolivebot#BRKRST-2095

5

Enterprise requirements and High level Architecture


Requirements

• Security as scale

• Transport independent control/Data plane (MPLS, IPsec, LTE)

• Ubiquitous data plane across all transport

• Segmentation

• Convergence

• Service oriented network

• Cloud user experience

BRKRST-2095 7


• Control Plane• Control plane packet go to the router

• Builds and maintains the network topology

• Makes decisions on where the traffic will flow

• Policies are applied before sending the packet or frame

• Scale, convergence, flexibility, policy

• Data Plane• Data plane packets go through the router

• Data plane is about capacity not scale

• Forwards packets based on decision from the control plane

Requirements of Control Plane and Data Plane

BRKRST-2095 8


• Why separate control plane?

• Moving computationally intensive hardware from forwarding devices

• Single state-distribution algorithm for a network

• Programmer interacts with entire network instead of individual nodes

• If protocols are implemented properly, discreet configuration should be minimal

• Proper separation of control and data plane

• Insourced control plane, outsourced data plane

• Outsourced control plane and data plane

• Hybrid control plane (SP, Enterprise)

• Private control plane over public or private data plane

Flexibility with the architecture

BRKRST-2095 9

Flexible Control PlanePower Of Abstraction


Control Plane

Data Plane(Physical or Virtual)

Management Plane

Orchestration Plane

vManage

vSmart

vEdge

Viptela ArchitectureThe Power of Abstraction

vBond

ANALYTICS

Data Center Campus Branch Home Office

API

4GINTERNET MPLS

CONTROL

ORCHESTRATION

MANAGEMENT

BRKRST-2095 11


Orchestration Plane

Viptela ArchitectureControl Plane

ANALYTICS


API

4GINTERNET MPLS

CONTROL

ORCHESTRATION

MANAGEMENT

• Orchestrates connectivity

• First point of authentication

• Requires public IP Address

• Facilitates NAT traversal

• All other components need to know the vBond IP or DNS information

• Authorizes all control connections (white-list model)

• Distributes list of vSmarts to all vEdges

vBond

BRKRST-2095 12


Control Plane

Viptela ArchitectureControl Plane

ANALYTICS


API

4GINTERNET MPLS

CONTROL

ORCHESTRATION

MANAGEMENT

• Centralized brain of the solution

• Establishes OMP Peering with vEdges

• Acts like Route Reflector

• Enables central control and central data policy creation and distribution:

• TE• Service Chaining• Hub and spoke• Partial or full mesh

• Orchestrates secure data plane connectivity between the edges

vSmart

BRKRST-2095 13


Data Plane(Physical or Virtual)

vEdge

Viptela ArchitectureData Plane

ANALYTICS


API

4GINTERNET MPLS

CONTROL

ORCHESTRATION

MANAGEMENT

• WAN edge router of the site

• Leverages traditional routing protocols like OSPF, BGP

• Applies policies on data plane traffic

• Establishes control plane (OMP) peering with vSmart

• Provides secure data plane

• Either hardware devices or software VNF support

BRKRST-2095 14


MPLS2 InternetMPLS1

Flexibility of Control plane

BRKRST-2095 15


MPLS2 InternetMPLS1

Flexibility of Control plane Max Control (0,1,..)

BRKRST-2095 16


MPLS2 InternetMPLS1


vBond as stun server

BRKRST-2095 17


Internet

MPLS

Permanent connections

Temporary connections

ge0/1

interface ge0/1

ip address 10.0.26.11/24

tunnel-interface

encapsulation ipsec

vbond-as-stun-server

vpn 0

ge0/2

vpn 0

interface ge0/2

ip address 131.108.1.1

tunnel-interface

encapsulation ipsec

Upload the serial file manually


BRKRST-2095 18


MPLS2 InternetMPLS1

Flexibility of Control plane Color

Restrict

BRKRST-2095 19

Overlay Management Protocol


• Existing Control plane protocols

• OSPF

• ISIS

• BGP

• EIGRP

• RIP

• Spanning Tree (Layer 2)

• IPSec (IKE)

• Control plane requirements

• Loop detection

• Scale (Core, Edge)

• Convergence

Existing Control plane architecture

} Link State

Path Vector

} Distance Vector

Key distribution

• Challenges

• Authenticated through peer configuration

• Adjacencies are dependent peers IP address

• No Encryption

• Loop avoidance

• Within a Protocol

• Between Protocols always an issue

BRKRST-2095 21


• Authentication through cryptographic endpoints

• Encrypted control plane peering

• Scalable key distribution for data plane peering

• Scalable overlay routing

• Availability

• Convergence


BRKRST-2095 22



• Creates two network domains that are syntactically identical

• Interoperates with the existing control plane protocols

• Service side routing (IPv4, IPv6)

• Service side are used to uniquely identify end nodes (User, devices)

• Service nodes are independent of topology, location and are typically routed with in the enterprise

• TLOCs (IPv4,IPv6)

• Assigned topologically to network/transport by third party

• Typically tied to the routed inter-domain/carrier network

• Interacts with underlay network

BRKRST-2095 23


Multi-Domain Routing Fabric Transport Locators Advertisement

vEdge

vEdgevEdge

vEdge

vEdge

vSmart

TLOCs advertised to vSmartsWith set of attributes

vSmarts advertise TLOCs to all vEdges

FabricTOP of underlay

Service prefixes advertised to vSmartswith set of attributes

S1 S2

S3 S4

Service Prefix=S1…Sx

BRKRST-2095 24


Overlay Routing: OMP Routes

INETMPLS

vSmart

• Routes learnt from local service side

• Advertised to vSmart controllers

• Most prominent attributes:- TLOC - Site-ID- Label- VPN-ID- Tag- Preference- Originator System IP- Origin Protocol- Origin Metric

Connected

Static

Dynamic (OSPF/BGP)

vEdge

OMP Update

Service Side

BRKRST-2095 25


Overlay Routing: TLOC Routes

INETMPLS

vSmart

• Routes connecting locations to physical networks


• Most prominent attributes:- Site-ID- Encap-SPI- Encap-Authentication- Encap-Encryption- Public IP- Public Port- Private IP- Private Port- BFD-Status- Tag- Preference- Weight

Connected

Static

Dynamic (OSPF/BGP)

vEdgeTLOCs

OMP Update

BRKRST-2095 26


Overlay Routing: Network Service Routes

INETMPLS

vSmart • Routes for advertised network services, i.e. Firewall, IDS, IPS, generic


• Attributes:- VPN-ID- Service-ID- Label- Originator System IP- TLOC

vEdge

Firewall

OMP Update

Network Service

BRKRST-2095 27

MigrationsApplying the architectural principles


Requirements• Network

• BW Augmentation (Existing MPLS, Adding Broadband)

• Full mesh Site to Site connectivity for large file transfers

• Secure ubiquitous data plane

• VPNs need to move to the Overlay (segmentation)

• LOB based segmentation

• Active/Active

• Application-centric

• Intelligent application Steering

• Interactive SLA-monitoring/influence

• Expectations of detailed statistics and monitoring capabilities

• Best performing SaaS locations from different carriers to SaaS providers

• IaaS part of WAN fabric

BRKRST-2095 29


Network Layout

• Two Data centers

• OSPF and BGP within each site

• BGP between sites

• BGP with MPLS providers

• Dual MPLS carrier

• Three categories of branch sites

• Type 1. Small size branch (VRRP)

• Type 2. Medium size branch(OSPF)

• Type 3. Large size branch(BGP-OSPF)

BRKRST-2095 30


Migration Requirements

• Control plane

• Maintain existing MPLS network

• In line migration

• Existing circuit migration

• Non-migrated sites use existing MPLS network

• Migrated sites use overlay

• New circuit (Broad band) runs SD-WAN

• Data plane models

• Hub and spoke during migration(Not acceptable to the customer)

• Full mesh during migration (Same as MPLS)

BRKRST-2095 31


Network Migration layout

Transport VPNVPN0

Service VPNVPNx

BRKRST-2095 32


Migration

• Site Type1

• Make vEdge as the VRRP master

• vEdge points its default route to existing MPLS router

• ‘In case of vEdge failure existing router becomes the default gateway

• Existing MPLS router advertises the existing network into MPLS network

• vEdge router also advertises the connected subnet into overlay

• All not migrated sites connected via MPLS

• Migrated sites connect via overlay

BRKRST-2095 33


Migration Method

• Site Type2

• Enable OSPF on vEdge router

• Connect vEdge with MPLS routers in VPN0

• Connect vEdge with MPLS routers in VPN1

• Receive default route 0/0 via OMP

• Generate default route from vEdge with a better metric in OSPF

• Change admin distance of BGP to be higher then OMP

• Learn ALL underlay routes via BGP

• No BGP redistribution into OMP

BRKRST-2095 34


Migration Method

• Site Type2• vEdge router’s OSPF default route depends on OMP

default

• Run BGP between the vEdges and MPLS routers

• Advertise only the local site connected and learned routes into OMP

• By default we advertise OSPF (intra and inter area) routes into OMP

• Existing CE still advertising the local routes into MPLS network

• ALL traffic comes to vEdge

• Traffic destined to SD-WAN Sites is sent on the overlay

• Traffic destined to non-migrated site will be sent to local CPE and sent natively on MPLS

BRKRST-2095 35


Migration Method

• Data Center

• Connect vEdge to the existing OSPF and BGP routers

• Peer vEdge with existing MPLS CPE routers

• Advertise all migrated site routes via OMP

• vEdges in data center redistributes OMP into BGP with higher preference

• Migrated sites will take the vEdgerouter non migrated will take MPLS

• There is no overlay connection between the vEdges in data centers

BRKRST-2095 36


Overlay routing walkthrough

• Site 2 to DC overlay routing prefix b

• Site 2 source of the route in vE3 is OSPF

• vE3 advertises the route to vE5 and vE7 via OMP

• vE3 advertises the route to vE4 and vE6 via OMP

• vE3 preserves the origin of the routing protocol as OSPF

• vE5 and vE7 redistributes the route into BGP

• vE4 and vE6 redistributes the route into BGP

• All DC non viptela routers see prefix b as BGP route

• vE4,vE5,vE6,vE7 can also receive route to b via OMP and BGP

BRKRST-2095 37


Overlay-Underlay walkthrough

• Site 2 to DC overlay routing prefix b

• Simple overlay topology

• Looped overlay and underlay topology

• vEdges will receive all migrated site routes via OMP

• All non migrated site routes will be received via BGP

• vEdge and MPLS peering routers will run BGP between them

• All overlay redistributed routes have to be filtered between MPLS and routers and vEdges

BRKRST-2095 38


Overlay-Underlay routing loop

• Site 2 Migration with routing loop

• Change the admin distance of OMP on vEdge to be lower then BGP

• Route b is in vE3 routing table via OSPF

• Route b is in vE4,vE6 and vE5,vE7 via OMP

• vE4 and vE6 advertise route b to C9,C10 and C11 via iBGP

• vE5 and vE7 advertises route b to C7,C6,and C8 via iBGP

• Same route is advertised by multiple routers in both Data centers

• Inside DC1 prefix b is advertised by vE4 and vE6 to C10 and C11

• C10 and C11 advertise prefix b back into MPLS network

• We can have a routing loop/inefficient routing

BRKRST-2095 39


Overlay-Underlay routing loop

• Site 2 Migration with routing loop

• Filter site 2 routes being advertised to MPLS peering routers

• Ideally people use BGP communities to control the redistribution loops

• BGP communities per sites are not easy to provision and maintain

• If communities already exist prior to SD-WAN you will have to setup new community method

• Rest of your BGP network will have to pass the communities as well

BRKRST-2095 40


Migration Methods Overlay-Underlay

• Smart protocols are always better

• Only three command can take care of all the loops for iBGP

VPN 1

bgp 65003

propagate-aspath

OMP

overlay-as 65005

Configure the following AS path filter inbound on MPLS peering router

_65005$

BRKRST-2095 41


Migration Methods Overlay-Underlay

• Smart protocols are always better

• Only two command can take care of all the loops for eBGP

VPN 1

bgp 65005

propagate-aspath

OMP

overlay-as 65005

Configure the following AS path filter inbound on MPLS peering router

_65005$

BRKRST-2095 42


Migrated Overlay

• Data Center

• Remove the _65005_ AS path filter

• All circuits are visible to all vEdges

• Remote site

• Tloc Extension virtually extends non connected circuits to all devices

• Provides application SLA on all available paths

• Helps with faster convergence

BRKRST-2095 43


Migrated sites

• Tloc Extension

• vEdge3 is connected physically to Internet and virtually through Tloc extension to MPLS via vEdge31

• vEdge31 is connected physically to MPLS and virtually through Tloc extension to Internet via vEdge3

• Any failure of the Tloc link rerouting is a local matter

BRKRST-2095 44


Migrated Overlay

• Very seamless to add new transport without any new requirement

• Best cloud experience cloud be enabled based on SaaS and IaaS

• All transport application and connectivity polices remain the same just additional transport

• If there is an IP connection across transport you can connect a internet only site to MPLS only site

• Ubiquitous data plane

BRKRST-2095 45

Network Convergence


Network Convergence

• Steps:• Detect event has occurred

• Propagate the event

• Process the event

• Update related forwarding structures

BRKRST-2095 47


Network Convergence

• Transport Convergence

• MPLS transport

• Internet Transport

• Single carrier

• Multiple carriers

• Service convergence

• Data Center convergence

• Site convergence

BRKRST-2095 48


Network Convergence

• MPLS Transport

• Most MPLS providers use Cisco fast convergence to avoid BGP event

• BGP and IGP Convergence tuning have a different focus

• IGP Convergence• Rebuild network topology as quickly as possible

• Link states are more efficient due to topology data base

• BGP Convergence

• Transfer large amounts of prefix information very quickly

• Seconds to Minutes so avoid as much as possible by using RD, Add path at RR

BRKRST-2095 49


• MPLS Transport

• IGP Convergence - Rebuild the topology quickly following an event

• IGP - Sub-Second

• Fast IGP Convergence plays a role in maintaining availability for BGP prefixes

• Often topological changes can result in no BGP changes

• IGP updates the next-hop information for BGP prefixes

Network Convergence

BRKRST-2095 50


Network Convergence

• MPLS transport

• Detection

• BFD is used between the vEdges

• Default timers are 1 hello 7 dead

• Most MPLS core networks convergence within sub-second

• Only concern is the PE-CE link

• Arp is send every second on Transport interface

• Link failure triggers TLOC update immediately

BRKRST-2095 51


Network convergence

• Internet Transport single carrier

• Understand your carriers network clearly

• Carriers run CGN in their network

• CGN at the POP

• CGN at the Internet exit

BRKRST-2095 52


Private Color to Private Color

Private Color to Public Color

Public Color to Public Color

1

2

Private IP/PortPublic IP/Port

IPsec Tunnel / BFD Session

3



Network Convergence

TLOCs, Colors, Site-IDs and Carriers

BRKRST-2095 53


• Carrier setting is final influencer to decide on Private/Public IP/Port

• Use if two endpoints are using private colors and you need session between them to be established between their Public IP/Port

Network Convergence


vpn 0

interface ge0/0

tunnel-interface

carrier carrier4

color Private2

vpn 0

interface ge0/0

tunnel-interface

carrier carrier2

color Private1

TLOCs, Colors, Site-IDs and Carriers

BRKRST-2095 54


• Single Carrier convergence• If CGN at every pop then long pole is the

NAT device failure

• Discovery of NAT device failure is considered a brownout

• End to end failure will depend on BFD

• Default timers convergence will be 7 sec

Network convergence

BRKRST-2095 55


Network convergence

• Single Carrier• If CGN not at the POP router

• Any failure within the same carrier will depend on the SP network convergence

• Any core link failure within the SP network should sub-second

• Again local link failure is detected by ARP send every second

• Lack of ARP response will trigger TLOC update

• Within the same carrier with private color and carrier command failure detection is not dependent on NAT

BRKRST-2095 56


• Dual Carrier

• Depend on NAT devices

• Depends on BGP design of the second carrier

• Two long poles

• NAT device failure

• BGP convergence of the second carrier

Network convergence

BRKRST-2095 57


Network convergence

• Service side routing

• Three protocols convergence

• OMP carries alternate paths to destinations

• Primary path is installed in the routing table

• Tloc for every site is installed in the table even if the service route is not in the routing table

• If Connectivity to the primary path is lost its not a convergence event

• Back path is installed immediately as it’s a local decision on the vEdge

• Advertise all/alternate the paths between the EBGP peers on service side

• Use MED to influence return routing

BRKRST-2095 58


Network convergence • Service side routing

• Remote site routers use Tloc extension

• All path are available to all vEdges

• Any link failure is quickly detected and alternate path is immediately used

• Enable OSPF on transport routers with passive interface towards the carrier

• OSPF helps in convergence of Tloc extension on non connected transport

• Point your static 0.0.0.0 towards the OSPF learned Tloc route of non connected interface

• Change the admin distance of static or ospf default

• Non connected Tloc failure removes the default route for Tloc extension

BRKRST-2095 59

High Availability


Horizontal Solution Scale


4G/LTE

MPLS

Internet

Control Plane (Containers or VMs)

(vSmart)

Management Plane(Multi-tenant or Dedicated)

(vManage)

Orchestration Plane(vBond)

Horizontal Scale Out Model

Add vSmart Controllers for more control plane capacity

Create vManage cluster to accommodate more vEdge routers

Add vBond Orchestrators to increase vEdge bringup capacity

• Choose vEdge platform with appropriate IPSec tunnel scale

• Use control policies to define VPN topologies

BRKRST-2095 61


vSmartControllers

Data Plane

Control Plane

• vSmart controllers exchange OMP messages and they have identical view of the SD-WAN fabric

• vEdge routers connect to upto three vSmart controllers for redundancy

• No impact as long as vEdge routers can connect to at least one vSmart Controller

• If all vSmart controllers fail or become unreachable, vEdge routers will continue operating on a last known good state for a configurable amount of time

- No changes allowed

4GMPLS

INET

Branch

Campus

CloudData Center

Small OfficeHome Office

Data Center

Redundancy – vSmart Control Controllers

BRKRST-2095 62



• Affinity On Control Plane allow us to split this single Global Pool of controllers into Sub-Groups across which we could scale out the control connections.

vSmart Controllers

Software on x86

vEdge Routers

Single Pool of

Vsmarts

vBond (SW)

Vmanage

vSmart(config)# system controller-group-id 1

vEdge-1(config)# system controller-group-list 1

system max-omp-sessions 2

vpn 0 interface ge0/2 tunnel-interface

vEdge-1(config-tunnel-interface)# max-control-connections 1


vEdge-1(config-tunnel-interface)# max-control-connections 1

system max-omp-sessions 2

BRKRST-2095 63


Vmanage

DC1-USA DC2-Germay


vSmart-DC1(config)# system controller-group-id 1

vSmart-DC2(config)# system controller-group-id 2

vEdge-West(config)# system controller-group-list 1 2

vEdge-East(config)# system controller-group-list 2 1

vEdge-East(config-system)# max-omp-sessions 2


vEdge-East(config-tunnel-interface)# max-control-connections 1

vEdge-East(config-tunnel-interface)# exclude-controller-group-list 1

BRKRST-2095 64


vManageCluster

• vManage servers form a cluster for redundancy and high availability

• All servers in the cluster act as active/active nodes

- All members of the cluster must be in the same DC / metro area

• For geo-redundancy, vManage servers operate in active/standby mode

- Not clustered

- Database replication between sites

• Loss of all vManage servers has no impact on fabric operation

- No administrative changes

- No statistics collection

Data Plane

Management Plane

4GMPLS

INET

Branch

Campus

CloudData Center

Small OfficeHome Office

Data Center

Redundancy – vManage System

BRKRST-2095 65


INET

MPLS

Site

DataCenter

Network/Headend Redundancy

MPLS

INET

vSmart Controllers

Control

Data

Control Redundancy

INET INETMPLSMPLS

Transport Redundancy

High Availability and Redundancy Overview

VRRP OSPF/BGP

OSPF/BGP

Site Redundancy

BRKRST-2095 66


Conclusion • SDN does not eliminate the challenges of existing network

• Scale

• Convergence

• Availability

• Make the technology decision on architecture not on forwarding features or nice GUI

• Remember Spanning Tree

BRKRST-2095 67

Complete your online session evaluation

• Please complete your session survey after each session. Your feedback is very important.

• Complete a minimum of 4 session surveys and the Overall Conference survey (starting on Thursday) to receive your Cisco Live water bottle.

• All surveys can be taken in the Cisco Live Mobile App or by logging in to the Session Catalog on ciscolive.cisco.com/us.

Cisco Live sessions will be available for viewing on demand after the event at ciscolive.cisco.com.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS BRKRST-2095 68

http://ciscolive.cisco.com/us

https://ciscolive.cisco.com/


Continue your education

Related sessions

Walk-in labsDemos in the Cisco campus

Meet the engineer 1:1 meetings

BRKRST-2095 69

Thank you

#CLUS

SD-WAN - Cisco Live

Documents

Transcript of SD-WAN - Cisco Live