Multicloud Networking - Cisco Live

#CLMEL

#CLMEL

Shannon McFarland – CCIE#5245

Distinguished EngineerCloud CTO@eyepv6

BRKCLD-3440

Multicloud Networking – Design and Deployment

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Webex Teams

Questions? Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session

Open the Cisco Events Mobile App

Find your desired session in the “Session Scheduler”

Click “Join the Discussion”

Install Webex Teams or go directly to the team space

Enter messages/questions in the team space

How

1

2

3

4

5

3

cs.co/ciscolivebot#BRKCLD-3440

Agenda

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLMEL

• Multicloud Networking Overview

• Native IPsec VPN Services

• Multicloud with Cisco SD-WAN

• DMVPN

• Automation

• Conclusion

BRKCLD-3440 4

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLMEL 5BRKCLD-3440

Disclaimer• You won’t learn security, routing, HA, performance best practices

• There are a gazillion ways to accomplish the same thing for ALL of this

• Be smart – Know IPsec/IKE, BGP, OSPF, EIGRP and FHRP stuff

• Dead Peer Detection

• IPsec SA lifetimes

• IPsec SA replay window-size

• Perfect Forward Secrecy (PFS)

• BGP timers, Local Preference, MED, inbound soft reset (check if cloud provider supports dynamic inbound soft reset)

• BGP graceful restart - Note: Each cloud provider uses BGP graceful restart with default timers (120 sec) – My configs do not show that due to slide space but know that it is enabled on each On Premises router

• IGP timers, configuration best practices

• HSRP timers, tracking

router bgp 65002

bgp log-neighbor-changes

bgp graceful-restart restart-time 120

bgp graceful-restart stalepath-time 360

bgp graceful-restart

Multicloud Networking Overview


Hybrid vs Multicloud Networking

• Hybrid Cloud Networking = Network transport from on-premises to a single public cloud provider

• Multicloud Networking = Network transport from on-premises to multiple public cloud providers and/or between multiple public cloud providers

• The technologies used can be identical for every connection or they can be per-provider, per-region, per-project, etc..

• Common network transport ingredients for hybrid and multicloud:

• Encryption (IPsec/IKEv2/IKEv2, SSL, PKI)

• Routing (Static, BGP and with supported public cloud-hosted routers: OSPF, EIGRP)

• Tunnelling (IPsec tunnel mode, GRE, mGRE, MPLS, segment routing, etc..)

• Common network endpoint options:

• Native VPN (IPsec over Internet) using public cloud provider services that connect to on-premises router/firewall

• Commercial/Open Source VPN platform hosted on the public cloud provider connecting to an on-premises router/firewall

• Colocation/Direct Peering: Service from public cloud provider to on-premises via a 3rd party colo facility

• Google Cloud Platform Dedicated Internconnect/Direct Peering/Carrier Peering: https://cloud.google.com/interconnect/

• Amazon Web Services Direct Connect/PrivateLink: https://aws.amazon.com/directconnect/

• Microsoft Azure ExpressRoute: https://azure.microsoft.com/en-us/services/expressroute/

https://cloud.google.com/interconnect/

https://aws.amazon.com/directconnect/

https://azure.microsoft.com/en-us/services/expressroute/


Why Would You Use Multiple Cloud Providers?

• Cloud provider high availability

• M&A may dictate public cloud provider preference (for a time)

• Regional cloud provider access

• Feature disparity between providers, regions and/or services

• Per-project service requirements

Extending On Premises Private Cloud to a Public Cloud


• Enterprise users/applications connect to Cloud Service Provider (CSP) public endpoints and/or public IPs of applications

• No ‘traditional’ IPsec VPN

• TLS/SSL capable

• Can be at odds with Enterprise InfoSec policies

Internet Over-the-Top (OTT)

10BRKCLD-3440

Internet Gateway

Enterprise Edge

AmazonECR

Enterprise Application

VPCRouterpod

AZ: us-west-2b

Internet

Data Centre

Region us-west-2

Private Subnet 2

Public Subnet 2

NAT GW 2 172.16.1.0/24

172.16.4.0/24

EIP

AmazonS3

Campus

Enterprise Site

CSP-published service

endpoints


Cloud Service Provider - Native IPsec VPN Service

11BRKCLD-3440

Default Network

10.138.0.0/20

IPsec/IKEv2

Google Cloud VPN

Google Cloud Router

BGP

Private Network

Cisco ASR, CSR, ISR


IPsec VPN - Cisco SD-WAN Example

Private Network(s)VPC Subnet(s)

VPCRouter

vEdgeCloud IPsec

vEdge

Per-VPC Cisco vEdge

On-Premises

Transit VPC: Cisco vEdge + Per-VPC vEdge


VPCRouter

vEdgeCloud

Transit VPC

IPsecvEdgevEdge

CloudOn-Premises


VPCRouter

VPNGateway

vEdgeCloud

Transit VPC

IPsecvEdge

Transit VPC: Cisco vEdge + CSP VPN

On-PremisesIPsec


IPsec VPN - Cisco CSR 1000v ExamplePer-VPC Cisco CSR 1000v

Private Network(s)

On-Premises

VPC Subnet(s)

VPCRouter

CSRs

DMVPN/IPsecCisco

ASR/CSR/ISR

Transit VPC: Cisco CSR + Per-VPC CSR


VPCRouter

CSRs

Transit VPC

DMVPN/IPsec On-PremisesCisco

ASR/CSR/ISRCSRs

Transit VPC: Cisco CSR + CSP VPN


VPCRouter

VPNGateway

CSRs

Transit VPC

DMVPN/IPsec On-PremisesCisco

ASR/CSR/ISR

IPsec


Multicloud with Transit VPC

14BRKCLD-3440

VNet Subnet

SD-WAN

VPC Subnet

AWS VPN GW

VPC Subnet

Google Cloud VPN

On-Premises

Cisco vEdge

vEdge

Transit VPC

vEdge

Transit VPC

vEdge

Transit VPC

Azure VPN GW

Private Network(s)


VPN

WAN

AWS Direct Connect

Transit Gateway

Dev Prod Dev Prod Dev Prod

Dev Prod Dev Prod Dev Prod

15BRKCLD-3440

AWS – Transit Gateway (TGW)

This replaces this

Transit VPC

Private Network(s)

On-Premises

VPC Subnet(s)

VPCRouter

VPNGateway

CSR

Transit VPC

IPsecCisco ASR/

CSR/ISR


Colocation - With or Without VPN

Cisco SD-WAN + Some Combo of Colocation/peering

Private Network(s)

On-Premises

VPC Subnet(s)

VPCRouter

VPNGateway

vEdgevEdgeDX

Endpoint

VLANs

IPsecIPsec


VPCRouter

VPNGateway

Cisco ASR 1000

DX Endpoint

VLANs

Cisco ASR/CSR/ASA

IPsec

On-Premises

Cisco Routers or Firewalls + Some Combo of Colocation/peering


VPN over the Internet vs Direct Connect/ExpressRoute/Dedicated Interconnect

VPN over the Internet Direct/Express/Dedicated

Throughput Winner

QoS Winner

Latency Winner

Inline Services Winner

Managed Services Winner

Cost Winner

Time to Provision Winner

Flexibility Winner

Location Availability Winner


Server

Hypervisor

Virtual Switch

OS

App

OS

App

CSR 1000VSoftware

• Familiar IOS XE software with ASR1000 and ISR4000

Infrastructure Agnostic

• Runs on x86 platforms

• Supported Hypervisors: VMware ESXi, Linux KVM, Citrix Xen, Microsoft Hyper-V, Cisco NFVIS and CSP2100

• Supported Cloud Platforms: Amazon AWS, Microsoft Azure, and Google Cloud Platform

Performance Elasticity

• Available licenses range from 10 Mbps to 10 Gbps

• CPU footprint ranges from 1vCPU to 8vCPU

License Options

• Term based 1 year, 3 year or 5 year

• Smart License enabled

Programmability

• NetConf/Yang, RESTConf, Guest Shell and SSH/Telnet

Cisco Cloud Services Router (CSR) 1000VCisco IOS XE Software in a Virtual Appliance Form-Factor

18BRKCLD-3440

https://www.youtube.com/playlist?list=PLCi

TBLSYkcoTUS6b4MFthdvhDrseo6MeN

https://www.youtube.com/playlist?list=PLCiTBLSYkcoTUS6b4MFthdvhDrseo6MeN


Public Cloud Provider Native VPN Services

• Google Cloud Platform (GCP):

• VPN: https://cloud.google.com/compute/docs/vpn/overview

• Dedicated Interconnect: https://cloud.google.com/interconnect/

• Amazon Web Services (AWS):

• VPN: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpn-connections.html

• Direct Connect: https://aws.amazon.com/directconnect/

• Microsoft Azure:

• VPN: https://docs.microsoft.com/en-us/azure/vpn-gateway/

• ExpressRoute: https://azure.microsoft.com/en-us/services/expressroute/

• OpenStack public cloud goodness: https://www.openstack.org/passport

The Big Three

19BRKCLD-3440

Reference

https://cloud.google.com/compute/docs/vpn/overview

https://cloud.google.com/interconnect/

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpn-connections.html

https://aws.amazon.com/directconnect/

https://docs.microsoft.com/en-us/azure/vpn-gateway/

https://azure.microsoft.com/en-us/services/expressroute/

https://www.openstack.org/passport

Let’s Backup


Starting SimplePublic Cloud Provider Native IPsec VPN Service

21BRKCLD-3440

Private Network

172.16.0.0/24

VPC Network

10.138.0.0/20

IPsec/IKEv2

Tunnel Mode

BGP/OSPF/EIGRP

eBGP<>IGP Redistribution

On-Premises

Google Cloud VPN

Google Cloud Router

BGP AS65000

BGP AS65003


Add More On-Premises StuffPublic Cloud Provider Native IPsec VPN Service

22BRKCLD-3440

VPC Network

10.138.0.0/20

BGP AS65000 BGP AS65003

Routes this side should see:10.138.0.0/20

Private Network

172.16.0.0/24

Private Network

192.168.100.0/24BGP AS65002

On-Premises Tenant 1

On-Premises Tenant 2Routes this side should see:172.16.0.0/24

192.168.100.0/24

Google Cloud VPN

Google Cloud Router

BGP/OSPF/EIGRP

BGP/OSPF/EIGRP

CSR1000v

CSR1000v


On-Premises Physical/Virtual Public Cloud Provider Native IPsec VPN Service

23BRKCLD-3440

VPC Network

10.138.0.0/20

Private Network

172.16.yyy.0/24

Private Network

192.168.yyy.0/24

Physical Router

Physical Firewall

Google Cloud VPN

Google Cloud Router

ASR 1000

ASA Firewall

Add More Public Cloud Providers to the Mix


Stepping into Multicloud NetworkingMultiple Native IPsec VPN Services

25BRKCLD-3440

Private Network

172.16.0.0/24

VPC Network

10.138.0.0/20

BGP/OSPF/EIGRP

On Premises Private Cloud

Google Cloud VPN

Google Cloud Router

VPC Network

172.31.0.0/16VPN

GatewayVPC

Router


Stepping into Multicloud NetworkingMultiple Native IPsec VPN Services

26BRKCLD-3440

Private Network

172.16.0.0/24

VPC Network

10.138.0.0/20

BGP/OSPF/EIGRP


Google Cloud VPN

Google Cloud Router

VPC Network

172.31.0.0/16VPN

GatewayVPC

Router

As the number of these connections increase and/or change frequently... You can see where this is going


Moving Away From Native VPN Services

• If On Premises routers/firewalls are behind NAT – Check for provider support of NAT-T

• You need to extend your On Premises IGP (OSPF/EIGRP) into the public cloud

• Operational consistency

• You need different IPsec/IKE configurations than what the provider offers

• You need SSL-based VPNs

• You need MPLS VPN

• QoS, specific network monitoring (IP SLA, NetFlow), Enterprise toolsets for configuration and monitoring

What Conditions Cause a Change in Design?

27BRKCLD-3440

Options


Cisco SD-WAN

29BRKCLD-3440

Private Network

172.16.0.0/24

VNet Network

10.50.0.0/16

SD-WANOn Premises Private Cloud

VPC Network

172.31.0.0/16

vEdge/cEdge

vEdge/cEdge

vEdge/cEdge

vManage vBond vSmart

Cisco SD-WAN:

https://www.cisco.com/c/en/us/solutions/en

terprise-networks/sd-wan/index.html

https://www.cisco.com/c/en/us/solutions/enterprise-networks/sd-wan/index.html


DMVPN – Enable Dynamic Multicloud NetworkingCisco DMVPN - A Brownfield Way to Bolt on Multicloud

30BRKCLD-3440

Private Network

172.16.0.0/24

VNet Network

10.50.0.0/16

DMVPN

FHRP

On Premises Private CloudVPC Network

172.31.0.0/16Cisco

CSR1000v

Cisco CSR1000v

Cisco CSR1000v

IGP Support: OSPF, EIGRP, iBGPQoS Policies

IP SLA, NetFlowNAT-T (Transparency)

MPLSetc...

Hub

Spoke

Spoke

Cisco DMVPN:

https://www.cisco.com/c/en/us/products

/security/dynamic-multipoint-vpn-

dmvpn/index.html

https://www.cisco.com/c/en/us/products/security/dynamic-multipoint-vpn-dmvpn/index.html


A Note On MTU

• All three providers recommend a different size interface MTU for the IPsec tunnel interface:

• Google recommends 1460 on the tunnel: https://cloud.google.com/vpn/docs/concepts/advanced#mtu

• AWS recommends 1399 on the tunnel: https://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/Introduction.html

• Azure recommends 1400 on the tunnel: https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpn-devices

• In addition to MTU, you need to set and test your TCP MSS values

• In my testing, an IP MTU of 1400 and TCP MSS of 1360 worked for all sites but this may need to change based on your applications and if you are adding other encapslike MPLS

https://cloud.google.com/vpn/docs/concepts/advanced#mtu

https://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/Introduction.html

https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpn-devices

Google Cloud Platform – Native VPN


Google Cloud Platform – VPN Gateway

• GCP Cloud VPN overview

• https://cloud.google.com/vpn/docs/concepts/overview

• GCP Cloud VPN documentation

• https://cloud.google.com/vpn/docs/how-to/creating-vpns

• GCP Advanced VPN documentation

• https://cloud.google.com/vpn/docs/concepts/advanced

Reference

https://cloud.google.com/vpn/docs/concepts/overview

https://cloud.google.com/vpn/docs/how-to/creating-vpns

https://cloud.google.com/vpn/docs/concepts/advanced


Topology for GCP to On Premises CSR – IPsec VPNBGP Routing

34BRKCLD-3440

Default Network

10.138.0.0/20

IPsec/IKEv2

Tunnel Mode

BGP<>OSPF Redistribution

192.xxx.xxx.x

35.xxx.xxx.x


Google Cloud VPN

Google Cloud Router


169.254.0.1

169.254.0.2 Private Network

192.168.100.0/24

.1

OSPF 10 Area 0


Hypervisor

Cisco CSR1000v


gcloud – Create the VPN GW, External IP and Forwarding RulesCreate a VPN gateway

# gcloud compute target-vpn-gateways create csr-gcp-vm-gw --region us-west1 --network default

Create an external IP to use for the VPN

# gcloud compute addresses create gcp-to-csr --region us-west1

Capture the external IP address

# gcloud compute addresses list --filter="gcp-to-csr”

NAME REGION ADDRESS STATUS

gcp-to-csr us-west1 35.xxx.xxx.x RESERVED

Create a forwarding rule for ESP, UDP500 and UDP4500 – These are used by IKE/IPsec

# gcloud compute forwarding-rules create csr-gcp-vm-rule-esp \

--region us-west1 \

--address 35.xxx.xxx.x \

--ip-protocol ESP \

--target-vpn-gateway csr-gcp-vm-gw

# gcloud compute forwarding-rules create csr-gcp-vm-rule-udp500 \

--region us-west1 \


--ip-protocol UDP --ports 500 \


# gcloud compute forwarding-rules create csr-gcp-vm-rule-udp4500 \

--region us-west1 \


--ip-protocol UDP --ports 4500 \



gcloud – Create Cloud Router, VPN Tunnel and BGP session

Create the Cloud router that is used for BGP (an existing router can be used)

# gcloud compute routers create csr-gcp-vm-bgp-rtr \

--region us-west1 \

--asn=65000 \

--network default

Create a VPN tunnel and link it to the router created in the previous step

# gcloud compute vpn-tunnels create csr-gcp-vm-gw-tunnel-1 \

--region us-west1 \

--peer-address 192.xxx.xxx.x --shared-secret <pre-shared-password-goes-here> \

--ike-version 2 \

--target-vpn-gateway csr-gcp-vm-gw \

--router csr-gcp-vm-bgp-rtr

Add a new interface to the router and set the BGP session IP address for the GCP side of the connection

# gcloud compute routers add-interface csr-gcp-vm-bgp-rtr \

--interface-name if-csr-gcp-vm-bgp-rtr-01 \

--ip-address 169.254.0.1 \

--mask-length 30 \

--vpn-tunnel csr-gcp-vm-gw-tunnel-1 \

--region us-west1

Create a new BGP peer – This peer will be the Cisco CSR at the On Premises cloud

# gcloud compute routers add-bgp-peer csr-gcp-vm-bgp-rtr \

--interface if-csr-gcp-vm-bgp-rtr-01 \

--peer-asn 65002 \

--peer-name csr-gcp-vm-bgp-peer \

--peer-ip-address 169.254.0.2 \

--region us-west1


Cisco CSR Route Informationcsr-gcp-01# show ip route

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2

i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

ia - IS-IS inter area, * - candidate default, U - per-user static route

o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

a - application route

+ - replicated route, % - next hop override, p - overrides from PfR

S* 0.0.0.0/0 [1/0] via 192.xxx.xxx.x

10.0.0.0/20 is subnetted, 1 subnets

B 10.138.0.0 [20/100] via 169.254.0.1, 00:16:59

169.254.0.0/16 is variably subnetted, 2 subnets, 2 masks

C 169.254.0.0/30 is directly connected, Tunnel0

L 169.254.0.2/32 is directly connected, Tunnel0

192.xxx.xxx.x/24 is variably subnetted, 2 subnets, 2 masks

C 192.xxx.xxx.x/26 is directly connected, GigabitEthernet1

L 192.xxx.xxx.x/32 is directly connected, GigabitEthernet1


C 192.168.100.0/24 is directly connected, GigabitEthernet2

L 192.168.100.1/32 is directly connected, GigabitEthernet2

.1

BGP

169.254.0.1

169.254.0.2

192.168.100.0/24

Default Network

10.138.0.0/20

Area 0Google Cloud VPN

Google Cloud Router

... Output summarised

Google VPN –Dual/Redundant On Premises Cisco CSRs


Reference Topology for Dual Cisco CSR Design

Default Network

10.138.0.0/20

BGP AS65000

169.254.0.1


Google Cloud VPN

Google Cloud Router

.2

BGP AS65002


Priv

ate

Netw

ork

19

2.1

68

.10

0.0

/24

OSPF 10 Area 0

BGP AS65002

On Premises Cloud 1

vSphere Hosted Cisco CSR

ESXi Host 1

ESXi Host 2

vSphere Distributed vSwitch (DVS) with a Distributed PortGroup for the Private Network

.2

.3

HSRP – VIP = .1

169.254.0.9

169.254.0.10

169.254.0.2

192.yyy.yyy.y

192.xxx.xxx.x

35.xxx.xxx.x35.yyy.yyy.y

VM.20

Compute

Engine

2 1


Pre-Failure State (1)

[shmcfarl@instance-3 ~]$ traceroute 192.168.100.20

traceroute to 192.168.100.20 (192.168.100.20), 30 hops max, 60 byte packets

1 169.254.0.2 (169.254.0.2) 24.728 ms 24.780 ms 24.766 ms

2 192.168.100.20 (192.168.100.20) 24.480 ms 24.495 ms 24.482 ms


GCE Instance traceroutes via 169.254.0.2 GCP BGP Path

[root@k8s-m-01 ~]# traceroute 10.138.0.2


1 192.168.100.2 (192.168.100.2) 0.545 ms 0.468 ms 0.415 ms

2 10.138.0.2 (10.138.0.2) 96.261 ms 58.105 ms 77.147 ms

On Premises VM traceroutes via HSRP Active CSR (192.168.100.2)

csr-gcp-01#show ip route

. . .

B 10.138.0.0/20 [20/100] via 169.254.0.1, 00:03:41

HSRP Active CSR Route to GCP Default Network (10.138.0.0)

csr-gcp-02#show ip route

. . .

B 10.138.0.0/20 [20/100] via 169.254.0.9, 00:08:47

HSRP Standby CSR Route to GCP Default Network (10.138.0.0)

csr-gcp-01#show stand

GigabitEthernet2 - Group 0 (version 2)

State is Active

HSRP Active



State is Standby

HSRP Standby



• Determining best path

• If Cloud Router receives multiple routes for the same destination, GCP uses route metrics and, in some cases, AS path length to determine the best path. To help you configure your On Premises routers, the following list describes the algorithm that GCP uses for egress traffic.

• If you have multiple BGP sessions on a single Cloud Router, GCP uses the route with the shortest AS path length.

• If routes have the same AS path length, GCP uses the route with the lower MED value.

• If routes have equal costs (same AS path length and metric), GCP uses ECMP to balance traffic across multiple paths.

• If you use multiple Cloud Routers, GCP uses only the MED value to determine the best path. The AS path length doesn't influence the path selection between multiple Cloud Routers.

• If a static and dynamic route have the same prefix and metric, GCP uses the static route.

# gcloud compute routers get-status csr-gcp-vm-bgp-rtr

kind: compute#routerStatusResponse

result:

bestRoutes:

- creationTimestamp: '2017-09-19T14:48:49.137-07:00'

destRange: 192.168.100.0/24

kind: compute#route

nextHopIp: 169.254.0.10

priority: 0


destRange: 192.168.100.0/24

kind: compute#route

nextHopIp: 169.254.0.2

priority: 0

bestRoutesForRouter:


destRange: 192.168.100.0/24

kind: compute#route


priority: 0

bgpPeerStatus:

- advertisedRoutes:

- destRange: 10.138.0.0/20

kind: compute#route


priority: 100

ipAddress: 169.254.0.1

name: csr-gcp-vm-bgp-peer

numLearnedRoutes: 1

peerIpAddress: 169.254.0.2

state: Established

status: UP

uptime: 1 minutes, 48 seconds

uptimeSeconds: '108'

network: https://www.googleapis.com/compute/v1/projects/public-175116/global/networks/default


First Google Cloud Router BGP State https://cloud.google.com/router/docs/concepts/overview

https://cloud.google.com/router/docs/concepts/overview



• Determining best path

• If Cloud Router receives multiple routes for the same destination, GCP uses route metrics and, in some cases, AS path length to determine the best path. To help you configure your On Premises routers, the following list describes the algorithm that GCP uses for egress traffic.

• If you have multiple BGP sessions on a single Cloud Router, GCP uses the route with the shortest AS path length.

• If routes have the same AS path length, GCP uses the route with the lower MED value.

• If routes have equal costs (same AS path length and metric), GCP uses ECMP to balance traffic across multiple paths.

• If you use multiple Cloud Routers, GCP uses only the MED value to determine the best path. The AS path length doesn't influence the path selection between multiple Cloud Routers.

• If a static and dynamic route have the same prefix and metric, GCP uses the static route.

# gcloud compute routers get-status csr-gcp-vm-bgp-rtr-02

kind: compute#routerStatusResponse

result:

bestRoutes:


destRange: 192.168.100.0/24

kind: compute#route

nextHopIp: 169.254.0.10

priority: 0


destRange: 192.168.100.0/24

kind: compute#route


priority: 0

bestRoutesForRouter:


destRange: 192.168.100.0/24

kind: compute#route

nextHopIp: 169.254.0.10

priority: 0

bgpPeerStatus:

- advertisedRoutes:

- destRange: 10.138.0.0/20

kind: compute#route


priority: 100

ipAddress: 169.254.0.9

name: csr-gcp-vm-bgp-peer-02

numLearnedRoutes: 1

peerIpAddress: 169.254.0.10

state: Established

status: UP

uptime: 6 minutes, 50 seconds

uptimeSeconds: '410'

network: https://www.googleapis.com/compute/v1/projects/public-175116/global/networks/default


Second Google Cloud Router BGP State https://cloud.google.com/router/docs/concepts/overview

https://cloud.google.com/router/docs/concepts/overview


Failure Scenario 1 – HSRP Primary CSR VM Reload

csr-gcp-02#

*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Standby: i/Resign rcvd (110/192.168.100.2)

*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Active router is local, was 192.168.100.2

*Sep 19 21:59:17.396: HSRP: Gi2 Nbr 192.168.100.2 no longer active for group 0 (Standby)

*Sep 19 21:59:17.396: HSRP: Gi2 Nbr 192.168.100.2 Was active or standby - start passive holddown

*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Standby router is unknown, was local

*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Standby -> Active

*Sep 19 21:59:17.396: %HSRP-5-STATECHANGE: GigabitEthernet2 Grp 0 state Standby -> Active

*Sep 19 21:59:17.396: HSRP: Peer not present

*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Redundancy "hsrp-Gi2-0" state Standby -> Active

*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Added 192.168.100.1 to ARP (0000.0c9f.f000)

*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Activating MAC 0000.0c9f.f000

*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Adding 0000.0c9f.f000 to MAC address filter

*Sep 19 21:59:17.396: HSRP: Gi2 IP Redundancy "hsrp-Gi2-0" standby, local -> unknown

*Sep 19 21:59:17.398: HSRP: Gi2 IP Redundancy "hsrp-Gi2-0" update, Standby -> Active

*Sep 19 21:59:20.379: HSRP: Gi2 IP Redundancy "hsrp-Gi2-0" update, Active -> Active

*Sep 19 21:59:57.361: %OSPF-5-ADJCHG: Process 10, Nbr 192.168.100.2 on GigabitEthernet2 from FULL to DOWN, Neighbor Down: Dead timer expired


HSRP Debug on HSRP Standby

[root@k8s-m-01 ~]# traceroute 10.138.0.2


1 192.168.100.3 (192.168.100.3) 0.545 ms 0.468 ms 0.415 ms

2 10.138.0.2 (10.138.0.2) 96.261 ms 58.105 ms 77.147 ms

On Premises VM traceroutes via HSRP Newly Active CSR (192.168.100.3)


Failure Scenario 2 – Shut HSRP Primary LAN Interface(BGP session is still active)

44BRKCLD-3440



1 169.254.0.2 (169.254.0.2) 24.223 ms 24.430 ms 24.716 ms

2 192.168.100.20 (192.168.100.20) 24.180 ms 24.595 ms 24.422 ms

Pre-Failure GCE Instance traceroutes via 169.254.0.2 GCP BGP Path



1 169.254.0.10 (169.254.0.10) 32.756 ms 42.796 ms 25.635 ms

2 192.168.100.20 (192.168.100.20) 66.674 ms 72.234 ms 74.331 ms

Post-Failure GCE Instance traceroutes via 169.254.0.10 GCP BGP Path


Failure Scenario 3 – Shut IPsec Tunnel on HSRP Primary CSR – With/Without HSRP Interface Tracking



1 169.254.0.2 (169.254.0.2) 24.728 ms 24.780 ms 24.766 ms

2 192.168.100.20 (192.168.100.20) 24.480 ms 24.495 ms 24.482 ms

Pre-Failure GCE Instance traceroutes via 169.254.0.2 GCP BGP Path



1 169.254.0.10 (169.254.0.10) 24.863 ms 42.763 ms 32.908 ms

2 192.168.100.2 (192.168.100.2) 54.069 ms 86.788 ms 70.963 ms

3 192.168.100.20 (192.168.100.20) 174.753 ms * 134.706 ms

Post-Failure GCE Instance traceroutes via 169.254.0.10 GCP BGP Path BUT traffic is re-routed to the HSRP Primary (192.168.100.2) before going to the end host

On Premises LAN re-route to HSRP Active

on router with failed IPsec Tunnel

track 10 interface Tunnel0 line-protocol

!

interface GigabitEthernet2

description Private Network On Premises

ip address 192.168.100.2 255.255.255.0

standby version 2

standby 0 ip 192.168.100.1

standby 0 priority 110

standby 0 preempt

standby 0 authentication md5 key-string 7 01300F175804575D720D

standby 0 track 10 decrement 10

LAN Re-Route Issue Resolved – Use Track



1 169.254.0.10 (169.254.0.10) 43.113 ms 25.269 ms 33.033 ms

2 192.168.100.20 (192.168.100.20) 72.879 ms 111.849 ms 53.904 ms



State is Standby

. . .

Priority 100 (configured 110)

Track object 10 state Down decrement 10

Tunnel failed and

track changed

HSRP state


Reference Cisco CSR Config – Primary

46BRKCLD-3440

crypto ikev2 proposal PHASE1-PROP

encryption aes-cbc-256

integrity sha1

group 14

!

crypto ikev2 policy IKE-POL

proposal PHASE1-PROP

!

crypto ikev2 keyring KEY

peer GCP-PEER

address 35.yyy.yyy.y

hostname csr-gcp-dmz-sjc

pre-shared-key local <PSK_PASSWORD_GOES_HERE>

pre-shared-key remote <PSK_PASSWORD_GOES_HERE>

!

crypto ikev2 profile IKEV2-SETUP

match identity remote address 0.0.0.0

authentication local pre-share

authentication remote pre-share

keyring local KEY

lifetime 36000

!

crypto ikev2 dpd 10 2 periodic

!

track 10 interface Tunnel0 line-protocol

!

crypto ipsec transform-set CSR-GCP-SET esp-aes 256 esp-sha-hmac

mode tunnel

!

crypto ipsec profile CSR-GCP

set transform-set CSR-GCP-SET

set pfs group14

set ikev2-profile IKEV2-SETUP

... Output summarizedinterface Tunnel0

ip address 169.254.0.2 255.255.255.252

ip mtu 1400

ip tcp adjust-mss 1360

tunnel source GigabitEthernet1

tunnel mode ipsec ipv4

tunnel destination 35.yyy.yyy.y

tunnel protection ipsec profile CSR-GCP

!


ip address 192.yyy.yyy.y 255.255.255.192

!



ip address 192.168.100.2 255.255.255.0

standby version 2

standby 0 ip 192.168.100.1


standby 0 preempt

standby 0 authentication md5 key-string 7 <HSRP_KEY>

standby 0 track 10 decrement 10

!

router ospf 10

redistribute bgp 65002 subnets

network 192.168.100.0 0.0.0.255 area 0

!

router bgp 65002


neighbor 169.254.0.1 remote-as 65000

neighbor 169.254.0.1 timers 20 60 60

!

address-family ipv4

redistribute ospf 10

neighbor 169.254.0.1 activate

neighbor 169.254.0.1 soft-reconfiguration inbound

!

ip route 0.0.0.0 0.0.0.0 192.yyy.yyy.y


Reference Cisco CSR Config – Secondary

47BRKCLD-3440

crypto ikev2 proposal PHASE1-PROP


integrity sha1

group 14

!



!


peer GCP-PEER

address 35.xxx.xxx.x

hostname csr-vpn-gw-02



!





keyring local KEY

lifetime 36000

!


!

crypto ipsec transform-set CSR-GCP-SET esp-aes 256 esp-sha-hmac

mode tunnel

!

crypto ipsec profile CSR-GCP

set transform-set CSR-GCP-SET

set pfs group14


interface Tunnel0

ip address 169.254.0.10 255.255.255.252

ip mtu 1400




tunnel destination 35.xxx.xxx.x

tunnel protection ipsec profile CSR-GCP

!


ip address 192.xxx.xxx.x 255.255.255.192

!



ip address 192.168.100.3 255.255.255.0

standby version 2

standby 0 ip 192.168.100.1


standby 0 preempt

standby 0 authentication md5 key-string 7 <HSRP_KEY>

!

router ospf 10


network 192.168.100.0 0.0.0.255 area 0

!

router bgp 65002



neighbor 169.254.0.9 timers 20 60 60

!

address-family ipv4




!

ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x

Amazon Web Services –Native VPN


AWS – VPN Gateway

• AWS VPN Overview http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpn-connections.html

• AWS VPN Setup http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/SetUpVPNConnections.html

• AWS does support NAT-T: https://aws.amazon.com/blogs/aws/ec2-vpc-vpn-update-nat-traversal-additional-encryption-options-and-more/

• Example templates for Cisco IOS: http://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/Cisco.html

https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpngateways

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/SetUpVPNConnections.html

https://aws.amazon.com/blogs/aws/ec2-vpc-vpn-update-nat-traversal-additional-encryption-options-and-more/

http://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/Cisco.html


Topology for AWS to On Premises CSR – IPsec VPNBGP Routing

50BRKCLD-3440

VPC Network

172.31.0.0/16


VPN Gateway

VPCRouter

BGP <> OSPF Redistribution

192.xxx.xxx.x

Private Network

192.168.200.0/24

.1

OSPF 10 Area 0

Hypervisor

Cisco CSR1000v

52.xxx.xxx.x

IPsec/IKEv2

Tunnel Mode

169.254.11.177

BGP AS65002

169.254.11.178

BGP AS64512



AWS CLI: Create VPC, VPN GW, Customer GW and VPN Connection

Create a new AWS VPC (or use an existing one)

# aws ec2 create-vpc --cidr-block 172.31.0.0/16

Create VPN Gateway and set the AWS BGP ASN

# aws ec2 create-vpn-gateway --type ipsec.1 --amazon-side-asn 64512

Attach VPN Gateway to the VPC

# aws ec2 attach-vpn-gateway --vpc-id vpc-ce2124aa --vpn-gateway-id vgw-64277e21

Create a new customer gateway with the On Premises BGP ASN and the On Premises router IP address (do this for each connection)

# aws ec2 create-customer-gateway --bgp-asn 65002 --public-ip 192.xxx.xxx.x --type ipsec.1

Create a new VPN connection

# aws ec2 create-vpn-connection --customer-gateway-id cgw-d6055d93 --type ipsec.1 --vpn-gateway-id vgw-64277e21

Note: Lots of output will come from the above VPN creation command.

This information can be used to build the On Premises CSR config. The best method for getting the configuration is

shown on the next slide.

Enable route propagation for the VPC

# aws ec2 enable-vgw-route-propagation --gateway-id vgw-64277e21 --route-table-id rtb-515e8e36

Permit SSH and ICMP

# aws ec2 authorize-security-group-ingress --group-name default --protocol tcp --port 22 --cidr 0.0.0.0/0

# aws ec2 authorize-security-group-ingress --group-name default --protocol icmp --port -1 --cidr 0.0.0.0/0


Optional: Download Router Configuration

• VPC Dashboard > VPN Connections


Reference Cisco CSR Config - Primarycrypto isakmp policy 200

encryption aes 128

authentication pre-share

group 2

lifetime 28800

hash sha

!

crypto keyring keyring-vpn-cec15996-0

local-address 192.xxx.xxx.x

pre-shared-key address 52.xxx.xxx.x key

<PSK_PASSWORD_GOES_HERE>

!

crypto isakmp profile isakmp-vpn-cec15996-0

local-address 192.xxx.xxx.x

match identity address 52.xxx.xxx.x

keyring keyring-vpn-cec15996-0

!

crypto ipsec transform-set ipsec-prop-vpn-cec15996-0 esp-aes

128 esp-sha-hmac

mode tunnel

!

crypto ipsec profile ipsec-vpn-cec15996-0

set pfs group2

set security-association lifetime seconds 3600

set transform-set ipsec-prop-vpn-cec15996-0

!

crypto ipsec df-bit clear

!

crypto isakmp keepalive 10 10 on-demand

!

crypto ipsec fragmentation before-encryption


interface Tunnel1

ip address 169.254.11.178 255.255.255.252

ip virtual-reassembly

ip mtu 1400

tunnel source 192.xxx.xxx.x



tunnel protection ipsec profile ipsec-vpn-cec15996-0


!

router ospf 10


network 192.168.200.0 0.0.0.255 area 0

!

router bgp 65002



neighbor 169.254.11.177 timers 10 30 30

!

address-family ipv4






Verify Routing and Reachability


ubuntu@ip-172-31-0-121:~$ ping 192.168.200.30

PING 192.168.200.30 (192.168.200.30) 56(84) bytes of data.

64 bytes from 192.168.200.30: icmp_seq=1 ttl=63 time=4.95 ms


Connect to an AWS instance and ping to the on-premises private network

csr-mc-01#show ip route | i 172.31.0.0

B 172.31.0.0/16 [20/100] via 169.254.11.177, 00:13:35

On the on-premises CSR check the route for the the AWS VPC network 172.31.0.0/16

# aws ec2 describe-route-tables | grep 192.168.200.0

ROUTES 192.168.200.0/24 vgw-64277e21 EnableVgwRoutePropagation active

On AWS check for the route for the on-premises network (192.168.200.0/24)

Private Network

192.168.200.0/24

.1

VPC Network

172.31.0.0/16

Hypervisor

Cisco CSR1000v

.1.121

VM.30

169.254.11.178

169.254.11.177

BGP


Topology for Dual Cisco CSR on AWS

VPC Network

172.31.0.0/16

169.254.11.177


BGP AS65002


Priv

ate

Netw

ork

192.1

68.2

00.0

/24

OSPF 10 Area 0BGP AS65002

On Premises Cloud 1

vSphere Hosted Cisco CSR

ESXi Host 1

ESXi Host 2

vSphere Distributed vSwitch (DVS) with a Distributed PortGroup for the Private Network

.2

.3

HSRP – VIP = .1

169.254.10.214

169.254.11.178

VPN Gateway

VPCRouter

169.254.10.213

BGP AS64512

Microsoft Azure – Native VPN


Microsoft Azure – VPN Gateway

• Azure VPN Overview

• https://azure.microsoft.com/en-us/services/vpn-gateway/

• https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpngateways

• In order to use BGP you must use Route-Based VPN and SKUs VpnGw1, VpnGw2, VpnGw3, Standard or HighPerformance SKUs : https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-bgp-overview

https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-bgp-overview


Azure to On Premises CSR – IPsec VPNBGP Routing

58BRKCLD-3440

VPN Gateway


192.xxx.xxx.x

Private Network

192.168.200.0/24

.1

OSPF 10 Area 0

Hypervisor

Cisco CSR1000v

40.xxx.xxx.x

IPsec/IKEv2

Tunnel Mode

10.10.255.30

BGP AS65002

10.11.255.1

BGP AS64512

Vnet Subnet

10.10.0.0/16


Azure CLI: Create Resource Group, Networks, Subnets

Create a new Azure Resource Group (rg)

# az group create --name azure-vpn-rg --location westus

# az configure --defaults location=westus

# az configure --defaults group=azure-vpn-rg

Create a new virtual network (vnet) and a new ‘outside’ subnet

# az network vnet create \

--name vnet1 \

--address-prefix 10.10.0.0/16 \

--subnet-name outside \

--subnet-prefix 10.10.0.0/24

Create a ’inside’ subnet

# az network vnet subnet create \

--vnet-name vnet1 \

--name inside \

--address-prefix 10.10.1.0/24

Create a new subnet that is used for the IPsec/BGP interface on the Azure side


--vnet-name vnet1 \

--name gatewaysubnet \



Azure CLI: Create a Public IP, VPN/Vnet Gateway and Local Gateway

Create a new public IP address (Using Azure VPN service, the allocation must be ‘dynamic’)

# az network public-ip create \

--name azure-vpn-gw-eip \

--allocation-method dynamic

Create Vnet gateway using ‘RouteBased’ (BGP) and a supported sku (see earlier links for requirements). THIS TAKES AWHILE

# az network vnet-gateway create \

--name vpn-gw \

--public-ip-address azure-vpn-gw-eip \

--vnet vnet1 \

--gateway-type Vpn \

--sku VpnGw1 \

--vpn-type RouteBased \

--asn 65010

Once the Vnet gateway is up, get the Azure-side BGP Peering address (Needed for On Premises configuration)

# az network vnet-gateway list | grep bgpPeeringAddress

"bgpPeeringAddress": "10.10.255.30",

Create the local gateway (On Premises target). Local prefix/BGP peer should be the On Premises CSR tunnel info. Can’t be in Azure vnet range

# az network local-gateway create \

--gateway-ip-address 192.xxx.xxx.x \

--name azure-lng \

--local-address-prefixes 10.11.255.1/32 \

--asn 65002 \

--bgp-peering-address 10.11.255.1


Azure CLI: Vnet GW, Local GW, VPN ConnectionCopy the full path from the “id” line (under the ‘gatewayType: Vpn’ line) that is shown in the vnet-gateway output

# az network vnet-gateway show --name vpn-gw

"gatewayType": "Vpn",

"id": "/subscriptions/<YOUR_ID>/resourceGroups/azure-vpn-rg/providers/Microsoft.Network/virtualNetworkGateways/vpn-gw",

Copy the full path from the “id” line that is shown in the vnet-gateway output

# az network local-gateway show --name azure-ln

"id": "/subscriptions/<YOUR_ID>/resourceGroups/azure-vpn-rg/providers/Microsoft.Network/localNetworkGateways/azure-lng"

Create the VPN connection using information from above

# az network vpn-connection create \

--name azure-to-csr \

--vnet-gateway1 /subscriptions/<YOUR_ID>/resourceGroups/azure-vpn-rg/providers/Microsoft.Network/virtualNetworkGateways/vpn-gw \

--enable-bgp \

--shared-key ”<YOUR_PRE_SHARED_KEY>" \

--local-gateway2 /subscriptions/<YOUR_ID>/resourceGroups/azure-vpn-rg/providers/Microsoft.Network/localNetworkGateways/azure-lng

Optional: Create a new test VM on Azure and associate it with the ‘inside’ subnet

# az vm create \

--name AzTestVm \

--authentication-type ssh \

--ssh-key-value "$(< ~/.ssh/id_rsa.pub)" \

--image Canonical:UbuntuServer:16.04-LTS:latest \

--size Standard_DS1_v2 \

--vnet-name vnet1 \

--subnet inside \

--public-ip-address-allocation dynamic


On Premises Cisco CSR IPsec/Routing Configcrypto ikev2 proposal PHASE1-PROP


integrity sha1

group 2

!



!


peer AZURE-PEER

address 40.xxx.xxx.x



!





keyring local KEY

lifetime 36000

!


!

crypto ipsec security-association replay window-size 1024

!

crypto ipsec transform-set CSR-AZURE-SET esp-aes 256 esp-sha-hmac

mode tunnel

!

crypto ipsec profile CSR-AZURE

set transform-set CSR-AZURE-SET

set pfs group14


... Output summarisedinterface Tunnel2

ip address 10.11.255.1 255.255.255.255

ip mtu 1400





tunnel protection ipsec profile CSR-AZURE

!


description Internet


no ip redirects

no ip unreachables

no ip proxy-arp

negotiation auto

!

router ospf 10

router-id 10.1.0.2


network 192.168.200.0 0.0.0.255 area 0

!

router bgp 65002



neighbor 10.10.255.30 ebgp-multihop 255

!

address-family ipv4




!

ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x

ip route 10.10.255.30 255.255.255.255 Tunnel2




shmcfarl@AzTestVm:~$ ping 192.168.200.30




Connect to an Azure instance and ping to the on-premises private network


B 10.10.0.0/16 [20/0] via 10.10.255.30, 00:51:26

On the on-premises CSR check the route for the Azure Vnet route of 10.10.0.0/16

PS Azure:\> Get-AzureRmEffectiveRouteTable -NetworkInterfaceName AzTestVmVMNic -ResourceGroupName azure-vpn-rg | Format-Table

Name State Source AddressPrefix NextHopType NextHopIpAddress

---- ----- ------ ------------- ----------- ----------------

Active VirtualNetworkGateway {192.168.200.0/24} VirtualNetworkGateway {40.xxx.xxx.x}

On Azure check for the route for the on-premises network (192.168.200.0/24)

Private Network

192.168.200.0/24

.1

Inside Subnet

10.10.1.0/24

Hypervisor

Cisco CSR1000v

.4

VM.30

192.xxx.xxx.x

40.xxx.xxx.x

10.10.255.30

10.11.255.1

VPN Gateway

Multicloud with Cisco SD-WAN


Cisco SD-WAN ArchitectureThe Power of Abstraction

65BRKCLD-3440

Management Plane- vManage- UI- Policies, templates- Monitoring

Control Plane- vSmart- Fabric discovery- Control plane policies

Data Plane- vEdge

APIs

vSmart Controllers

vAnalytics3rd Party

Automation

vManage

Data Centre Campus Branch SOHOCloud

vBond

vEdge Routers

4GMPLS

INET

Orchestration Plane- vBond- Orchestrates control

and mgmt. plane- First point of auth

Reference


Cisco SD-WAN

66BRKCLD-3440

Private Network

10.1.1.0/24

VNet Network

10.10.1.0/16

SD-WANOn-Premises

VPC Network

172.3.0.0/24

vEdge/cEdge

vEdge/cEdge

vEdge/cEdge


Cisco SD-WAN:

https://www.cisco.com/c/en/us/solutions/en

terprise-networks/sd-wan/index.html

https://www.cisco.com/c/en/us/solutions/enterprise-networks/sd-wan/index.html


Cisco SD-WAN

• Cisco SD-WAN (vEdge) on AWS: https://sdwan-docs.cisco.com/Product_Documentation/Getting_Started/Viptela_Overlay_Network_Bringup/07Deploy_the_vEdge_Routers/01Create_vEdge_Cloud_VM_Instance_on_AWS

• AWS Marketplace: https://aws.amazon.com/marketplace/pp/B07BZ53FJT

• Cisco SD-WAN on Microsoft Azure: https://sdwan-docs.cisco.com/Product_Documentation/Getting_Started/Viptela_Overlay_Network_Bringup/07Deploy_the_vEdge_Routers/02Create_vEdge_Cloud_VM_Instance_on_Azure

• Microsoft Azure Marketplace: https://azuremarketplace.microsoft.com/en-us/marketplace/apps/cisco.cisco_cloud_vedge_4_nics?tab=Overview

• Brand New SD-WAN Design/Deployment Guides: https://www.cisco.com/c/en/us/solutions/design-zone/networking-design-guides/branch-wan-edge.html

Public Cloud Support

67BRKCLD-3440

https://sdwan-docs.cisco.com/Product_Documentation/Getting_Started/Viptela_Overlay_Network_Bringup/07Deploy_the_vEdge_Routers/01Create_vEdge_Cloud_VM_Instance_on_AWS

https://aws.amazon.com/marketplace/pp/B07BZ53FJT

https://sdwan-docs.cisco.com/Product_Documentation/Getting_Started/Viptela_Overlay_Network_Bringup/07Deploy_the_vEdge_Routers/02Create_vEdge_Cloud_VM_Instance_on_Azure

https://azuremarketplace.microsoft.com/en-us/marketplace/apps/cisco.cisco_cloud_vedge_4_nics?tab=Overview

https://www.cisco.com/c/en/us/solutions/design-zone/networking-design-guides/branch-wan-edge.html


Cisco SD-WAN and AWS OptionsSD-WAN + Internet + Host VPC

Private Network(s)

On-Premises

VPC Subnet(s)

VPCRouter

vEdgeCloud IPsec

vEdge

SD-WAN + Transit VPC

Private Network(s)

On-Premises

VPC Subnet(s)

VPCRouter

VPNGateway

vEdgeCloud

Transit VPC

IPsecvEdge

SD-WAN + Some Combo of Colocation/peering

Private Network(s)

On-Premises

VPC Subnet(s)

VPCRouter

VPNGateway

vEdgeCloud

vEdgeDX Endpoint

VLANs

IPsecIPsec


Cisco SD-WAN – Transit VPC

• AWS: https://sdwan-docs.cisco.com/Product_Documentation/vManage_Help/Release_18.3/Configuration/Cloud_OnRamp_with_AWS

• Azure: https://sdwan-docs.cisco.com/Product_Documentation/vManage_Help/Release_18.3/Configuration/Cloud_OnRamp_with_Azure

Cloud onRamp for IaaS - AWS

69BRKCLD-3440

VPC Network

VPCRouter

Private Network

vEdge


vEdgeCloud

VPNGateway

Transit VPC

IPsec

On-Premises

https://sdwan-docs.cisco.com/Product_Documentation/vManage_Help/Release_18.3/Configuration/Cloud_OnRamp_with_AWS

https://sdwan-docs.cisco.com/Product_Documentation/vManage_Help/Release_18.3/Configuration/Cloud_OnRamp_with_Azure


AWS with Cisco SD-WANCloud onRamp for IaaS - AWS

70BRKCLD-3440

GatewayVpc (192.168.0.0/16)

Transit Subnet 1Transit Subnet 2


Transit Subnet 0

vpn 0192.168.59.199

vpn 1192.168.85.0

vpn

512

19

2.1

68.3

0.3

1Transit Subnet 0

vpn 0192.168.139.23

vpn 1192.168.176.185

vpn

51

21

92

.168

.12

6.1

06

EIP

EIP

EIP

EIP

IGWVPC Router

HostVpc (172.16.0.0/16)

PublicSubnet

PrivateSubnet

172.16.0.0/24

VPCRouter

172.16.3.0/24VPN GW (VGW)

VPN Tunnel

Private Network

10.1.1.0/24vedge


On-Premises

VPN Tunnel

EIP

EIP



71BRKCLD-3440

GatewayVpc (192.168.0.0/16)



Transit Subnet 0

vpn 0192.168.59.199

vpn 1192.168.85.0

vpn

512

19

2.1

68.3

0.3

1Transit Subnet 0

vpn 0192.168.139.23

vpn 1192.168.176.185

vpn

51

21

92

.168

.12

6.1

06

EIP

EIP

EIP

EIP

IGWVPC Router

HostVpc (172.16.0.0/16)

PublicSubnet

PrivateSubnet

172.16.0.0/24

VPCRouter

172.16.3.0/24VPN GW (VGW)

VPN Tunnel

VPN Tunnel

EIP

EIP

Private Network

10.1.1.0/24vedge


On-Premises

IPsec VPN

IPsec VPN



72BRKCLD-3440

GatewayVpc (192.168.0.0/16)



Transit Subnet 0

vpn 0192.168.59.199

vpn 1192.168.85.0

vpn

512

19

2.1

68.3

0.3

1Transit Subnet 0

vpn 0192.168.139.23

vpn 1192.168.176.185

vpn

51

21

92

.168

.12

6.1

06

EIP

EIP

EIP

EIP

IGWVPC Router

HostVpc (172.16.0.0/16)

PublicSubnet

PrivateSubnet

172.16.0.0/24

VPCRouter

172.16.3.0/24VPN GW (VGW)

VPN Tunnel

VPN Tunnel

EIP

EIP

Private Network

10.1.1.0/24vedge


On-Premises

IPsec VPN

IPsec VPN

IPsecVPN


vManageCloud onRamp for IaaS - AWS

73BRKCLD-3440

Dashboard View (Yeah, I know, no HA on the control plane )

Cloud onRamp for IaaS - AWS

Host VPCs are ‘mapped’ (connected via VPN) to the Transit VPCs

Transit VPCs – Two vEdge-Cloud EC2 Instances – These connect to the on-premises via SD-WAN setup


AWS – VPC/Subnet ViewCloud onRamp for IaaS - AWS

74BRKCLD-3440

VPC View

Subnet View


AWS – Host VPC –to- Transit VPC Mapping

75BRKCLD-3440

VPN Gateway (VPG) View

Customer Gateway Endpoints (EIPs of each Transit vEdge Cloud)

VPN Connections (only one of the two is shown below)


AWS – Host VPC –to- Transit VPC Mapping - IPsec

76BRKCLD-3440

interface ipsec8

ip address 169.254.10.14/30

tunnel-source 192.168.59.199

tunnel-destination 52.xx.xx.xx

ike

version 1

mode main

rekey 28800

cipher-suite aes128-cbc-sha1

group 2

authentication-type

pre-shared-key

pre-shared-secret <PSK_HERE>

!

!

!

ipsec

rekey 3600

replay-window 512

cipher-suite aes256-cbc-sha1

perfect-forward-secrecy group-16

!

vEdge-Cloud – Transit VPC

vpn 0192.168.59.199

Transit VPC vEdge

HostVpc (172.16.0.0/16)

PublicSubnet

PrivateSubnet

172.16.0.0/24

VPCRouter

172.16.3.0/24VPN GW (VGW)

VPN Tunnel

VPN Tunnel

EIP

EIP 169.254.10.13/30

169.254.10.14/30


AWS – Host VPC –to- Transit VPC Mapping - BGP

77BRKCLD-3440

vpn 1

router

bgp 9988

timers

holdtime 30

!

address-family ipv4-unicast

network 0.0.0.0/0

redistribute omp

!

neighbor 169.254.10.13

no shutdown

remote-as 64512

update-source ipsec8

vEdge-Cloud – Transit VPC

vpn 0192.168.59.199

Transit VPC vEdge

vedge-aws-01# show ip route

OUTPUT OMITTED...

PROTOCOL NEXTHOP NEXTHOP NEXTHOP

VPN PREFIX PROTOCOL SUB TYPE IF NAME ADDR VPN TLOC IP COLOR ENCAP STATUS

-------------------------------------------------------------------------------------------------------------------------------------

1 172.16.0.0/16 bgp e ipsec8 169.254.10.13 - - - - F,S

HostVpc (172.16.0.0/16)

PublicSubnet

PrivateSubnet

172.16.0.0/24

VPCRouter

172.16.3.0/24VPN GW (VGW)

VPN Tunnel

VPN Tunnel

EIP

EIP 169.254.10.13/30

169.254.10.14/30


Transit VPC –to- On-Premises - IPsec

78BRKCLD-3440

vedge-aws-01# show ipsec outbound-connections

OUTPUT SUMMARIZED...

SOURCE SOURCE DEST DEST REMOTE REMOTE AUTHENTICATION

IP PORT IP PORT SPI TUNNEL MTU TLOC ADDRESS TLOC COLOR USED

--------------------------------------------------------------------------------------------------------------------------------------

192.168.59.199 12406 <ON_PREMISES_vEDGE_PUBLIC_IP> 12346 270 1441 1.1.1.4 public-internet AH_SHA1_HMAC

Transit VPC vEdge - IPsec

vedge-01# show ipsec outbound-connections


SOURCE SOURCE DEST DEST REMOTE REMOTE AUTHENTICATION

IP PORT IP PORT SPI TUNNEL MTU TLOC ADDRESS TLOC COLOR USED

--------------------------------------------------------------------------------------------------------------------------------------

<ON_PREMISES_vEDGE_PUBLIC_IP> 12346 <TRANSIT-vEDGE-EIP> 12346 258 1441 2.2.2.5 default AH_SHA1_HMAC

<ON_PREMISES_vEDGE_PUBLIC_IP> 12346 <TRANSIT-vEDGE-EIP> 12406 258 1441 2.2.2.6 default AH_SHA1_HMAC

On-Premises vEdge - IPsec

Private Network

10.1.1.0/24

VPC CIDR

172.16.0.0/16

vEdge

EIP

vpn 0192.168.59.199

Transit VPC vEdge

VPN GW (VGW)

VPN Tunnel

EIP

IPsec VPN

IPsec VPN


Transit VPC –to- On-Premises - BGP

79BRKCLD-3440

vedge-aws-01# show ip route




--------------------------------------------------------------------------------------------------------------------------------------

1 10.1.1.0/24 omp - - - - 1.1.1.4 public-internet ipsec F,S

1 169.254.8.40/30 connected - ipsec7 - - - - - F,S

1 169.254.10.12/30 connected - ipsec8 - - - - - F,S

1 172.16.0.0/16 bgp e ipsec8 169.254.10.13 - - - - F,S

Transit VPC vEdge - BGP

vedge-01# show ip route




--------------------------------------------------------------------------------------------------------------------------------------

1 10.1.1.0/24 connected - ge0/1 - - - - - F,S

1 172.16.0.0/16 omp - - - - 2.2.2.5 default ipsec F,S

1 172.16.0.0/16 omp - - - - 2.2.2.6 default ipsec F,S

On-Premises vEdge - IPsec

Private Network

10.1.1.0/24

VPC CIDR

172.16.0.0/16

vEdge

EIP

vpn 0192.168.59.199

Transit VPC vEdge

VPN GW (VGW)

VPN Tunnel

EIP IPsec VPN

IPsec VPN

DMVPN – Dynamic Multipoint VPN

Merging in Multicloud to an Existing Branch/WAN Deployment


DMVPN (Dynamic Multipoint VPN)

• Cisco DMVPN

• https://www.cisco.com/c/en/us/products/security/dynamic-multipoint-vpn-dmvpn/index.html

• Cisco Live DMVPN

• https://www.ciscolive.com/global/on-demand-library/?search=dmvpn#/

• Cisco IWAN CVD

• https://www.cisco.com/c/en/us/solutions/design-zone/networking-design-guides/branch-wan-edge.html

• DMVPN is a Cisco innovation for building GRE/mGRE + IPsec VPN connections in a dynamic and scalable manner

https://www.ciscolive.com/global/on-demand-library/?search=dmvpn#/

https://www.ciscolive.com/global/on-demand-library/?search=dmvpn#/

https://www.cisco.com/c/en/us/solutions/design-zone/networking-design-guides/branch-wan-edge.html


Terminology and Features

192.168.102.0/24

Hub1

Spoke 1

Hub 2

Spoke 2

192.168.101.0/24

192.168.1.0/24 192.168.2.0/24

Tunnel: 10.0.0.101

Physical: 172.16.101.1

Tunnel: 10.0.0.1

Physical: 172.16.1.1Tunnel: 10.0.0.2

Physical: 172.16.2.1

Overlay Addresses

NBMA Address

Core Network

192.168.128.0/17

On Demand

Spoke Tunnels

Tunnel Address

Tunnel: 10.0.0.102

Physical: 172.16.102.1

GRE/IPsec

Tunnels


DMVPN Components

• Next Hop Resolution Protocol (NHRP)

• Creates a distributed (NHRP) mapping database of all the spoke’s tunnel to real (public interface) addresses

• Multipoint GRE Tunnel Interface (mGRE)

• Single GRE interface to support multiple GRE/IPsec tunnels

• Simplifies size and complexity of configuration

• IPsec tunnel protection

• Dynamically creates and applies encryption policies

• Routing

• Dynamic advertisement of branch networks; almost all routing protocols (EIGRP, RIP, OSPF, BGP, ODR) are supported


DMVPN Implementation

.

Hub and spoke

(Phase 1)Spoke-to-spoke

(Phase 2)

Server Load BalancingHierarchical (Phase 3)

VRF-lite

2547oDMVPN

Spoke-to-hub tunnels

Spoke-to-spoke tunnels

2547oDMVPN tunnels

Google Cloud Platform–Cisco CSR and DMVPN


GCP to On Premises CSR – IPsec VPN Example 1

86BRKCLD-3440

Default Network

10.138.0.0/20

IPsec/IKEv2

Tunnel Mode


192.xxx.xxx.x

35.xxx.xxx.x

Google Cloud VPN

Google Cloud Router


169.254.0.1

169.254.0.2 Private Network

192.168.200.0/24

.1

OSPF 10 Area 0

Hypervisor

Cisco CSR1000v

VM.30

Compute

Engine

2 1


GCP CSR to On Premises CSR – IPsec VPNExample 2

87BRKCLD-3440

Private Network

192.168.200.0/24

.1

inside-network

10.0.1.0/24

OSPF 10 Area 0

192.xxx.xxx.x

35.xxx.xxx.x

Cisco CSR1000v

Hypervisor

Cisco CSR1000v

Default Network

10.138.0.0/20

.100.3 .2

.1

IPsec/IKEv2

Tunnel Mode VM.30

OSPF

Compute

Engine

2 1


Private Network

192.168.200.0/24

.1

inside-network

10.0.1.0/24

OSPF 10 Area 0



Cisco CSR1000v

Hypervisor

Cisco CSR1000v

Default Network

10.138.0.0/20

.100.3 .2

.1

VM.30

192.xxx.xxx.x35.xxx.xxx.x

DMVPN

Hub

CSR Tunnel:

10.1.0.2

Spoke

CSR Tunnel:

10.1.0.1

88BRKCLD-3440

GCP CSR to On Premises CSR – DMVPN

OSPF

Compute

Engine

2 1


gcloud – Create the GCP External IP, Inside VPC Network and Route

Create a new external IP reservation that will be used for the GCP CSR NATed connection (or use an existing one)

# gcloud compute addresses create csr-to-csr-ext-ip --region us-west1

Capture the external IP address

# gcloud compute addresses list --filter="csr-to-csr-ext-ip"

NAME REGION ADDRESS STATUS

csr-to-csr-ext-ip us-west1 35.xxx.xxx.x RESERVED

Create a new GCP inside network that will be attached to the ‘inside’ interface of the CSR

# gcloud compute networks create inside-network --subnet-mode=custom

Create a new GCP inside subnet - Associate it with the inside network

# gcloud compute networks subnets create inside-subnet \

--network=inside-network \

--range=10.0.1.0/24

Create a new GCP route from the CSR inside network to the On Premises private network which routes through the IPsec VPN

# gcloud compute routes create inside-to-csr-private \


--destination-range=192.168.200.0/24 \

--next-hop-address=10.0.1.2


gcloud – Create GCP Firewall Rules

Create a new GCP firewall rule to allow traffic into the inside CSR network from the default network

# gcloud compute firewall-rules create allow-default-to-csr-inside \

--direction=INGRESS \


--action=ALLOW \

--rules=all \

--source-ranges=0.0.0.0/0

Create a new GCP firewall rule to allow traffic between the default network and the On Premises CSR public IP for IKE, IPsec

# gcloud compute firewall-rules create csr-csr-vpn \

--direction=INGRESS \

--network=default \

--action=ALLOW \

--rules=udp:500,udp:4500,esp \

--source-ranges=192.xxx.xxx.x


gcloud – Create CSR and Test Instances

Create a new GCE CSR instance and set fixed IPv4 addresses to each of the two interfaces

# gcloud compute instances create "csr-gcp-01" \

--zone "us-west1-a" \

--machine-type "n1-standard-4" \

--network-interface subnet="default",private-network-ip="10.138.0.100",address="35.xxx.xxx.x" \

--can-ip-forward \

--network-interface subnet="inside-subnet",private-network-ip="10.0.1.2",no-address \

--image ”name_of_csr_image" \

--boot-disk-size "10" \

--boot-disk-type "pd-standard" \

--boot-disk-device-name "csr-gcp-01"

Create a new GCE test instance that will be used to validate the VPN and routing

# gcloud compute instances create "csr-inside-vm" \

--zone "us-west1-a" \

--machine-type "g1-small" \

--subnet "inside-subnet" \

--private-network-ip "10.0.1.3" \

--image "debian-9-stretch-v20170918" \

--image-project "debian-cloud" \

--boot-disk-size "10" \

--boot-disk-type "pd-standard" \

--boot-disk-device-name "csr-inside-vm"


Connect to the GCP CSR – Enable Interfaces

# gcloud compute ssh cisco-user@csr-gcp-01

csr1kv-gcp#configure terminal

Enter configuration commands, one per line. End with CNTL/Z.

csr1kv-gcp(config)#interface gigabitEthernet 2

csr1kv-gcp(config-if)#ip address dhcp

csr1kv-gcp(config-if)#no shutdown


Connect to the new GCP-hosted CSR and enable the GigabitEthernet 2 interface for DHCP

csr1kv-gcp#show ip interface brief

Interface IP-Address OK? Method Status Protocol

GigabitEthernet1 10.138.0.100 YES TFTP up up

GigabitEthernet2 10.0.1.2 YES DHCP up up

Wait a few seconds and check to make sure that both interfaces on the CSR are up and with the correct IP addresses:


GCP Cisco CSR DMVPN ConfigSpoke

93BRKCLD-3440

crypto ikev2 proposal AES/GCM/256

encryption aes-gcm-256

prf sha512

group 19

!

crypto ikev2 policy AES/GCM/256

match fvrf any

proposal AES/GCM/256

!

crypto ikev2 keyring DMVPN-KEYRING

peer ANY

address 0.0.0.0 0.0.0.0

pre-shared-key <PSK_PASSWORD_GOES_HERE>

!

crypto ikev2 profile DMVPN-IKEv2-PROFILE

description PSK Profile


identity local address 35.xxx.xxx.x



keyring local DMVPN-KEYRING

dpd 40 5 on-demand

!


!

crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256

mode transport

!

crypto ipsec profile DMVPN-IPSEC-PROFILE

set transform-set AES256/GCM/TRANSFORM

set ikev2-profile DMVPN-IKEv2-PROFILE


interface Tunnel0

description DMVPN

ip address 10.1.0.1 255.255.255.0

no ip redirects

ip mtu 1400

ip nhrp authentication <NHRP_PASSWORD>

ip nhrp network-id 100

ip nhrp nhs 10.1.0.2 nbma 192.xxx.xxx.x multicast


ip ospf authentication-key 7 <OSPF_PASSWORD>

ip ospf network point-to-multipoint

ip ospf hello-interval 10


tunnel mode gre multipoint

tunnel key 100

tunnel protection ipsec profile DMVPN-IPSEC-PROFILE

!



ip address 10.138.0.100 255.255.255.0

no ip redirects

no ip unreachables

no ip proxy-arp

negotiation auto

!

router ospf 10

router-id 10.1.0.1

network 10.0.1.0 0.0.0.255 area 1

network 10.1.0.0 0.0.0.255 area 0

!

ip route 0.0.0.0 0.0.0.0 138.0.0.1


On Premises Cisco CSR DMVPN ConfigHub

94BRKCLD-3440



prf sha512

group 19

!


match fvrf any


!


peer ANY

address 0.0.0.0 0.0.0.0


!








dpd 40 5 on-demand

!


!


mode transport

!





interface Tunnel0

description DMVPN

ip address 10.1.0.2 255.255.255.0

no ip redirects

ip mtu 1400


ip nhrp map multicast dynamic


ip nhrp redirect







tunnel key 100


!




no ip redirects

no ip unreachables

no ip proxy-arp

negotiation auto

!

router ospf 10

router-id 10.1.0.2

network 10.1.0.0 0.0.0.255 area 0

network 192.168.200.0 0.0.0.255 area 0

!

ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x




# gcloud compute ssh "csr-inside-vm“

shmcfarl@csr-inside-vm:~$ ping 192.168.200.30





Connect to the GCP test instance that was created earlier and ping to the on-premises private network

csr1kv-gcp#show ip route | i 192.168.200.0

. . .

O 192.168.200.0/24 [110/1001] via 10.1.0.2, 00:09:51, Tunnel0

On the GCP CSR, check for the private network route from the on-premises side(192.168.200.0/24)


. . .

O IA 10.0.1.0/24 [110/1001] via 10.1.0.1, 00:40:08, Tunnel0

On the on-premises CSR, check for the VPC inside network route (10.0.1.0/24)

csr1kv-gcp#show ip nhrp

10.1.0.2/32 via 10.1.0.2

Tunnel0 created 5d14h, never expire

Type: static, Flags:

NBMA address: 192.xxx.xxx.x

Check the DMVPN Next-Hop Resolution Protocol (NHRP) Statuscsr-mc-01#show ip nhrp

10.1.0.1/32 via 10.1.0.1

Tunnel0 created 00:40:25, expire 00:08:20

Type: dynamic, Flags: registered used nhop


(Claimed NBMA address: 10.138.0.100)

Amazon Web Services –Cisco CSR and DMVPN


AWS with Cisco CSR 1000v Support

• Amazon Web Services Marketplace + Cisco CSR:

• https://aws.amazon.com/marketplace/search/results?x=0&y=0&searchTerms=csr&page=1&ref_=nav_search_box

• Cisco CSR for AWS Deployment

• DMVPNhttps://www.cisco.com/c/en/us/td/docs/solutions/Hybrid_Cloud/Intercloud/CSR/AWS/CSRAWS/CSRAWS_3.html

• Deployment https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/aws/b_csraws.html

• Cisco Live Session for AWS with Cisco CSR:

• https://www.ciscolive.com/global/on-demand-library/?search=brkarc-2023#/session/1486155288098001AhER

• Transit VPC with CSR: http://d2zmdbbm9feqrf.cloudfront.net/2017/usa/pdf/BRKARC-2749.pdf

https://www.cisco.com/c/en/us/td/docs/solutions/Hybrid_Cloud/Intercloud/CSR/AWS/CSRAWS/CSRAWS_3.html

https://www.cisco.com/c/en/us/td/docs/solutions/Hybrid_Cloud/Intercloud/CSR/AWS/CSRAWS/CSRAWS_3.html

https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/aws/b_csraws.html

https://www.ciscolive.com/global/on-demand-library/?search=brkarc-2023#/session/1486155288098001AhER

http://d2zmdbbm9feqrf.cloudfront.net/2017/usa/pdf/BRKARC-2749.pdf


AWS to On Premises CSR – IPsec VPNExample 1

98BRKCLD-3440

VPC Network

172.31.0.0/16VPN

GatewayVPC

Router


192.xxx.xxx.x

Private Network

192.168.200.0/24

.1

OSPF 10 Area 0

Hypervisor

Cisco CSR1000v

52.xxx.xxx.x

IPsec/IKEv2

Tunnel Mode

169.254.11.177

BGP AS65002

169.254.11.178

BGP AS64512


AWS CSR to On Premises CSR – IPsec VPNExample 2

99BRKCLD-3440

VPC Network

172.16.2.0/24VPC

Router

OSPF

192.xxx.xxx.x

Private Network

192.168.200.0/24

.1

OSPF 10 Area 0

Hypervisor

Cisco CSR1000v

52.xxx.xxx.x

IPsec/IKEv2

Tunnel Mode

Public-side Network

172.16.1.0/24

Cisco CSR1000v


AWS CSR to On Premises CSR – DMVPN

VPC Network

172.16.2.0/24

VPCRouter

192.xxx.xxx.x Private Network

192.168.200.0/24

.1

OSPF 10 Area 0

Hypervisor

Cisco CSR1000v

52.xxx.xxx.x



Public-side Network

172.16.1.0/24

Cisco CSR1000v

DMVPN

Hub

CSR Tunnel:

10.1.0.2

Spoke

CSR Tunnel:

10.1.0.4

OSPF


AWS CLI: Create VPC, Subnets and Internet GW

Create a new AWS VPC (vpc)

# aws ec2 create-vpc --cidr-block 172.16.0.0/16

Create a new subnet in the VPC (this one will be used for the CSR’s ’outside’ interface)

# aws ec2 create-subnet --vpc-id vpc-66a0a102 --cidr-block 172.16.1.0/24

Create another new subnet in the VPC (this one will be used for the CSR’s ‘inside’ interface)

# aws ec2 create-subnet --vpc-id vpc-66a0a102 --cidr-block 172.16.2.0/24

Create a new AWS Internet Gateway (igw)

# aws ec2 create-internet-gateway

Attach the Internet gateway to the VPC

# aws ec2 attach-internet-gateway --vpc-id vpc-66a0a102 --internet-gateway-id igw-591fba3d


AWS CLI: Create Route Tables

Create a new route table in the VPC (rtb) that will be used by the CSR’s ‘outside’ subnet

# aws ec2 create-route-table --vpc-id vpc-66a0a102

Create a new default route in the route table and point it to the Internet gateway

# aws ec2 create-route --route-table-id rtb-aaa37dcd --destination-cidr-block 0.0.0.0/0 --gateway-id igw-591fba3d

Associate the new routable with the ‘outside’ VPC subnet

# aws ec2 associate-route-table --subnet-id subnet-0c15b86b --route-table-id rtb-aaa37dcd

Create a new route table in the VPC (rtb) that will be used by the CSR’s ‘inside’ subnet

# aws ec2 create-route-table --vpc-id vpc-66a0a102

Create a new default route in the route table for the ‘inside’ subnet and point it to the Internet gateway

# aws ec2 create-route --route-table-id rtb-3741e750 --destination-cidr-block 0.0.0.0/0 --gateway-id igw-591fba3d

Create a new default route in the route table and point it to the Internet gateway

# aws ec2 create-route --route-table-id rtb-3741e750 --destination-cidr-block 192.168.200.0/24 --network-interface-id eni-af67db80

Associate the new route table with the ‘inside’ VPC subnet

# aws ec2 associate-route-table --subnet-id subnet-c617baa1 --route-table-id rtb-3741e750


AWS CLI: Create a Security Group/RulesCreate a new security group for the ‘outside’ facing interface (Optional: You can just use an existing group)

# aws ec2 create-security-group --group-name csr --description csr-rules --vpc-id vpc-66a0a102

Create a new security group rule for SSH to the CSR

# aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 --protocol tcp --port 22 --cidr 0.0.0.0/0

Create a new security group rule for ICMP from the other CSRs (On Premises and GCP CSR [optional: Just showing the format for your use])

# aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 \

--ip-permissions '[{"IpProtocol": "icmp", "FromPort": -1, "ToPort": -1, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]

Create a new security group rule for ESP (IP 50) from the other CSRs


--ip-permissions '[{"IpProtocol": "50", "IpRanges": [{"CidrIp": ”192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]'

Create a new security group rule for IKE from the other CSRs


--ip-permissions '[{"IpProtocol": "17", "FromPort": 500, "ToPort": 500, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]'

Create a new security group rule for IKE/NAT-T from the other CSRs


--ip-permissions '[{"IpProtocol": "17", "FromPort": 4500, "ToPort": 4500, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]’

Optional: You may want to create a security group just for the ’inside’ subnet that has different rules than the one for the ‘outside’ subnet

Create a new security group for the ‘outside’ facing interface (Optional: You can just use an existing group)

# aws ec2 authorize-security-group-ingress --group-id sg-84aef7e2 --protocol all --cidr 192.168.200.0/24

Create a new security group for the ‘outside’ facing interface (Optional: You can just use an existing group)

# aws ec2 authorize-security-group-ingress --group-id sg-84aef7e2 --protocol all --cidr 172.16.2.0/24

Reference


AWS CLI: Run a new CSR Instance Using Previous Parameters

{

"ImageId": "ami-99e5d0f9",

"InstanceType": "t2.medium",

"KeyName": "mc-aws-key",

"NetworkInterfaces": [

{

"DeviceIndex": 0,

"Description": "Primary network interface",

"Groups": [

"sg-65c39b03"

],

"PrivateIpAddresses": [

{

"Primary": true,

"PrivateIpAddress": "172.16.1.10"

}

],

"SubnetId": "subnet-0c15b86b"

},

{

"DeviceIndex": 1,

"PrivateIpAddresses": [

{

"Primary": true,

"PrivateIpAddress": "172.16.2.10"

}

],

"SubnetId": "subnet-c617baa1"

}

]

}

csr-create.json

Create a CSR instance using the JSON file shown to the left

# aws ec2 run-instances --cli-input-json file://csr-create.json

Create a tag/name and associate it with the CSR (Optional)

# aws ec2 create-tags --resources i-0f2a0ee857e9c2540 \

--tags Key=Name,Value=csr-aws-01

Create a new External IP (EIP) allocation (or use an existing one)

# aws ec2 allocate-address

eipalloc-ab35cb96 vpc 52.xxx.xxx.x

Associate the EIP with the ’outside’ interface of the CSR (GigabitEthernet 1)

# aws ec2 associate-address --allocation-id eipalloc-ab35cb96 \

--network-interface-id eni-dd5bd6f2

Modify the ’inside’ subnet to disable source/destination checking

# aws ec2 modify-network-interface-attribute \

--network-interface-id eni-af67db80 \

--source-dest-check "{\"Value\": false}"

A note about NAT: If you plan to use the CSR for NAT operation, you must disable source/destination checking on the outside CSR interface/subnet

http://docs.aws.amazon.com/AmazonVPC/latest/UserG

uide/VPC_NAT_Instance.html#EIP_Disable_SrcDestCh

eck

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_NAT_Instance.html#EIP_Disable_SrcDestCheck


Connect to the AWS CSR – Enable Interfaces

# ssh -i "mc-aws-key.pem" [email protected]

csr-aws-01#configure terminal


csr-aws-01(config)#interface gigabitEthernet 2

csr-aws-01(config-if)#ip address dhcp

csr-aws-01(config-if)#no shutdown

Connect to the new AWS-hosted CSR and enable the GigabitEthernet 2 interface for DHCP

csr-aws-01#show ip interface brief




VirtualPortGroup0 192.168.35.1 YES TFTP up up

Wait a few seconds and check to make sure that both interfaces on the CSR are up and have the correct IP addresses:

Note: This can all be automated (along with the DMVPN configs) by creating AWS CloudFormation templates


AWS Cisco CSR DMVPN ConfigSpoke

106BRKCLD-3440



prf sha512

group 19

!


match fvrf any


!


peer ANY

address 0.0.0.0 0.0.0.0


!








dpd 40 5 on-demand

!


!


mode transport

!





interface Tunnel0

description DMVPN

ip address 10.1.0.4 255.255.255.0

no ip redirects

ip mtu 1400










tunnel key 100


!



ip address dhcp

no ip redirects

no ip unreachables

no ip proxy-arp

negotiation auto

!

router ospf 10

router-id 10.1.0.4

network 172.16.2.0 0.0.0.255 area 2

network 10.1.0.0 0.0.0.255 area 0

!

ip route 0.0.0.0 0.0.0.0 172.16.1.1


On Premises Cisco CSR DMVPN ConfigHub – Nothing ever changes on the hub for each example

107BRKCLD-3440



prf sha512

group 19

!


match fvrf any


!


peer ANY

address 0.0.0.0 0.0.0.0


!








dpd 40 5 on-demand

!


!


mode transport

!





interface Tunnel0

description DMVPN

ip address 10.1.0.2 255.255.255.0

no ip redirects

ip mtu 1400




ip nhrp redirect







tunnel key 100


!




no ip redirects

no ip unreachables

no ip proxy-arp

negotiation auto

!

router ospf 10

router-id 10.1.0.2

network 10.1.0.0 0.0.0.255 area 0

network 192.168.200.0 0.0.0.255 area 0

!

ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x




[ec2-user@ip-172-16-2-192 ~]$ ping 192.168.200.30





Connect to an AWS instances and ping to the on-premises private network


O IA 172.16.2.0 [110/1001] via 10.1.0.4, 00:11:41, Tunnel0

On the on-premises CSR check the route for the AWS VPC network 172.16.2.0/24

csr-aws-01#show ip route | i 192.168.200.0

O 192.168.200.0/24 [110/1001] via 10.1.0.2, 6d17h, Tunnel0

On AWS check for the route for the on-premises network (192.168.200.0/24)

Private Network

192.168.200.0/24

.1

VPC Network

172.16.2.0/24

Hypervisor

Cisco CSR1000v

.10.192

VM.30

OSPF

Hub

CSR Tunnel:

10.1.0.2

Spoke

CSR Tunnel:

10.1.0.4

Cisco CSR1000v

Amazon Web Services –Marketplace-based Launch Walk-thru

For Reference


AWS Marketplace CSR Launch – Console (1)


AWS Launch CSR as an Instance – Console (1)



1

2

3

4

Microsoft Azure – Cisco CSR and DMVPN


Azure to On Premises CSR – IPsec VPNExample 1

122BRKCLD-3440

VPN Gateway


192.xxx.xxx.x

Private Network

192.168.200.0/24

.1

OSPF 10 Area 0

Hypervisor

Cisco CSR1000v

40.xxx.xxx.x

IPsec/IKEv2

Tunnel Mode

169.254.11.177

BGP AS65002

169.254.11.178

BGP AS64512

Vnet Subnet

10.10.0.0/16


Azure CSR to On Premises CSR – IPsec VPNExample 2

123BRKCLD-3440

Inside Subnet

10.10.1.0/24

OSPF

192.xxx.xxx.x

Private Network

192.168.200.0/24

.1

OSPF 10 Area 0

Hypervisor

Cisco CSR1000v

40.xxx.xxx.x

IPsec/IKEv2

Tunnel Mode

Outside Subnet

10.10.0.0/24

Cisco CSR1000v


Azure CSR to On Premises CSR – DMVPN

192.xxx.xxx.x Private Network

192.168.200.0/24

.1

OSPF 10 Area 0

Hypervisor

Cisco CSR1000v

40.xxx.xxx.x



Cisco CSR1000v

DMVPN

Hub

CSR Tunnel:

10.1.0.2

Spoke

CSR Tunnel:

10.1.0.6

OSPF

Inside Subnet

10.10.1.0/24

Outside Subnet

10.10.0.0/24


Microsoft Azure with Cisco CSR 1000v

• Microsoft Azure Marketplace

• https://azuremarketplace.microsoft.com/en-us/marketplace/apps/cisco.cisco-csr-basic-template

• https://github.com/Azure/azure-quickstart-templates/tree/master/cisco-csr-1000v

• Cisco CSR 1000v with Azure Deployment

• https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/azu/b_csr1000config-azure.html

https://azuremarketplace.microsoft.com/en-us/marketplace/apps/cisco.cisco-csr-basic-template

https://github.com/Azure/azure-quickstart-templates/tree/master/cisco-csr-1000v

https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/azu/b_csr1000config-azure.html


Azure CLI: Create Resource Group, Networks, Subnets

Create a new Azure Resource Group (rg)

# az group create --name multicloud-rg --location westus

Create a new public (external IP) IPv4 address to be used for the CSR’s ‘outside’ interface

# az network public-ip create --resource-group multicloud-rg --name csr-azure-01-eip --allocation-method static

Create a new virtual network (vnet) and a subnet to be used for the CSR’s ‘outside interface

# az network vnet create \

--resource-group multicloud-rg \

--name mc-csr-vnet \

--address-prefix 10.10.0.0/16 \

--subnet-name csr-outside \

--subnet-prefix 10.10.0.0/24

Create a new subnet for the CSR’s ‘inside’ interface and associate it with the vnet created above



--vnet-name mc-csr-vnet \

--name csr-inside \



Azure CLI: Create Route Tables

127BRKCLD-3440

Create a new route table (rt) that will be used for the CSR’s ’outside’ subnet

# az network route-table create \


--name csr-outside-rt

Create a new route table that will used for the CSR’s ‘inside’ subnet

# az network route-table create \


--name csr-inside-rt

Create a new route table entry for the ‘inside’ subnet to reach the On Premises network (192.168.200.0) via the CSR’s IP (10.10.1.4)

# az network route-table route create \


--name csr-to-On Premises-route \

--route-table-name csr-inside-rt \

--address-prefix 192.168.200.0/24 \

--next-hop-type VirtualAppliance \

--next-hop-ip-address 10.10.1.4

Associate the ‘outside’ route table with the ‘outside’ subnet

# az network vnet subnet update \



--name csr-outside \

--route-table csr-outside-rt

Associate the ‘inside’ route table with the ‘inside’ subnet

# az network vnet subnet update \



--name csr-inside \

--route-table csr-inside-rt


Azure CLI: Create Network Security Group (NSG)

128BRKCLD-3440

Create a new Network Security Group (NSG) to be used for the ‘outside’ CSR interface

# az network nsg create \


--name csr-nsg-outside

Create a new NSG rule to allow inbound SSH access to the CSR (can make it specific to an IP/Prefix)

# az network nsg rule create \


--nsg-name csr-nsg-outside \

--name SSHRule \

--priority 100 \

--source-address-prefixes 'Internet' \

--source-port-ranges '*' \

--destination-address-prefixes '*' \

--destination-port-ranges 22 \

--access Allow \

--protocol Tcp \

--direction inbound

Create a new NSG rule to allow inbound UDP 500 (IKE) traffic to the CSR (can make it specific to an IP/Prefix)




--name UDP-500 \

--priority 101 \





--access Allow \

--protocol Udp \

--direction inbound

Reference


Azure CLI: Create NSG Rule and NICs

129BRKCLD-3440

Create a new Network Security Group (NSG) to be used for the ‘outside’ CSR interface




--name UDP-4500 \

--priority 102 \





--access Allow \

--protocol Udp \

--direction inbound

Create a new NIC to be used by the CSR’s ‘outside’ interface. Associate the NIC with the NSG, Subnet, Public IP & enable forwarding

# az network nic create \


--name csr-nic-g1 \


--subnet csr-outside \

--network-security-group csr-nsg-outside \

--ip-forwarding true \

--public-ip-address csr-azure-01-eip

Create a new NIC to be used by the CSR’s ‘inside’ interface. Associate the NIC with the NSG, Subnet and enable forwarding

# az network nic create \


--name csr-nic-g2 \


--subnet csr-inside \

--ip-forwarding true

Reference


Azure CLI: Run a new CSR Instance Using Previous Parameters

Create a CSR VM using a Azure Marketplace image. Associate the VM with two the NICs created earlier.

# Note: The VM can be created with a large number of options to include SSH keys, image (BYOL, # of NICs), and size

# az vm create \


--name csr-azure-01 \

--admin-username csr-azure \

--admin-password <PASSWORD> \

--authentication-type password \

--image cisco:cisco-csr-1000v:16_6:16.6.120170804 \

--nics csr-nic-g1 csr-nic-g2 \

--size Standard_D2_v2


Connect to the Azure CSR – Enable Interfaces

# ssh [email protected]

csr-azure-01#configure terminal


csr-azure-01(config)#interface gigabitEthernet 2

csr-azure-01(config-if)#ip address dhcp

csr-azure-01(config-if)#no shutdown

Connect to the new Azure-hosted CSR and enable the GigabitEthernet 2 interface for DHCP

csr-azure-01#show ip interface brief




VirtualPortGroup0 192.168.35.1 YES TFTP up up

Wait a few seconds and check to make sure that both interfaces on the CSR are up and have the correct IP addresses:

Note: This can all be automated (along with the DMVPN configs) by creating Azure Automation/Resource Manager


Azure Cisco CSR DMVPN ConfigSpoke

132BRKCLD-3440



prf sha512

group 19

!


match fvrf any


!


peer ANY

address 0.0.0.0 0.0.0.0


!








dpd 40 5 on-demand

!


!


mode transport

!





interface Tunnel0

description DMVPN

ip address 10.1.0.6 255.255.255.0

no ip redirects

ip mtu 1400










tunnel key 100


!



ip address dhcp

no ip redirects

no ip unreachables

no ip proxy-arp

negotiation auto

!

router ospf 10

router-id 10.1.0.6

network 10.1.0.0 0.0.0.255 area 0

network 10.10.1.0 0.0.0.255 area 3

!

ip route 0.0.0.0 0.0.0.0 10.10.0.1


On Premises Cisco CSR DMVPN ConfigHub - Nothing ever changes on the hub for each example

133BRKCLD-3440



prf sha512

group 19

!


match fvrf any


!


peer ANY

address 0.0.0.0 0.0.0.0


!








dpd 40 5 on-demand

!


!


mode transport

!





interface Tunnel0

description DMVPN

ip address 10.1.0.2 255.255.255.0

no ip redirects

ip mtu 1400




ip nhrp redirect







tunnel key 100


!




no ip redirects

no ip unreachables

no ip proxy-arp

negotiation auto

!

router ospf 10

router-id 10.1.0.2

network 10.1.0.0 0.0.0.255 area 0

network 192.168.200.0 0.0.0.255 area 0

!

ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x




shmcfarl@AzTestVm:~$ping 192.168.200.30




Connect to an Azure instance and ping to the on-premises private network


O IA 10.10.1.0/24 [110/1001] via 10.1.0.6, 00:19:15, Tunnel0

On the on-premises CSR check the route for the Azure Vnet 10.10.1.0/24

csr-azure-01#show ip route | i 192.168.200.0

O 192.168.200.0/24 [110/1001] via 10.1.0.2, 00:17:57, Tunnel0

On Azure check for the route for the on-premises network (192.168.200.0/24)

Private Network

192.168.200.0/24

.1

Inside Subnet

10.10.1.0/24

Hypervisor

Cisco CSR1000v

.4.5

VM.30

OSPF

Hub

CSR Tunnel:

10.1.0.2

Spoke

CSR Tunnel:

10.1.0.6

Cisco CSR1000v

Azure – Marketplace-based Launch Walk-thru

For Reference


Azure Marketplace/Resource Search


Azure Marketplace– There are multiple CSR types to pick from


Azure Marketplace


Deployment Flow

139BRKCLD-3440

1 2 3

4

5

6

Linking DMVPN Sites


DMVPN – Enable Dynamic Multicloud NetworkingCisco DMVPN

141BRKCLD-3440

Private Network

192.168.200.0/24

VNet Network

10.10.1.0/24

DMVPN

BGP/OSPF/EIGRP


VPC Network

172.16.2.0/24Cisco

CSR1000v

Cisco CSR1000v

Cisco CSR1000v

Hub

Spoke

Spoke

VPC Network

10.0.1.0/24Cisco

CSR1000v

Spoke


General Guidelines for DMVPN Between Clouds

142BRKCLD-3440

• Set the VPC routes for each site

• Set the firewall/security groups/network security groups for each site/protocol

gcloud compute routes create inside-to-aws \

--network=csr-inside-network \



gcloud compute routes create inside-to-azure \

--network=csr-inside-network \



Create a very specific per-site route (AWS example allowing UDP500 from each cloud provider public IP)

aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 \

--ip-permissions '[{"IpProtocol": " ", "FromPort": , "ToPort": , "IpRanges": [{"CidrIp": " .x.x.x/32"},

{"CidrIp": " .x.x.x/32"}, {"CidrIp": " .x.x.x/32"}]}]’

Alternatively, you can open it up (Azure example)

az network nsg rule create \



--name UDP-4500 \

--priority 102 \





--access Allow \

--protocol Udp \

--direction inbound


Routing Example – All Sites• For spoke-to-spoke direct routing with DMVPN/NHRP:

• ‘ip nhrp redirect’ on the hubs

• ‘ip nhrp shortcut’ on the spokes

csr-mc-01#show ip route ospf


O IA 10.0.1.0/24 [110/1001] via 10.1.0.1, 02:40:45, Tunnel0

O 10.1.0.1/32 [110/1000] via 10.1.0.1, 02:40:45, Tunnel0

O 10.1.0.4/32 [110/1000] via 10.1.0.4, 01:18:49, Tunnel0

O 10.1.0.6/32 [110/1000] via 10.1.0.6, 00:56:19, Tunnel0

O IA 10.10.1.0/24 [110/1001] via 10.1.0.6, 00:55:34, Tunnel0


O IA 172.16.2.0 [110/1001] via 10.1.0.4, 01:18:49, Tunnel0


Hub On Premises CSR

csr1kv-gcp#show ip route ospf


O 10.1.0.2/32 [110/1000] via 10.1.0.2, 02:43:14, Tunnel0

O % 10.1.0.4/32 [110/2000] via 10.1.0.2, 01:21:14, Tunnel0

O % 10.1.0.6/32 [110/2000] via 10.1.0.2, 00:58:47, Tunnel0

O IA% 10.10.1.0/24 [110/2001] via 10.1.0.2, 00:58:00, Tunnel0


O IA% 172.16.2.0 [110/2001] via 10.1.0.2, 01:21:14, Tunnel0

O 192.168.200.0/24 [110/1001] via 10.1.0.2, 02:43:14, Tunnel0

Spoke – Google Cloud Platform CSR

csr-aws-01#show ip route ospf


O IA% 10.0.1.0/24 [110/2001] via 10.1.0.2, 01:21:32, Tunnel0

O % 10.1.0.1/32 [110/2000] via 10.1.0.2, 01:21:32, Tunnel0

O 10.1.0.2/32 [110/1000] via 10.1.0.2, 01:21:32, Tunnel0

O % 10.1.0.6/32 [110/2000] via 10.1.0.2, 00:59:01, Tunnel0

O IA% 10.10.1.0/24 [110/2001] via 10.1.0.2, 00:58:14, Tunnel0

O 192.168.200.0/24 [110/1001] via 10.1.0.2, 01:21:32, Tunnel0

Spoke – Amazon Web Services CSR

csr-azure-01#show ip route ospf


O IA% 10.0.1.0/24 [110/2001] via 10.1.0.2, 00:58:44, Tunnel0

O % 10.1.0.1/32 [110/2000] via 10.1.0.2, 00:58:44, Tunnel0

O 10.1.0.2/32 [110/1000] via 10.1.0.2, 00:58:44, Tunnel0

O % 10.1.0.4/32 [110/2000] via 10.1.0.2, 00:58:44, Tunnel0


O IA% 172.16.2.0 [110/2001] via 10.1.0.2, 00:58:44, Tunnel0

O 192.168.200.0/24 [110/1001] via 10.1.0.2, 00:58:44, Tunnel0

Spoke – Azure CSR

IA - OSPF inter area

% - next hop override


NHRP Example – Hub/Spoke

csr-mc-01#show ip nhrp

10.1.0.1/32 via 10.1.0.1





10.1.0.4/32 via 10.1.0.4





10.1.0.6/32 via 10.1.0.6





csr-mc-01#show ip nhrp multicast

I/F NBMA address

Tunnel0 35.xxx.xxx.x Flags: dynamic (Enabled)



Hub On Premises CSRcsr-azure-01#show ip nhrp

10.0.1.0/24 via 10.1.0.1


Type: dynamic, Flags: router rib nho



10.1.0.1/32 via 10.1.0.1


Type: dynamic, Flags: router nhop rib nho



10.1.0.2/32 via 10.1.0.2

Tunnel0 created 00:21:28, never expire

Type: static, Flags:


10.1.0.4/32 via 10.1.0.4


Type: dynamic, Flags: router nhop rib nho



10.10.1.0/24 via 10.1.0.6


Type: dynamic, Flags: router unique local

NBMA address: 10.10.0.4

(no-socket)

172.16.2.0/24 via 10.1.0.4


Type: dynamic, Flags: router rib nho



csr-azure-01#show ip nhrp multicast

I/F NBMA address

Tunnel0 192.xxx.xxx.x Flags: nhs (Enabled)

Spoke – Azure CSR

shmcfarl@AzureTestVm:~$ traceroute 10.0.1.3


1 10.10.1.4 (10.10.1.4) 1.220 ms 1.192 ms 1.328 ms

2 10.0.1.3 (10.0.1.3) 25.794 ms * 25.782 ms

Spoke – Azure VM

Split-Tunnelling/Routing Options


Split-Tunnel/Routing Options• All three public cloud providers allow for either split-tunnelling or forced/direct routing

• Split-tunnelling:

• Public cloud resources (instances/VMs, container clusters) will use the default VPC gateway for non-On Premises routes

• Public cloud resources will use the On Premises-specific routes advertised by the CSR

• Forced/Direct routing – All public cloud resources will use the VPN connection as their default route for ALL traffic (forces traffic through the On Premises site)

BGP

Google Cloud VPN

Google Cloud Router

Cisco CSR1000v10.0.0.1

VPC Subnetwork GW

External/NAT

Routing

192.xxx.xxx.x

35.xxx.xxx.x10.0.0.5

Compute

Engine

2 1

CSR High Availability


Public Cloud Provider – CSR High-Availability

• Common challenge with all public cloud provider is that there is not true layer 2 support on a VPC subnet – this prevents FHRPs from working properly

• Must setup a monitoring/tracking feature to watch for CSR interface/instance failure and adjust the VPC route table to point to 2nd CSR inside interface

• AWS CSR High-Availability:

• https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/aws/b_csraws/b_csraws_chapter_0100.pdf

• Azure CSR High-Availability:

• https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/azu/b_csr1000config-azure/b_csr1000config-azure_chapter_0110.html

https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/aws/b_csraws/b_csraws_chapter_0100.pdf

https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/azu/b_csr1000config-azure/b_csr1000config-azure_chapter_0110.html

Automation Challenges


Automating the Multicloud Network• Challenges:

• Different toolsets for different jobs (Ansible, Python, Bash scripts, Terraform, etc..)

• Different toolsets for different clouds (Heat for OpenStack, CloudFormation for AWS, Deployment Manager for GCP, Azure Automation)

• Different toolsets for different vendor products (Cisco NSO, CloudCentre, Prime, YANG development kit, etc..)

• There is no silver bullet - Start simple:

• Use what your team knows – Perform a gap analysis on what you have against what you need

• Initially, automate the things that hurt a lot to do by hand and that change frequently – I use free tools but that doesn’t mean the process is free

• Native Tools: It’s safe to use the cloud provider’s native automation toolset (e.g., AWS CloudFormation) when that is the only provider you need to deal with

• Abstracted Tools: When you are dealing with multiple providers to include on-premises providers (e.g., VMware vSphere or Microsoft Azure Stack), it makes life easier to abstract away from native cloud provider tool sets and use something like Terraform and/or combo of tools

• Full Stack Tools: When you want to stop pulling your hair out and you want to build full ‘stacks’ in nearly any environment, move to something that can treat the environment as a whole – Cisco CloudCentre: https://www.cisco.com/c/en/us/products/cloud-systems-management/cloudcenter/index.html

https://www.cisco.com/c/en/us/products/cloud-systems-management/cloudcenter/index.html


Amazon CloudFormation

• https://aws.amazon.com/cloudformation/

• Template-based (JSON/YAML) – Build a stack(s) from a template file

• Sometimes you need to run more than one stack (in order) to get what you need

• Example template: https://github.com/shmcfarl/multicloud/tree/master/aws/cloudformation

https://aws.amazon.com/cloudformation/

https://github.com/shmcfarl/multicloud/tree/master/aws/cloudformation


Google Cloud Platform – Deployment Manager

• https://cloud.google.com/deployment-manager/

• Configuration files (YAML), Templates (Python/Jinja2), Schema files (JSON)

• Sometimes you need to run more than one stack (in order) to get what you need

• Example templates: https://github.com/shmcfarl/multicloud/tree/master/gcp/deployment-manager

https://cloud.google.com/deployment-manager/

https://github.com/shmcfarl/multicloud/tree/master/gcp/deployment-manager


Microsoft Azure Automation/Resource Manager

• https://azure.microsoft.com/en-us/services/automation/

• Runbooks (create graphically, PowerShell, Python)

• Read and select these carefully: https://docs.microsoft.com/en-us/azure/automation/automation-runbook-types

• Resource Manager: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview

• https://github.com/Azure/azure-quickstart-templates/tree/master/cisco-csr-1000v

• Example template: https://github.com/shmcfarl/multicloud/blob/master/azure/resource-manager/az-arm-csr-cleaned.json

https://azure.microsoft.com/en-us/services/automation/

https://docs.microsoft.com/en-us/azure/automation/automation-runbook-types

https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview

https://github.com/Azure/azure-quickstart-templates/tree/master/cisco-csr-1000v

https://github.com/shmcfarl/multicloud/blob/master/azure/resource-manager/az-arm-csr-cleaned.json


Call APIs Directly

• Google Cloud Platform: https://cloud.google.com/compute/docs/reference/latest/

• Amazon Web Services: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Welcome.html

• Microsoft Azure: https://docs.microsoft.com/en-us/rest/api/

https://cloud.google.com/compute/docs/reference/latest/

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Welcome.html

https://docs.microsoft.com/en-us/rest/api/

Google VPN – Creating Google VPN, Router, IPsec, BGP via REST APIs


Google Cloud API – Creating GCP Cloud VPN/Routers

• Assumptions/environment:

• Understand how to authenticate to GCP APIs: https://cloud.google.com/docs/authentication/

• In this example, the Paw application was used to craft GET, POST and PATCH calls

• Some configurations have been sanitised for security purposes

• Have On Premises Cloud infrastructure deployed and a CSR/ASR configured (can be done after GCP side is deployed)

• In this example, the configuration will be deployed against the OpenStack use case discussed in the earlier slides

• In this example, the default network created by GCP will be used

• Note: gcloud has VERY long delays in commands if you have IPv6 enabled on your local machine – set to “link-local” mode on your Mac

https://cloud.google.com/docs/authentication/


Reference Topology for GCP API Example

Private Network

172.16.0.0/24

.11Default Network

10.138.0.0/20

IPsec/IKEv2

Tunnel Mode

OSPF 10 Area 0

OSPF<>BGP Redistribution

192.yyy.yyy.y

35.yyy.yyy.y



On Premises Cloud

Google Cloud VPN

Google Cloud Router


169.254.0.5

169.254.0.6


GCP API (1) – Create VPN GW and External IP

158BRKCLD-3440

POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways HTTP/1.1

Authorization: Bearer XXXX

Content-Type: application/json; charset=utf-8

Host: www.googleapis.com

Connection: close

Content-Length: 138

{

"name": "csr-gcp-os-aio-gw",

"network": "projects/<gcp_project_number>/global/networks/default",

"region": "projects/<gcp_project_number>/regions/us-west1"

}

POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/addresses HTTP/1.1




Connection: close

Content-Length: 29

{

"name": "gcp-to-os-dmz"

}

GET /compute/v1/projects/<gcp_project_number>/regions/us-west1/addresses/gcp-to-os-dmz HTTP/1.1




Connection: close

RESPONSE - SUMMARIZED:

"name": "gcp-to-os-dmz",

"description": "",

"address": ”35.yyy.yyy.y",

"status": "RESERVED",


POST: Create VPN Gateway

POST: Create External IP Address

GET: Get the External IP Address


GCP API (2) – Create Forwarding RulesPOST /compute/v1/projects/<gcp_project_number>/regions/us-west1/forwardingRules HTTP/1.1




Connection: close

Content-Length: 257

{

"name": "csr-gcp-os-aio-rule-esp",

"IPProtocol": "ESP",

"IPAddress": "35.yyy.yyy.y",

"region": "projects/<gcp_project_number>/regions/us-west1",

"target": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw"

}

POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/forwardingRules HTTP/1.1




Connection: close

Content-Length: 278

{

"name": "csr-gcp-os-aio-rule-udp500",

"IPProtocol": "UDP",



"target": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw",

"portRange": "500"

}

POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/forwardingRules HTTP/1.1




Connection: close

Content-Length: 280

{

"name": "csr-gcp-os-aio-rule-udp4500",

"IPProtocol": "UDP",



"target": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw",

"portRange": "4500"

}

POST: Create Forwarding rule for ESP


POST: Create Forwarding rule for UDP 500

POST: Create Forwarding rule for UDP 4500


GCP API (3) – Create Cloud Router and BGP Session

POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/routers HTTP/1.1




Connection: close

Content-Length: 574

{

"name": "csr-gcp-os-bgp-rtr",

"bgp": {

"asn": "65000"

},

"interfaces": [

{

"name": "if-csr-gcp-os-bgp-rtr-02",

"linkedVpnTunnel": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/vpnTunnels/csr-gcp-os-aio-gw-tunnel-1",

"ipRange": "169.254.0.5/30"

}

],

"bgpPeers": [

{

"name": "csr-gcp-os-bgp-peer",

"interfaceName": "if-csr-gcp-os-bgp-rtr-02",

"ipAddress": "169.254.0.5",

"peerIpAddress": "169.254.0.6",

"peerAsn": "65003"

}

],


"network": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/global/networks/default"

}

POST: Create Cloud Router, BGP session and link to the Cloud VPN tunnel



GCP API (5) – Create Cloud VPN TunnelPOST /compute/v1/projects/<gcp_project_number>/regions/us-west1/vpnTunnels HTTP/1.1




Connection: close

Content-Length: 417

{

"name": "csr-gcp-os-aio-gw-tunnel-1",

"sharedSecret": " <pre-shared-password-goes-here> ",

"router": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/routers/csr-gcp-os-bgp-rtr",

"peerIp": "192.yyy.yyy.y",


"ikeVersion": "2",

"targetVpnGateway": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw"

}

POST: Create a Cloud VPN tunnel and associated it with the Cloud router



Summary• Cisco Multicloud Solutions: https://www.cisco.com/c/en/us/solutions/cloud/multicloud-portfolio.html

• Public cloud native IPsec VPN support is good, but it is always point-to-point, does not have consistent support and lacks network-rich features - It may be good enough for your initial use case(s)

• If you have deployed or want to deploy SD-WAN, adding in your public cloud sites into your overall SD-WAN design can reap many operational and cost benefits

• If you have an existing WAN/Branch deployment of DMVPN, adding spokes at public cloud site(s) can help optimize traffic flow (no hair-pinning), enable rich network features at the public cloud site and allow for a consistent technical and operation experience

• Multicloud between multiple public cloud providers and on-premises look like distinctly separate hybrid cloud deployments but..

• You have to take into consideration:

• Team knowledge of public cloud operations, tools, automation

• Cross cloud tools and automation

• Diversity of network designs, protocols, security

• Multi-region designs

• Availability zones within and across providers

https://www.cisco.com/c/en/us/solutions/cloud/multicloud-portfolio.html

Q & A

#CLMEL

Complete Your Online Session Evaluation


• Give us your feedback and receive a complimentary Cisco Live 2019 Power Bank after completing the overall event evaluation and 5 session evaluations.

• All evaluations can be completed via the Cisco Live Melbourne Mobile App.

• Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at:

https://ciscolive.cisco.com/on-demand-library/

BRKCLD-3440 165


Thank you

#CLMEL

Reference

Application Deployment

GKE, Cloud VPN, Cloud Router and an On Premises CSR Deployment with Dynamic Routing (IP Alias)


Google Container Engine (GKE) – Dynamic Routing

• Prior to the IP alias feature, GKE clusters did not advertise their IP ranges via the GCP Cloud Router (BGP) service: https://cloud.google.com/container-engine/docs/ip-aliases

• IP alias and self-directed alias ranges, cluster IP ranges and service IP ranges can all be enabled via REST, gcloud and the GKE console

# gcloud beta container clusters create gke-cls-istio \

> --enable-ip-alias \

> --create-subnetwork name=gke-istio-subnetwork

https://cloud.google.com/container-engine/docs/ip-aliases


GKE – Dynamic Routing with On Premises CSR

Private Network

192.168.100.0/24

.1

Default Network:

- Subnetwork:

- Nodes: 10.0.0.0/22

- Container Range: 10.56.0.0/14

- Services Range: 10.0.16.0/20

192.168.100.20BGP

Google Cloud VPN

Google Cloud Router

Google Container Cluster (GKE)

Hypervisor

Cisco CSR1000v

Pods

10.56.0.0/24

Pods

10.56.1.0/24

Pods

10.56.2.0/24

10.0.0.2

10.0.0.3

10.0.0.4

10.0.0.1

eth0

eth0

eth0

cbr0

cbr0

cbr0

VPC Subnetwork

GW

External/NAT

Routing

192.xxx.xxx.x

35.xxx.xxx.x


Google Container Engine - Setup

Create a basic GKE cluster with IP alias enabled

# gcloud beta container clusters create gke-cls-istio \

> --enable-ip-alias \

> --create-subnetwork name=gke-istio-subnetwork

Get a list of the nodes

# kubectl get nodes

NAME STATUS AGE VERSION

gke-gke-cls-istio-default-pool-6724d65b-6lsc Ready 2m v1.7.6

gke-gke-cls-istio-default-pool-6724d65b-x04p Ready 2m v1.7.6

gke-gke-cls-istio-default-pool-6724d65b-zgdq Ready 2m v1.7.6

Check the IP ranges of the new subnetwork “gke-istio-subnetwork”

# gcloud compute networks subnets describe gke-istio-subnetwork | grep ipCidrRange

ipCidrRange: 10.0.0.0/22

- ipCidrRange: 10.56.0.0/14

- ipCidrRange: 10.0.16.0/20

Default Network:

- Subnetwork:

- Nodes: 10.0.0.0/22

- Container Range: 10.56.0.0/14

- Services Range: 10.0.16.0/20


Pods

10.56.0.0/24

Pods

10.56.1.0/24

Pods

10.56.2.0/24

10.0.0.2

10.0.0.3

10.0.0.4

eth0

eth0

eth0

cbr0

cbr0

cbr0


Google Container Engine – Node/Pod IP Verification

NAME STATUS AGE VERSION

gke-gke-cls-istio-default-pool-6724d65b-6lsc Ready 2m v1.7.6

gke-gke-cls-istio-default-pool-6724d65b-x04p Ready 2m v1.7.6

gke-gke-cls-istio-default-pool-6724d65b-zgdq Ready 2m v1.7.6

Using the node list from above, check the IPs assignments of each node

# kubectl describe nodes gke-gke-cls-istio-default-pool-6724d65b-zgdq | grep 'InternalIP\|PodCIDR'

InternalIP: 10.0.0.2

PodCIDR: 10.56.0.0/24

# kubectl describe nodes gke-gke-cls-istio-default-pool-6724d65b-6lsc | grep 'InternalIP\|PodCIDR'


PodCIDR: 10.56.1.0/24

# kubectl describe nodes gke-gke-cls-istio-default-pool-6724d65b-x04p | grep 'InternalIP\|PodCIDR'


PodCIDR: 10.56.2.0/24


Pods

10.56.0.0/24

Pods

10.56.1.0/24

Pods

10.56.2.0/24

10.0.0.2

10.0.0.3

10.0.0.4

eth0

eth0

eth0

cbr0

cbr0

cbr0


GKE/GCP and On Premises CSR Dynamic Routing

Get the advertised route list from the GCP Cloud Router

# gcloud compute routers get-status csr-gcp-vm-bgp-rtr

. . .

result:

. . .

bgpPeerStatus:

- advertisedRoutes:

- destRange: 10.0.16.0/20

kind: compute#route


priority: 100

- destRange: 10.56.0.0/14

kind: compute#route


priority: 100

- destRange: 10.0.0.0/22

kind: compute#route


priority: 100

Check the BGP routes on the On Premises CSR

csr-gcp-01#show ip route bgp

. . .

B 10.0.0.0/22 [20/100] via 169.254.0.1, 00:00:04

B 10.0.16.0/20 [20/100] via 169.254.0.1, 00:00:04

B 10.56.0.0/14 [20/100] via 169.254.0.1, 00:00:04


169.254.0.1

169.254.0.2

BGP

Google Cloud VPN

Google Cloud Router

Cisco CSR


GKE and CSR Routing/Access Verification

From a VM at the On Premises network (192.168.100.0/24), ping a GKE nodes IP and the cbr0 interface on that node

[root@k8s-m-01 ~]# ip a

. . .

2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

link/ether 00:50:56:bc:4b:91 brd ff:ff:ff:ff:ff:ff

inet 192.168.100.20/24 brd 192.168.100.255 scope global ens192

valid_lft forever preferred_lft forever

inet6 fe80::50de:b58f:8dc8:2fd5/64 scope link

valid_lft forever preferred_lft forever

[root@k8s-m-01 ~]# ping 10.0.0.2




[root@k8s-m-01 ~]# ping 10.56.0.1





Pods

10.56.0.0/24

Pods

10.56.1.0/24

Pods

10.56.2.0/24

10.0.0.2

10.0.0.3

10.0.0.4

eth0

eth0

eth0

cbr0

cbr0

cbr0


GKE Pod Routing/Access VerificationDeploy an nginx pod

# kubectl run my-nginx --image=nginx --port=80

deployment "my-nginx" created

# kubectl get pods

NAME READY STATUS RESTARTS AGE

my-nginx-4293833666-1jbjl 1/1 Running 0 14s

Find the IP addres of the pod

# kubectl describe pods my-nginx-4293833666-1jbjl | grep IP:

IP: 10.56.0.5

Ping the IP address of the pod from the On Premises VM

[root@k8s-m-01 ~]# ping 10.56.0.5




curl the nginx pod

[root@k8s-m-01 ~]# curl -o /dev/null -s -w "%{http_code}\n" http://10.56.0.5

200


Google Container Engine• Deploy Pods

Deploy NGINX as a test

# kubectl run my-nginx --image=nginx --replicas=3 --port=80

deployment "my-nginx" created

Check to make sure the pods are running

# kubectl get pods

NAME READY STATUS RESTARTS AGE

my-nginx-858393261-7x8mp 1/1 Running 0 6s

my-nginx-858393261-rt9sp 1/1 Running 0 6s

my-nginx-858393261-vhq6f 1/1 Running 0 6s

Get the IPv4 address for each pod

# kubectl describe pods my-nginx-858393261-7x8mp | grep IP:

IP: 10.28.2.18

# kubectl describe pods my-nginx-858393261-rt9sp | grep IP:

IP: 10.28.3.36

# kubectl describe pods my-nginx-858393261-vhq6f | grep IP:

IP: 10.28.1.29

Complete Your Online Session Evaluation


• Give us your feedback and receive a complimentary Cisco Live 2019 Power Bank after completing the overall event evaluation and 5 session evaluations.

• All evaluations can be completed via the Cisco Live Melbourne Mobile App.

• Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at:



Thank you

#CLMEL

#CLMEL

Multicloud Networking - Cisco Live

Documents

Transcript of Multicloud Networking - Cisco Live