Advanced Visibility and Monitoring in Nexus 3000 ... - Cisco Live

Yogesh Ramdoss Technical Leader, Customer Experience

BRKDCN-3020

Advanced Visibility and Monitoring in Nexus 3000/9000 Switches

Agenda

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCN-3020 3

• Introduction

• Built-in CLIs and Tools

• NX-OS Programmability

• Use-cases

• Conclusion

Introduction


Source: Readers’ Digest

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCN-3020

Advanced Visibility and Monitoring Why we need it?

6

Monitor health and performance of the devices and network

Brings up efficiency of troubleshooting –reducing downtime

Helps to plan and schedule resources

Automation starts with having a better visibility


This session …

7

• Creates awareness of highly-useful CLIs and tools built into the Nexus 3000 and 9000 (standalone) platforms that can be used to get visibility and insights of the devices, and their performance.

• Discusses a solution that can be built with the platforms in discussion

• Highlights various programmability features and discusses telemetry streaming capabilities in detail.

• Brings various capabilities together to get valuable insights into the network, with real-world use-cases.

Reference


Are you ready?

8

THERE’S NEVER BEEN A BETTER TIME TO ….

Get More Visibility

…. and ….

Insights

Built-in CLIs & Tools


Built-in CLIs & ToolsTopics

10

• Latency Monitoring with Precision Time Protocol (PTP)

• SPAN to CPU

• Catena – Application Chaining Solution


Latency MonitoringWhy we need it ?

11

It is simple ….

Latency could impact applications’ performance and result in bad user experience

Accessing

Websites

Video

Conferencing

Online

Games

Trade

Floors and so on…

Video

Streaming


Precision Time Protocol (PTP)

• Time synchronization protocol defined by IEEE 1588. Defines PTP profiles, master selection and more.

• Modes: Multicast (destined to 224.0.1.129), and Unicast. In both modes, PTP used UDP ports 319/320.

• Clocks: Ordinary, Boundary or Transparent*(end-to-end or peer-to-peer).

• Process: Establish master-slave hierarchy and synchronize the clocks

• Clock Synch: One or Two step modes (Synch vs. Synch + Follow-up)

Introduction

Grand Master

Nexus Switch

GPS

12

* not supported by Cisco

Server Farms

Boundary Clock 1

Boundary Clock 2

Slave

Nexus

Nexus

Master Clock


ERSPAN with PTPWhat exactly happens in the switch?

13

1

2

34

Receive Transmit

Frame is received on Eth1/2

Forwarding decision. Send the frame to Eth1/26

Packet

GRE with PTP

5Transmit

Replicated GRE packet and PTP timestamp

Original packet transmitted out

ERSPAN packet transmitted out

PTP

192.168.100.100

monitor session 1 type erspan-source

source interface Ethernet 1/2 rx

header-type 3

destination ip 192.168.100.100


Latency MonitoringConfiguring ERSPAN with PTP

Nexus9k-A(config)# monitor session 1 type erspan-source

Nexus9k-A(config-erspan-src)# source interface Ethernet 1/2 rx

Nexus9k-A(config-erspan-src)# header-type 3

Nexus9k-A(config-erspan-src)# destination ip 192.168.100.100

Nexus9k-A(config-erspan-src)# filter vlan 1-10, 15

Nexus9k-A(config-erspan-src)# erspan-id 11 Eth1/2

Nexus9k-D

Network

14

Server running Wireshark 192.168.100.100

More config options:ERSPAN Source Intf

VRFIP TTL

DSCP MarkingMarker-Packet

Difference in the timestamps of the two captures provide latency incurred in the network

Nexus9k-A

PTP

PTP


Latency MonitoringConfiguring ERSPAN with PTP

Eth1/2

Nexus9k-D

Nexus9k-A

15

3rd-Party Application

Nexus9k-A(config)# monitor session 1 type erspan-source

Nexus9k-A(config-erspan-src)# source interface Ethernet 1/2 rx

Nexus9k-A(config-erspan-src)# header-type 3

Nexus9k-A(config-erspan-src)# destination ip 192.168.100.100

Nexus9k-A(config-erspan-src)# filter vlan 1-10, 15

Nexus9k-A(config-erspan-src)# erspan-id 11

Having captures at multiple points in the network help to monitor latencies incurred in different segments of the network

PTP

PTP

Network

PTP


Tools

Nexus Data Broker (NDB)TAP and Aggregation with PTP

16

Production Network

Cisco®

SPAN ports

Cisco Nexus Data Broker

3rd-Party Tools and Applications

Cisco Tools and Applications

Traffic forward with

PTP timestamp

NDB Controller

OpenFlow or NX-API

Nexus 3000/9000

PTP not supported?? No worries!!

PTP

Timestamp Tagging (TTAG)


Precision Time Protocol (PTP)Guidelines and Limitations

17

• No licenses required

• Supported in port-channel member ports, but not on FEX interfaces

• Limited to single domain per network

• Nexus3100 supports PTP, starting from 7.0(3)I4(1) release.

• All Nexus9000 series except 100G 9408PC and M4PC GEM modules support PTP. 9300-FX supports PTP from 7.0(3)I7(1) onwards.

• 9732C-EX, 9736C-EX, 97160YC-EX modules, and 9200/9300-EX switches support PTP Offloading and Timestamp Tagging (TTAG) starting from 7.0(3)I6(1) release

• PTP over IPv6 not supported



18


• SPAN to CPU



SPAN to CPUIntroduction and Configuration

19

Switch Port ANalyzer (SPAN) mirrors the traffic from source ports/VLANs to destination port(s).

monitor session 1

source interface eth1/1

destination interface eth1/6

SwitchEth1/1

SPAN Source

Sniffer Device

SPAN Replicated packet

SPAN Destination

SwitchEth1/1

SPAN Source SPAN Replicated packet

SPAN DestinationCPUIn SPAN to CPU, the destination port is the CPU in the switch.

monitor session 1

source interface eth1/1

destination interface sup-eth 0

<options>

Eth1/6

But, how to view the packets sent to CPU?


EthanalyzerProcess and Configuration

20

(1) Identify Capture Interface

• mgmt – captures traffic on mgmt0 interface

• Inband - captures traffic sent to the control-plane/CPU

(2) Configure Filter

• Display-filter – captures all traffic but displays only the traffic meeting the criteria

• Capture-filter - captures all traffic meeting the criteria

(3) Define Stop Criteria

• By default, it stops after capturing 10 frames. Can be changed with limit-captured-frames configuration. 0 means no limit, until user issues cntrl+C

• autostop can be used, to stop the capture after specified duration, filesize, or number of files.

Is there a way to differentiate the packets normally sent to CPU vs. SPAN to CPU packets?


EthanalyzerFilters Configuration

21

• There are two filtering approaches for configuring a packet capture

http://wiki.wireshark.org/DisplayFilters

http://wiki.wireshark.org/CaptureFilters

Display Filter Example Capture Filter Example

“eth.addr==00:00:0c:07:ac:01” “ether host 00:00:0c:07:ac:01”

“ip.src==10.1.1.1 && ip.dst==10.1.1.2” “src host 10.1.1.1 and dst host 10.1.1.2”

“snmp” "udp port 161”

“ospf” “ip proto 89”

http://wiki.wireshark.org/DisplayFilters

http://wiki.wireshark.org/CaptureFilters


SPAN to CPUTroubleshooting Packet Loss

22

monitor session 1source interface eth1/2 rxdestination interface sup-eth 0filter access-group ACL1no shut

ip access-list ACL1permit icmp 10.214.10.5/32 any

Eth1/2

Eth1/1

N9K-A N9K-B

NetworkNetworkNetwork

10.214.10.5/24

Host A

10.214.50.11/24

Host BICMP Packet loss

Eth1/1

monitor session 1source interface eth1/1 rxdestination interface sup-eth 0filter access-group ACL1no shut

ip access-list ACL1permit icmp 10.214.10.5/32 any



23

Eth1/2 Eth1/1

N9K-A N9K-B

NetworkNetwork

10.214.10.5/24

Host A

10.214.50.11/24


N9K-A#ethanalyzer local interface inband mirror display-filter "icmp”Capturing on inband2018-12-12 04:41:32.164790 10.214.10.5 -> 10.214.50.11 ICMP Echo (ping) request2018-12-12 04:41:32.165562 10.214.10.5 -> 10.214.50.11 ICMP Echo (ping) request2018-12-12 04:41:32.166266 10.214.10.5 -> 10.214.50.11 ICMP Echo (ping) request2018-12-12 04:41:32.166930 10.214.10.5 -> 10.214.50.11 ICMP Echo (ping) request2018-12-12 04:41:34.167589 10.214.10.5 -> 10.214.50.11 ICMP Echo (ping) request

Eth1/1

Network

Differentiates the packets normally sent to CPU vs. SPAN to CPU packets!!



24

N9K-B#ethanalyzer local interface inband mirror display-filter "icmp”Capturing on inband2018-12-12 04:41:32.164982 10.214.10.5 -> 10.214.50.11 ICMP Echo (ping) request2018-12-12 04:41:32.165941 10.214.10.5 -> 10.214.50.11 ICMP Echo (ping) request2018-12-12 04:41:32.166611 10.214.10.5 -> 10.214.50.11 ICMP Echo (ping) request2018-12-12 04:41:34.167992 10.214.10.5 -> 10.214.50.11 ICMP Echo (ping) request

Eth1/2 Eth1/1

N9K-A N9K-B

NetworkNetwork

10.214.10.5/24

Host A

10.214.50.11/24


Eth1/1

Network


SPAN to CPUGuidelines and Limitations

25

• No license required for SPAN

• All SPAN replication is done in the hardware, with no impact to CPU

• SPAN packets to CPU are rate-limited, and excess packets are dropped in the inband path. Use “hardware rate-limiter span …” command to change the rate.

• Nexus 3232C and 3264Q do NOT support SPAN to CPU

• SPAN packets truncation is supported only in Nexus 9300-EX/FX platforms

• SPAN is not supported for management ports



26


• SPAN to CPU



CatenaIntroduction

27

Hardware-based multi-terabit application chaining solution, which redirects traffic through multiple L4-L7 physical/virtual devices, without really changing the topology or the existing configuration. Natively generate Telemetry and Analytics on the switch.

Network

Network Functions…. e.g., Firewall, IPS/IDS, Load-balancer, NAT/PAT, Applications


Appliance 3Appliance 1 Appliance 2

CatenaChains

28

Eth1/1

Eth2/2Eth2/1 Eth2/3

Nexus

Traffic with Catena

Traffic without Catena

Eth1/9

Blocked with Catena


Firewall IDS/IPS

Eth1/1

Eth1/2 Eth1/3 Eth1/4 Eth1/5

Nexus

Vlan 10 Vlan 20 Vlan 30

Eth1/8

Catena DeploymentTransparent & Routed Mode

29

Eth1/1 Eth1/8Nexus

1.1.1.1 2.1.1.1 20.1.1.1

…Appliance1 Appliance2 Appliance20

Traffic without Catena

Blocked Traffic with Catena

Traffic with Catena


RR RR

Host AMAC_A

10.1.10.10

Host BMAC_B

10.1.20.20

Traffic with Catena

Blocked Traffic with Catena

VxLAN Encap Packet

Traffic with CatenaVXLAN Fabric

30

ASA Firewall

APP Firewall

VXLAN Overlay with BGP-EVPN

Leaf1 Leaf2 Leaf3

Spine1 Spine2


Catena ConfigurationSteps

31

• Enable feature

N9k(config)# feature catena

• Create port group and add interfaces to it

N9k(config)# catena port-group PG1

N9k(config-port-group)# interface eth2/2

N9k(config-port-group)# interface eth2/3

• Create VLAN group and add VLAN(s) to it

N9k(config)# catena vlan-group VG1

N9k(config-vlan-group)# vlan 10

N9k(config-vlan-group)# vlan 31-40, 45

• Create device group and add nodes to it

N9k(config)# catena device-group DG1

N9k(config-device-group)# node ip 1.1.1.1


N9k(config-device-group)# probe icmp

Alternate Probing Options:Link state (transparent mode)

TCP/UDP Port #DNSHTTP


Catena Configuration (contd.)Steps

32

• Create access-lists as required

• Create Catena Instance and Configure Chains

N9k(config)# catena instance1

N9k(config-catena-instance)#chain 10

<sequence-no> access-list <ACL> {vlan-group <VG> | ingress-port-group <PG>} {egress-port-group <PG> | egress-device-group <DG>} [ mode <forward | drop | bypass> ][ load-balance {algo-based {src-ip | dst-ip} | ecmp | port-channel} ]

More Options:reverse-port-group

reverse-device-group

TCAM-based load-balancing


Catena ConfigurationsTransparent Mode VACL

33

N9k(config)# catena port-group pg1

N9k(config-port-group)# interface Eth 3/1



N9k(config)# catena vlan-group vg1


N9k(config)# catena vlan-group vg2


N9k(config)# ip access-list acl10

N9k(config-acl)# 10 permit 10.1.1.0/24 any


N9k(config-catena-instance)# chain 10

N9k(config-catena)# 10 access-list acl10 vlan-group vg1 egress-port-group pg1 mode forward

N9k(config-catena)# 20 access-list acl10 vlan-group vg2 egress-port-group pg2 mode forward

N9k(config-catena-instance)# no shut

Firewall

Eth3/1 Eth3/2

N9k

Vlan 10Vlan 20

Vlan 30

IDS/IPS

Have a backup firewall?Add it to the port-group!!


Catena ConfigurationsTransparent Mode PACL

34











Firewall

Eth3/2 Eth3/3

N9k

Vlan 20

IDS/IPS



N9k(config-catena)# 10 access-list acl11 ingress-port-group pg1 egress-port-group pg2 mode forward

N9k(config-catena)# 20 access-list acl11 ingress-port-group pg3 egress-port-group pg4 mode forward


Eth3/1 Eth3/4 Eth3/5 Eth3/6


Catena ConfigurationsRouted Mode

35

App1

e3/2

N9k

App2

e3/3 e3/4 e3/5 e3/6e3/1





N9k(config)# catena device-group dg1



N9k(config)# catena device-group dg2







N9k(config-catena)# 10 access-list acl25 ingress-port-group pg1 egress-device-group dg1 mode forward

N9k(config-catena)# 20 access-list acl25 ingress-port-group pg2 egress-device-group dg2 mode forward


1.1.1.1 3.3.3.3


Catena AnalyticsCommands

36

N9k# show running-config catena

feature catena

catena vlan-group vg1

vlan 801

catena vlan-group vg2

vlan 802

catena device-group dg1

node 3.3.3.3

node 4.4.4.4

catena port-group pg1

interface Eth1/3


interface Eth1/4


interface Eth1/5

catena INS1

chain 10

10 access-list ACL10 vlan-group vg1 egress-port-group pg1 mode forward

20 access-list ACL10 vlan-group vg2 egress-port-group pg2 mode forward

chain 20

30 access-list ACL10 ingress-port-group pg3 egress-device-group dg1 mode forward

no shutdown

N9k# show catena INS1

--------------------------

Instance name Status

------------- ------

INS1 ACTIVE

--------------------------

chain 10

-----------------------------------------------------------------

sequence no access-list vlan-group egress-port-group mode

-----------------------------------------------------------------

10 ACL10 vg1 pg1 forward

20 ACL10 vg2 pg2 forward


Catena Analytics (Contd.)Commands

37

N9k# show catena analytics per-acl per-node

-----------------------------

Instance name : INS1

-----------------------------

Chain : 20

-------------------------------------------

Seqno Node #Packets

-------------------------------------------

30 dg1 62135

Total packets per-Node for all chains

========================================

Node Total Packets

========================================

dg1 62135

N9k# show catena analytics per-acl per-device-group

-----------------------------


-----------------------------

Device Group : dg1

--------------------------------------------------------

Element ACL Name Chain ID #Packets

--------------------------------------------------------

3.3.3.3 ACL10 20 24859

4.4.4.4 ACL10 20 37288

Total Count for dg1 : 62147

Total Count for ACL ACL10 : 62147

Total Count for Element 3.3.3.3 : 24859

Total Count for Element 4.4.4.4 : 37288

N9k# show catena analytics per-catena-ins INS1 per-chain 10

-----------------------------


-----------------------------

Chain : 10

<snip>


Catena – Benefits

• Insert/remove Network Functions as and when needed – build an elastic data center

• Securely segments / partitions traffic. Supports Multi-tenancy

• Multiple chains with same network elements

• Manageable via CLI, NX-API, XML or custom Apps

• Highly-granular telemetry

Flexibility• Special hardware required• Hardware dependency• Requirement of controllers• Proprietary packet headers• Load to the Supervisor

engine’s CPU• Added latency to traffic• Interoperability certification

or validation required• Need to stitch vlans or

create L3 default gateways.• Need to bring the instance

down to update config

List of NOs….


CatenaGuidelines and Limitations

39

• Requires Network Services License.

• Routed mode must have Policy-based Routing (PBR) and Service Level Agreements (SLA) features enabled

• For a given a port-group or vlan-group, only one instance is supported

• No IPv6 support as of now

• In Nexus9000 platforms, Catena is supported in Nexus 9200, 9300 and 9300-EX Cloud-scale switches

• Supported from 7.0(3)I6(1) onwards, and also in 9.x releases.


Built-in CLIs & ToolsMore to Leverage

40

• Consistency Checkers

• Packet Tracer Tool

• Active Latency Monitoring, Active Buffer Monitoring and Microburst Monitoring

• Flexible Netflow / sFlow

• iCAM* – provides visibility into which network traffic or applications utilize system’s TCAM/SRAM resources

• Pervasive Load Balancing (PLB)* – enables load-balancing to the servers/appliances while generate massive telemetry and analytics

* iCAM and PLB requires NETWORK SERVICES license


Source: Superscience Scholastic

NX-OS Programmability


On-the-Box

EEM

Scheduler

Bash

vi Editor

Python/Tcl

BCM Shell

Telemetry

Nexus 3000/9000 Programmability FeaturesSupported Options

43

Off-the-Box

Expect/Tcl

REST/NX-API

Python API

Container

Guest-shell

LXC

Kernel Stack

Docker

Mgmt/App

Puppet

Chef

Ansible

XMPP

Yocto

Cisco SDK

NX-SDK

List here is not exhaustive !!


NX-OS ProgrammabilityTopics

44

• Software Telemetry

• Hardware Telemetry

• Insights with Telemetry


Software TelemetryComponents and Process

45

Enable Telemetry feature and configure the device …

• What to send?

• How to send?

• How often to send?

• Where to send?

Sensor Group (E.g. QoS, BGP)

Encoding format (JSON, GPB) and Transport method (HTTP, gRPC, UDP)

Subscription

IPv4/v6 Address and Layer4 port of the Collector

Nexus

Telemetry Source

CollectorTelemetry Data

GPB – Google Packet Buffers

gRPC – Google Remote Procedure Call


!

feature telemetry

!

telemetry

destination-group 1

ip address 172.18.121.123 port 50001 protocol gRPC encoding GPB

sensor-group 1

path sys/intf/phys-[eth2/1]/dbgIfIn depth 0

subscription 1

dst-grp 1

snsr-grp 1 sample-interval 5000

!

Enable feature

Where to send?

What to send? … provide Sensor Path*

How often? (in milliseconds)

Software TelemetryTelemetry Source - Configuration

46

*Reference: Data Management Engine - Resource Path

How deep in the sensor path?

How to send?

https://developer.cisco.com/site/nxapi-dme-model-reference-api/


Programmability InfrastructureOverview

47

• Managed Objects (MO) in the Object Store

• All elements of the MOs are accessible –Config, Faults, Events, Operational Data and Statistics.

• Features supported*: BGP, VLAN, LACP, ACL, QoS, UDLD, MAC, DHCP, DNS, RBAC, AAA, SVI, NTP and VRRP.

NGINXServer

PythonAPI TCL Bash Netconf

NX-APIClient

Netconf Client

RESTClient

SNMPServer

CLI (VSH)

Data Management Engine (DME)

SNMPAgent

* check Release-notes for an updated list

Telemetry

Protocols / Features

Collector

Object Store

BGP VLAN QoS ACL LACP …

NX-OS


Telemetry Data CollectionProcess Flow for Data-source NX-API

48

NGINXServer

NX-APIClient

CLI (VSH)


Telemetry


Collector

Object Store


NX-OS

sensor-group 3data-source NX-APIpath “show processes cpu” depth unbounded

This method have scale limitations. Avoid commands that take 15sec or more.

Using DME Resource Path


Telemetry Data CollectionProcess Flow

49

1. Model-Driven Telemetry collects data for various resources in the Sensor Path

2. DME validates request against supported models and respond to the client

3. Transactions committed to the backend processes in NX-OS

4. Report data and status (success / failure)

Telemetry

Object Store


1 2

Data Management Engine(DME)

34

NX-OS

Collector


Software TelemetryGuidelines and Limitations

50

• No licenses required.

• Telemetry application can be upgraded or downgraded

• Up to five receivers supported. IPv6 receiver supported from 9.2(1) onwards.

• Not supported in switches with less than 8GB memory

• Telemetry can consume up to 20% of CPU resources

• Configuration and Streaming services are restored on System Reload, Supervisor Failover or Process Restart.

• Compression and Chunking supported for gRPC transport



51




Flow Tables (FT)Streaming Statistics

Export (SSX)

Flow Table Events (FTE)

Streams ASIC stats Streams complete data-plane flow info

with metadata

Issues notifications when data-plane flowmeets user-defined

criteria


Streaming Statistics Export (SSX)Statistics from ASICs

52

• Reads statistics from ASICs and send them to Collector for analysis

• Packet headers are programmable through software, as defined by user

• Each header is followed by metadata, which contains info like PTP, Sequence number etc.

• Payload formatted into TLV (Type, Length and Value) data

• Sent only through front—panel ports, as UDP packets, directly from ASIC.

L2/IP/UDP Header

MetadataPayload

(TLV format)

PTP timestamp (64 bits)Seq_num (32)System (16)Board (8)ASIC (4)Ver (4)[TOTAL=16 Bytes]

Type (6 Bytes):Group (8)Block (8)Index (32)

Length (2 Bytes):Length of data

Value:Variable payload


Streaming Statistics Export (SSX)Configure and Apply

53

N9k(config)# feature hardware-telemetry

N9k(config)# hardware-telemetry ssx

ssx exporter EXP1

source 1.2.3.4

destination 5.5.5.5 use-vrf default

transport udp src-port 2000 dst-port 3000

mtu 1500

dscp 0

ssx record REC1

collect egress queue depth

collect ingress queue drops

interval 100

ssx monitor MON1

exporter EXP1

record REC1

Available stats:

Egress Buffer depth

Egress pool-group depth

Egress queue depth

Egress queue drops

Egress queue micro-burst

Egress queue peak

Ingress queue depth

Ingress queue drops

Ethernet counters

in msec N9K(config)# ssx system system-id 11

N9K(config)# ssx system monitor MON1

Applying SSX


Flow Table Events (FTE)Generating Criteria-based Notifications

54

• In NXOS 9.2(1) and later releases, ASIC Flow Tables (FT) can generate notifications/events when specific criteria detected in the packets.

• These events are stored in FIFO fashion in the FT.

• Once threshold reached, events in the FT are exported to the collector as UDP packets.


Flow Table Events (FTE)Configure and Apply

55

N9k(config)# feature hardware-telemetry

N9k(config)#hardware-telemetry fte

fte exporter EXP1

source 1.2.3.4

destination 5.5.5.5 use-vrf default

transport udp source-port 2000

exporter-id 12

fte record REC1

match ipv4 protocol

match ipv4 transport source-port

match datalink ethertype

fte event EVENT1

group drop-events

capture acl-drops

capture buffer-drops

capture fwd-drops

flow-count 250

group latency-events

capture latency exceeding-thr 100 micro-sec

fte monitor MON1

record REC1

exporter EXP1

event EVENT1

N9K(config)# fte system monitor MON1also supports IPv6

threshold

system-wide


Hardware TelemetryGuidelines and Limitations

56

• As of now, SSX supports exporting the data only in the default VRF. FTE supports exporting the data in any VRF.

• Netflow and FTE cannot be enabled at the same time

• FTE exports are asynchronous as it is packet-driven, and supports only UDP transport. SSX supports only UDP transport.

• As FTE exports are hardware-driven, it can overwhelm collectors. Deploy appropriate throttling mechanism.

• SSX supported in N9364C and N9300-FX2 platforms. FTE supported in N9300-FX and N9300-FX2 platforms.



57





Insights with TelemetryArchitecture and Functions

58

Collection

Storage

Applications

Architecture

Ingest Aggregate Normalize

Store Index

Search

Visualize Alert / Notify

Automate

Main FunctionsTelemetry Data

Collector

Build and customize an open-source stack!

Commercial / Turnkey solutions available !!

Hosted Version too…

Build your own!!

Prometheus

Let me show how I built an open-source stack!


telemetry destination-group 1

ip address 172.18.121.123 port 50001 protocol gRPC encoding GPBsensor-group 1

path sys/intf/phys-[eth1/1]/dbgIfIn depth 0 path sys/intf/phys-[eth1/2]/dbgIfIn depth 0 path sys/intf/phys-[eth1/1]/dbgIfOut depth 0 path sys/intf/phys-[eth1/2]/dbgIfOut depth 0

sensor-group 2 path sys/intf/phys-[eth1/1]/phys depth 0path sys/intf/phys-[eth1/2]/phys depth 0

subscription 1 dst-grp 1 snsr-grp 1 sample-interval 5000

subscription 2 dst-grp 1 snsr-grp 2 sample-interval 5000

Eth1/1 Eth1/2

Eth1/43Network

172.18.121.123 Telemetry Receiver

Nexus9000

Insights with TelemetryExample – Interface Statistics and Status Changes

59

Sensory Path to Interfaces’ Ingress/Egress statistics


ELK Open-source Stack

Elastic Search

Telemetry Data

Telemetry AnalyticsOpen-Source Stack

60

Steps to build ELK Open-source Collector:

1. In a Ubuntu Linux, pull the telemetry receiver from Cisco Docker hub, and run.

$docker pull dockercisco/telemetryreceiver:latest

$sudo docker run -p 50001:50001 -it c7b476917147 bash

2. Start the telemetry receiver application

$./telemetry_receiver 50001 172.18.121.123 9200 1

3. Docker Pull ELK Stack, and run.

$docker pull dockercisco/elklat

$docker run -p 5601:5601 -p 9200:9200 -it 02ae097bd96d bash

4. Start ElasticSearch and Kibana

$service elasticsearch start $service kibana startCollector

172.18.121.123 Refer Cisco Telemetry Receiver - Docker Container

Telemetry Receiver

https://hub.docker.com/r/dockercisco/telemetryreceiver/


Telemetry – Visualization with Kibana

61

when traffic rate

changed

filtering for Operation State = UP


Telemetry – Visualization with Kibana

when traffic rate

changed

filtering for Operation State = UP

62


DCNM Version 10.4(2)

Nexus 9000 running 7.0(3)I6(1) or later

Telemetry streamed by the switches

Enable / Disable the streaming

63

Software TelemetryDCNM – A Simple Turnkey Receiver


Choose the parameter to explore

Choose one or more switches to explore

64

Software TelemetryDCNM – A Simple Turnkey Receiver


nfm-9300-leaf-1b# show running-config | section telemetryfeature telemetrytelemetry

destination-group 1ip address <DCNM-Server-IP> port 57500 protocol gRPC encoding GPB

sensor-group 300data-source NX-APIpath “show processes cpu” depth unboundedpath “show system resources” depth unbounded

sensor-group 301data-source NX-APIpath “show environment fan detail” depth unboundedpath “show environment power” depth unbounded

subscription 1dst-grp 1snsr-grp 300 sample-interval 30000snsr-grp 301 sample-interval 300000

Software TelemetryStreaming Data using DCNM – Switch Config

DCNM Version 10.4(2)

Making telemetry to use CLIs instead of DME Resource Path

CLIs provided

65


Telemetry Data CollectionProcess Flow for Data-source NX-API

66

NGINXServer

NX-APIClient

CLI (VSH)


Telemetry


Collector

Object Store


NX-OS

Using NX-API CLI. Have scale limitations. Avoid commands that take 15sec or more.

Using DME Resource Path


Are you ready?

67


Get More Visibility

…. and ….

Insights

Use-cases


Use-cases• Inventory Management

• Hardware Uptime Check

• Scalability Check

• Control-plane Health Check

• Configuration Consistency Check

• Traffic Profiling and Top-Talkers

• Tracking End-hosts Mobility

70

Leverage Docker Container capability in Nexus9000 and run a Python application to identify unexpected traffic sent to the CPU


Virtual Machines and Containers

71

App A

Bins/Libs

Hypervisor

Host OS

Server

VM

Host OS

Server

Bins/Libs Bins/LibsCo

nta

ine

rC

on

tro

l

App A App BContainer

Containers provide a way to run

applications in a securely isolated

environment, with all dependencies

and libraries packaged.

Guest OS

App B

Bins/Libs

Guest OS

Containers = Lightweight Virtualization


Container Network Models

72

Host platformNetwork namespace: Host

Container interfaces

eth0 eth1 eth2

Physical interfaces

eth0 eth1 eth2

SharedApplications inside the container appear as applications

running natively on the host

DedicatedApplications inside the container appear as appliances on a

subnet reachable from the host


eth0 eth1 eth2

Container 1Network namespace: Host

Container 2Network namespace: Host

Shared namespace:Interfaces are directly mapped to container

Examples: Nexus 3k, 9k, 6k, 7k, Cat 3k, 4k, NCS xK Examples: ASR 1k, CSR 1kv, ISR4k, ISR 819

Host platformNetwork namespace: Host


veth0 veth1 veth2


veth0 veth1 veth2

Container 1Network namespace: N1

Container 2Network namespace: N2

Physical interfaces

eth0 eth1 eth2

Multiple bridges and virtual topologies possible

Forwarding Plane

VPG

Linux Bridge


Nexus 9000 DockerIntroduction

• Supported from 9.2(1) release onwards, and on switches with 8GB+ RAM

• Docker version 1.13.1

• Pre-requisites:

• Enable Bash shell

• Set HTTP/HTTPS environments variables (if applicable)

• Make sure the switch system clock is in sync

• Make sure the switch domain name and DNS server IP are set correctly

73

Refer Configuring DNS in NX-OS Bash Shell for more information.

https://www.cisco.com/c/en/us/support/docs/switches/nexus-9000-series-switches/213959-nx-os-bash-shell-dns-configuration.html


Nexus 9000 DockerBuilding and Running a Containerized-Application - Example

74

Kernel

Physical interfaces

Eth1-1 Eth1-2 Eth1-N

Packets sent to the CPU

Docker Pull

GitHub

Floodlight(Dockerfile, Python & Requirements)

DockerHub

Floodlight(Containerized App)

Nexus9000

FloodlightNameSpace: Management

/startup-config/var/log/bootflash

Bash

Container

Python

eth1

Linux Host[Python Integrated

Development Environment]

12

3

4

Inband port

Control-Plane

Inband Channel

SupervisorNX-OS

...


Nexus 9000 DockerBuilding and Running a Containerized-Application - Example

75

1. Linux Python Integrated Development Environment (IDE) – Develop a Python code with core functions

2. GitHub - Build a Dockerfile, to set environment variables, install requirements and execute the Python code.

3. DockerHub - Build a containerized application.

4. Nexus9000 - In the Bash shell, under the management namespace, build a docker-compose file (e.g., docker-compose.yml) and execute.


Floodlight ApplicationGitHub Repository – Dockerfile & Requirements

76

Reference: Floodlight - GitHub Repository

GitHub

Floodlight(Dockerfile,

Requirements & Python)2

Dockerfile

Requirements

Execution !!

Application requirements

Required TCPDUMP and Tshark LibrariesOff latest Alpine

https://github.com/ChristopherJHart/floodlight/blob/master/Dockerfile


Floodlight ApplicationGitHub Repository – Python code

77

GitHub


Requirements & Python)

Used Scapy to read the packets in a PCAP file

Capture the traffic

Build the filters (based on the startup-config)

Sample: OSPF

Check for features enabled and configs

to build filters

Check the features enabled and configs to synthesize filters

2


Floodlight ApplicationGitHub Repository – Python code

78

GitHub


Requirements & Python)

Filter the packets for traffic not expected to be at the CPU, and summarize

Apply the filter synthesized, to identify the packets that are

not expected to be at the CPU

2


Floodlight ApplicationDockerHub Repository

79

DockerHub

Floodlight(Containerized App)

Containerized App is available at:

DockerHub - Floodlight Repository

Connected to GitHub repository

3

https://cloud.docker.com/repository/docker/chrisjhart/floodlight


Nexus9000

Floodlight ApplicationNexus9000 – Bash Shell and Docker

80

N93180(config)# feature bash-shell

N93180(config)# end

N93180# run bash sudo su –

root@N9380#

root@N9380#ip netns exec management bash

Bash

Floodlight

root@N93180#cd floodlight/

root@N93180#ls -l

-rw-r--r-- 1 root root 316 Jan 16 14:24 docker-compose.yml

root@N93180#

root@N93180#

root@N93180#more docker-compose.yml

version: "3"

services:

floodlight:

image: chrisjhart/floodlight:latest

container_name: floodlight

volumes:

- /var/sysmgr/startup-cfg/ascii/system.cfg:/startup-config

- /var/log/:/var/log/

- /bootflash:/bootflash

environment:

- DEBUG=1

- EXPORT=/bootflash/example_pcap.pcap

network_mode: "host"

root@N93180#

enable Bash shell

Namespace: Management

Make sure it has internet connectivity, if DockerHub is used

pulls the latest image from the dockerhub

Mount the required volumes

Container runs in “host” mode

run Bash shell

4

Reference: Installing Docker Compose in NX-

OS Bash Shell

https://www.cisco.com/c/en/us/support/docs/switches/nexus-9000-series-switches/213961-install-docker-compose-in-nx-os-bash-she.html


Nexus9000

Floodlight ApplicationNexus9000 – Bash Shell and Docker

81

Bash

Floodlight

root@N93180# docker-compose up

Starting floodlight ... done

Attaching to floodlight

floodlight INFO [LOG] Debug logging level set!

floodlight INFO [SETUP] NX-OS startup-config file detected

floodlight INFO [FILTER] OSPF feature and configuration found!

floodlight INFO [FILTER] HSRP configuration not found, skipping...

<snip>

floodlight INFO ==== FILTERS ====

floodlight | ‘ip': ['224.0.0.5', '224.0.0.6'],

floodlight | 'ip_protocol_type': ['89’],

<snip>

floodlight | 'protocols': ['OSPF', 'BGP', 'Spanning Tree Protocol', 'SSH',

'CDP', 'LLDP']}

floodlight INFO [CAPTURE] Beginning packet capture, be back in 60 seconds...

floodlight INFO [CAPTURE] Packet capture finished! 259 packets in capture

floodlight INFO [UNEXPECTED] Number of unexpected packets: 138

floodlight INFO ===== RESULTS =====

floodlight INFO 14,879 bytes (123 packets) | TCP (TCP )

00:01:02:03:04:05 10.150.53.63:50449 -> 10.122.53.229:2345 00:de:fb:fa:64:c7

<snip>

floodlight INFO [WRITE-PCAP] Successfully wrote unexpected packets to PCAP

at /bootflash/example_pcap.pcap

floodlight exited with code 0

root@N93180#

4That’s your App !!

CLI executes docker-compose.yml

FloodlightNameSpace: Management

/startup-config/var/log/bootflash

Bash

Container

App

eth1

Docker Pull

DockerHub

Inband port

Control-Plane

Packet sent to the CPU

Nexus9000


Use-cases• Inventory Management

• Hardware Uptime Check

• Scalability Check

• Control-plane Health Check

• Configuration Consistency Check

• Traffic Profiling and Top-Talkers

• Tracking End-hosts Mobility

82

In NX-OS SDK environment, develop a custom Python application and install it

in Nexus9000 using VSH capability.

Learn to build a custom NX-OS CLI !!


NXOS Software Development Kit (SDK)Introduction

• Simple, flexible and powerful tool for custom on-the-box applications to gain access to NX-OS infrastructure

• Languages supported: Python or C++

• Run natively. Startup and management handled by NX-OS

• Define your own custom CLIs, syslogs, events and more

• Supported from 7.0(3)I6(1) onwards

NXOS Infra SDK Abstraction Layer Library

HACLIs

Event Manager

Syslog, Events & Faults

DME etc.

NX-OS

Linux

Nexus9000

Custom Applications(Python, C++)

Native NX-OS Applications

83


NXOS Software Development Kit (SDK)End-to-End Process Flow

BASH:Python SDK Environment not needed.C++ SDK Environment is optional, but recommended.VSH:SDK environment is mandatory. Apps must be built as an RPM package, and installed as a package.

Building an application

using programming languages*

(1) Docker pull of ENXOS SDK Build environment (2) Start/Run Docker container.(3) [optional] Upgrade/Downgrade NX-SDK using git pull or git clone(4) For C++, add the application to the Makefile and make sure builds are

error-free. For Python, nothing is required.

Setting up the ENXOS SDK Environment

* Currently Python and C++ supported. Support for more languages in future road-map.

Generate RPM package, using built-in rpm_gen.py script.For complex applications, manually generate RPM package following steps provided here.

Packaging the application

84

https://github.com/CiscoDevNet/NX-SDK#b-manually-generate-rpm-package


NXOS Software Development Kit (SDK)End-to-End Process Flow

Copy the App (binary) or RPM package to the switch bootflash:VSH: Add package to the installer with install add bootflash:<file.rpm>and activate with install activate bootflash:<file.rpm> command.BASH:Install RPM package with yum install /bootflash/<file.rpm>

Installing the application in

Switch

VSH:Start the application with nxsdk service-name <app-name>. If App is installed at non-default location, then do nxsdk service-name <full-path/app-name>BASH:In the switch config, run bash sudo su, and then app-full-path &

Running the application in

Switch

85


Tracking End-Hosts MobilityBuilding Custom Application

Application: Track movement of an end-host

Switch: Nexus9000 C93180LC-EX

NX-OS: 7.0(3)I7(1)

Capability used: VSH

Language: Python

Procedure followed: 1) Build the application in a standalone host running CentOS 7

(which can also be done NXOS Bash Shell, using native Python capability)

2) Pull Docker container and setup NX-SDK environment3) Build RPM package of the Python App4) Transfer the RPM package to Nexus switch, install and activate5) Verify the Service 6) Use custom application’s CLI to track end-host(s)

Network

Nexus93180

Eth1/3

Eth1/4

Eth1/5

86


Tracking End-Hosts Mobility1) Building a Python application – CLIs

def get_mac_from_arp(cli_parser, clicmd, target_ip):

exec_cmd = "show ip arp {}".format(target_ip)

arp_cmd = cli_parser.execShowCmd(exec_cmd, nx_sdk_py.R_JSON)

if arp_cmd:

try:

arp_json = json.loads(arp_cmd)

except ValueError as exc:

return None

count = int(arp_json["TABLE_vrf"]["ROW_vrf"]["cnt-total"])

if count:

intf =arp_json["TABLE_vrf"]["ROW_vrf"]["TABLE_adj"]["ROW_adj"]

if intf.get("ip-addr-out") == target_ip:

target_mac = intf["mac"]

Check ARP and get

MAC-addr

def get_vlan_from_cam(cli_parser, clicmd, target_mac):

exec_cmd = "show mac address-table address {}".format(target_mac)

mac_cmd = cli_parser.execShowCmd(exec_cmd, nx_sdk_py.R_JSON)

if mac_cmd:

try:

cam_json = json.loads(mac_cmd)

except ValueError as exc:

return None

mac_entry = cam_json["TABLE_mac_address"]["ROW_mac_address"]

if mac_entry:

<snip>

From MAC entry, find the current Interface

87


Tracking End-Hosts Mobility1) Building a Python application – CLIs (cont’d)

def find_mac_movement(cli_parser, clicmd, target_mac, mac_vlan):

exec_cmd = "show system internal l2fm l2dbg macdb address {} vlan

{}".format(target_mac, mac_vlan)

l2fm_cmd = cli_parser.execShowCmd(exec_cmd)

if l2fm_cmd:

event_re = re.compile(r"^\s+(\w{3}) (\w{3}) (\d+) (\d{2}):(\d{2}):(\d{2})

(\d{4}) (0x\S{8}) (\d+)\s+(\S+) (\d+)\s+(\d+)\s+(\d+)")

unique_interfaces = []

l2fm_events = l2fm_cmd.splitlines()

for line in l2fm_events:

res = re.search(event_re, line)

if res:

day_name = res.group(1)

month = res.group(2)

day = res.group(3)

hour = res.group(4)

minute = res.group(5)

second = res.group(6)

year = res.group(7)

if_index = res.group(8)

db = res.group(9)

event = res.group(10)

src = res.group(11)

slot = res.group(12)

fe = res.group(13)

Check the end-host

movement

88


Tracking End-Hosts Mobility2-3) Setting up Docker environment and Building RPM package

2) Pull NX-SDK Docker container and run

[root@localhost ~]# yum -y install docker

[root@localhost ~]# docker pull dockercisco/nxsdk:v1

[root@localhost ~]# docker run –it dockercisco/nxsdk:v1 /bin/bash

root@b7d33ce8a7b8:/# cd /NX-SDK

root@b7d33ce8a7b8:/NX-SDK# git pull

3) Copy the Python App and build RPMroot@b7d33ce8a7b8:/# cd /root

root@b7d33ce8a7b8:~# mkdir nxsdk-scripts

root@b7d33ce8a7b8:~# cd nxsdk-scripts/

root@b7d33ce8a7b8:~# cp /bootflash/ip_move.py .

root@b7d33ce8a7b8:~/nxsdk-scripts# python /NX-SDK/scripts/rpm_gen.py ip_move.py –s

/root/nxsdk-scripts –u

<snip>

RPM package has been built

SPEC file: /nxsdk/rpm/SPECS/ip_move.py.spec

RPM file: /nxsdk/rpm/RPMS/ip_move.1.0-1.5.0.x86_64.rpm

89


Tracking End-Hosts Mobility4-5) Installing RPM in Nexus, Activate and Verify Service

5) Enable NX-SDK feature, activate and verify the service

C93180(config)# feature nxsdk

C93180(config)# nxsdk service-name ip_move.py

4) Move the RPM to Nexus, install and activate

C93180# copy ftp://<server>/ip_move.1.0-1.5.0.x86_64.rpm bootflash: vrf management

C93180# install add bootflash:ip_move.1.0-1.5.0.x86_64.rpm

C93180# install activate ip_move.1.0-1.5.0.x86_64

C93180# show nxsdk internal service

NXSDK Started/Temp unavailable/Max services : 0/0/32

NXSDK Default App Path : /isan/bin/nxsdk

NXSDK Supported Versions : 1.0

Service-name Base App Started(PID) Version RPM Package

--------------------- --------------- ------------ ------- ------------

/isan/bin/ip_move.py nxsdk_app1 VSH(28161) 1.0 ip_move.py-1.0-1.5.0.x86_64

90


Tracking End-Hosts Mobility6) Using the Service

C93180# show ip_move.py 20.20.20.3

20.20.20.3 is currently present in ARP table, MAC address 0010.9400.0002

0010.9400.0002 is currently present in MAC address table on interface Ethernet1/3, VLAN 20

0010.9400.0002 has been moving between the following interfaces, from most recent to least

recent:

Fri Apr 20 12:05:17 2018 - Ethernet1/3 (Current interface)

Fri Apr 20 12:04:13 2018 - Ethernet1/5

Fri Apr 20 12:04:13 2018 - Ethernet1/4

Fri Apr 20 12:03:50 2018 - Ethernet1/5

Fri Apr 20 12:03:50 2018 - Ethernet1/4

Fri Apr 20 12:03:26 2018 - Ethernet1/5

91

Summary


Things we have learnt today….

Capability Value

Precision Time Protocol / ESPAN / Nexus Data Broker

Monitor the latency in the network

SPAN to CPU Identify where the packets potentially are dropped

Catena Chain applications and network functions with minimal efforts, and get valuable insights

Model-driven or Streaming Telemetry Real-time network analytics, right from the hardware-level and all the way to control-plane

Bash and Docker Build and run your own App to automate day-to-day operations

Learn to build custom NX-OS CLI leveraging NX-SDK and VSH capabilities !!


Things we have learnt today….

Capability Value

Precision Time Protocol / ESPAN / Nexus Data Broker

Monitor the latency in the network

SPAN to CPU Identify where the packets potentially are dropped

Catena Chain applications and network functions with minimal efforts, and get valuable insights

Model-driven and Streaming Telemetry Real-time network analytics, right from the hardware-level and all the way to control-plane

Bash and Docker Build and run your own App to automate day-to-day operations

Learn to build custom NX-OS CLI leveraging NX-SDK and VSH capabilities !!


Take Aways …

Nexus3000/9000 have RICH SET OF CLIs and TOOLS that are developed keeping all of you in mind.

These platforms have several programmability features, and are very easy to use. YES, WE ARE OPEN!!

Cisco ENABLES AND EMPOWERS EACH ONE OF YOU to integrate them with your day-to-day operations, to get advanced visibility and insights.

95


Are you ready?

97


Get More Visibility

…. and ….

Insights


ReferencesIEEE 1588 PTP on Nexus 3100 and 9000 Series Switches White Paper

IEEE 1588 PTP and Analytics on the Cisco Nexus 3548 Switch

Latency Monitoring on Cisco Nexus Switches: Troubleshoot Network Latency

Catena – Configuration Guide

Nexus 9000 Programmability Guide

Nexus 3000 Programmability Guide

Cisco Nexus 3000/9000 NX-API REST SDK User Guide and API Reference

Cisco Telemetry Receiver - Docker Container

Nexus 3000/9000 Series Telemetry Sources

NX-SDK Use-case: Python Application at GitHub

Develop, Debug and Deploy NX-SDK Python Application in Nexus3K/9K Switches

Nexus 9000 GitHub Repository

98

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-733921.html

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-733921.html

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/white-paper-c11-733025.html

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus7000/sw/catena/config/cisco_nexus7000_catena_config_guide_8x/configuring_catena.html

http://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/products-programming-reference-guides-list.html

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/programmability/7_x/b_Cisco_Nexus_3000_Series_NX-OS_Programmability_Guide_7x.html

https://developer.cisco.com/docs/nx-os-n3k-n9k-api-ref/

https://hub.docker.com/r/dockercisco/telemetryreceiver/

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/programmability/telemetry/b_N9K_Telemetry_Sources.html

https://github.com/ChristopherJHart/NX-SDK

https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/nx-os-software/213396-develop-debug-and-deploy-nx-sdk-python.html

https://github.com/datacenter/nexus9000


Cisco Webex Teams

Questions? Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session

Find this session in the Cisco Events Mobile App

Click “Join the Discussion”

Install Webex Teams or go directly to the team space

Enter messages/questions in the team space

How

1

2

3

4

99

cs.co/ciscolivebot#BRKDCN-3020


Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at ciscolive.cisco.com

• Please complete your Online Session Survey after each session

• Complete 4 Session Surveys & the Overall Conference Survey (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Events Mobile App or the Communication Stations

Complete your online session survey

100


Demos in the Cisco Showcase

Walk-in self-paced

labs

Meet the engineer

1:1 meetings

Related sessions

Continue Your Education

101

Thank you

Advanced Visibility and Monitoring in Nexus 3000 ... - Cisco Live

Documents

Transcript of Advanced Visibility and Monitoring in Nexus 3000 ... - Cisco Live