BRKDGT-2601 From Theory to Reality - Cisco Live

From Theory to RealityA look at what impacts network design and deployment bliss, focusing on the Operational Technology space

Michael Boland, Distinguished Systems Engineer

BRKDGT-2601

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4BRKDGT-2601

“We have all been there. The network design looked great on PowerPoint or in the

design guide, but once implemented, unforeseen issues arose that challenged the

design. OK, sometimes they even torpedoed it! If only I knew this before I

implemented my design! Why was I not told?

Well, here is your chance. This session explores examples of different challenges,

with answers, to contemporary network design in the Operational Technology

space which are often overlooked in design guide theory.

This session is targeted at network architects and designers working in the

industrial and IT/OT convergence space. However, the topics covered can be

easily extrapolated into other network design areas, so in-depth knowledge of

industrial systems is not essential for attendees.”

Abstract


Key

Layer 2 Switch

Layer 3 Switch

PLC

Remote I/O

SDA Control Plane Nodes – Map System that manages Endpoint to Device relationships

SDA Fabric Edge Nodes – A Fabric device (e.g. Access or Distribution) that connects Wired Endpoints to the SDA Fabric

SDA Fabric Border Node – A Fabric device (e.g. Core) that connects External Layer 3 network(s) to the SDA Fabric

HMI

Surveillance Camera Router

SDA Fabric Enabled Wireless

Smart Lighting

6© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

• An Elegant Design

• Limitations

• When Two Worlds Collide

• Don’t Mention The War!

• Active Monitoring

• Designing with Fabrics

• Closing Words

Agenda

BRKDGT-2601

Attempting Elegant Design

… For an Inelegant World

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8

Core

Distribution

Access

Control

“My Precious!”

Datacentre


Core

Distribution

Access

Control

“My Precious!”

Datacentre

Limitations

10


Control Systems Limitations

11BRKIOT-2130

L3 → L2

• Legacy non-routable Layer 2

control protocols

• Poor support for multicast routing

by control systems e.g. IP TTL = 1

Single → Multiple Interfaces

• Bandwidth constrained Ethernet

ports may drive requirement for

multiple network interfaces

• Separate critical I/O from non-

critical network traffic

Core

Distribution

Access

Control

PLC PLCPLC


Switching Systems Limitations

12BRKIOT-2130

• No support for Encapsulated

Remote Switched Port ANalyzer

function (ERSPAN = SPAN over

GREoIP) on Cisco IE2000, IE3000,

IE4000, Rockwell Stratix switches

• RSPAN to VLAN is supported

• Impact – we must extend Layer 2 to

the closest upstream switches that

are capable of hosting network

analysis applications, e.g. Wireshark

• Alternative – distributed packet

capture appliances = cost &

complexity

Core

Distribution

Access

Control

PLC PLC

?

SPAN to

VLAN


Switching Systems Limitations

13BRKIOT-2130

• Cisco IE4000 switches support

multiple ring technologies but do

not support ODVA DLR (Device

Level Ring) networks

• Rockwell Stratix switches support

DLR but do not support NetFlow

or Scalable Group Tags (SGTs)

• What if I need DLR + SGTs +

NetFlow?

• Dual switch deployment!

Core

Distribution

Access

Control

PLC PLC

I/O I/O

DLR

SCADA,

Mgmt.

Traffic

IE4000

Stratix

5400

Zone /

Cell

SGT,

Netflow


Organisational Demarcation Limitations

14BRKIOT-2130

• Separate domains of control

• Production equipment enclosures

may be off-limits to non-Electrical

Engineering staff (plant safety

protocols)

• Different levels of network

knowledge/capability drives

network domain design:

o Electrical Engineering - Layer

2-centric, simpler designs,

minimal/no security, typically

managed via PLC control

systems

Core

Distribution

Access

ControlElectrical

Engineering

Domain

Network

Management

Domain


Network Capability Limitations

15BRKIOT-2130

• Rapid convergence requirement

exceeds standard network

convergence time capabilities

• Demands:

o Deploying rapid-convergence

ring technologies, e.g. DLR,

PROFINET RT/IRT or parallel

network technologies, e.g.

PRP

o Extending fibre for the Control

layer directly across the plant

to support rapid convergence

Core

Distribution

Access

Control

PLC PLC

Interlock

< 20ms

Convergence


Leaf-Spine Architectures are in Play (for Some)

16BRKIOT-2130

• Leveraging Datacentre architectures for

Plant control networking

• Heavy fibre deployment modelo Structured cabling/Blown-Fibre

o Hub-and-spoke access connections over

physical ring topology/paths

• Driven by:o Simplification of network design

o Datacentre automation/orchestration

o Layer 2 production networks

• Attention to detail:o Leaf-Spine distance limitations?

o Ring technology support e.g. REP?

o Management systems suitability for OT?

Core

Distribution

Access

Control

PLC PLC

PLC

IEEE

802.1Q

Trunks

WhenWorlds Collide


Probe Etiquette

18BRKIOT-2130

IP Device Tracking (IPDT) from IOS

15.2(1)E globally enabled, however only

becomes active when when dependent

features are enabled on a specific interface:

• Network Mobility Services Protocol (NMSP), Versions

3.2.0E, 15.2(1)E, 3.5.0E and later

• Device sensor, Versions 15.2(1)E, 3.5.0E and later

• 802.1x, MAC Authentication Bypass (MAB), session

manager

• Web-based authentication

• Auth-proxy

• IP Services Gateway (IPSG) for static hosts

• Flexible netflow

• Cisco TrustSec (CTS)

• Media trace

• HTTP redirects

Images Source: http://www.nasa.gov/sites/default/files/images

IP Device Tracking (IPDT) allows a Cisco switch to keep track of connected hosts (association of MAC and IP

address.) It achieves this by sending unicast ARP probes with a sender IP address of 0.0.0.0 as per RFC 5227


Some PLCs are known

to completely fail at this

point!

Probe Etiquette

19BRKIOT-2130

Cisco Switch

MAC=SWMAC

Control Equipment (CE)

IP=CEIP

MAC=CEMACARP Request (IPDT probe)

(Src IP=0.0.0.0, src MAC=SWMAC, tgt IP=CEIP, tgt MAC=CEMAC)

ARP Reply

(Src IP=CEIP, src MAC=CEMAC, tgt IP=0.0.0.0, tgt MAC=SWMAC)

DHCP Request

DHCPServer

DHCP ACK

ARP Request (IPDT probe)

(Src IP=0.0.0.0, src MAC=SWMAC, tgt IP=CEIP, tgt MAC=CEMAC)

ARP Request (ACD – Address Collision Detection - probe)

(Src IP=0.0.0.0, src MAC=CEMAC, tgt IP=CEIP, tgt MAC=0)

DHCP Decline

If the switch sends out

an ARP Probe while

the Control Equipment

is in its duplicate-

address detection

phase. CE detects the

probe as a duplicate

IP address of “0.0.0.0”

Please Refer to IP Device Tracking (IPDT) Overview for Remediation Methods

https://www.cisco.com/c/en/us/support/docs/ip/address-resolution-protocol-arp/118630-technote-ipdt-00.html

IPDT “Keepalive ARP Probe”

Some PLCs are known

to completely fail at this

point!


IPDT Probe Span - Any Upstream Switch @ Layer 2

20BRKIOT-2130

• IPDT probes can be initiated by

any IPDT capable switch within a

Layer 2 switching path

• Mitigation methods:

1) Set an IPDT probe delay

2) Set a defined ARP request

source IP address

3) Disable duplicate IP address

detection on the client

Core

Distribution

Access

Control

PLC

Any IPDT

capable switch

within the same

Layer 2 domain

can potentially

initiate an ARP

Probe to an

attached device

within the L2

Domain

Troubleshoot "Duplicate IP Address 0.0.0.0" Error Messages

https://www.cisco.com/c/en/us/support/docs/ios-nx-os-

software/8021x/116529-problemsolution-product-00.html

Don’t Mentionthe War !


Don’t Mention the War!

22BRKIOT-2130

OT comes from Mars

Images Source: http://www.nasa.gov/sites/default/files/images

IT comes from Venus


Overcoming Separate Domains of Control

23BRKDGT-2601

Core

Distribution

Access

Control

Datacentre

Maintain

HMISwitch

Switch

Switch

IO

PLC

Industrial Network

Director

Maintain

Prime Network

Infrastructure

Security

Identity Services

Engine

Security

HMISwitch

Switch

Switch

IO

PLC

Industrial Network

Director


OT user intent driven policy updatesPutting OT in the driver’s seat

Cell-1

OT User

Tag

assets as

Cell-1

ISE

Industrial Network Director

Topology UI

pxGrid

Update

PxGrid attribute “Cell-

1” matches profiling

policy-X and triggers

Authorization policy-Y

SGT

dACL

VLAN

N E W

N E W

N E W

OT personnel use with IND UI to express intent pxGrid update results in automatic policy update

IT manages ISE. OT uses IND to express intent to influence the IT owned

Security Policy

PLC

Switch

Port

Industrial

Network

Director

AlternateProductionWorkflowDemoDan Kirkwood

Virtual Systems Engineer


Maintaining Secure Access without the Headache

IT Wants

• Visibility over all devices

• Whitelist model

OT Wants

• Uptime

• Efficiency

Raise a ticket?

Training?

Give control

within Workflows

leveraging APIs

Active Monitoring


Flow Information Packets

SOURCE ADDRESS 10.1.8.3

DESTINATION ADDRESS 172.168.134.2

SOURCE PORT 47321

DESTINATION PORT 443

INTERFACE Gi0/0/0

IP TOS 0x00

IP PROTOCOL 6

NEXT HOP 172.168.25.1

TCP FLAGS 0x1A

SOURCE SGT 100

: :

APPLICATION NAME NBAR SECURE-HTTP

Routers

Switches

10.1.8.3

172.168.134.2Internet

NetFlow Provides

• A trace of every conversation in your network

• An ability to collect records everywhere in your network(switch, router, or firewall)

• Network usage measurements

• An ability to find north-south as well as east-west communication

• Lightweight visibility compared to Switched Port Analyzer (SPAN)-based traffic analysis

• Indications of compromise (IOC)

• Security group information

IE 4000 (NetFlow Lite)

Visibility Through NetFlow

BRKDGT-2601


Group /Segment

LAYER 7Threat FeedTrustSec NAT/ProxyUserInformation

CloudInterfaceInformation

Client Server Translation Service User Application Traffic Group Mac SGT

1.1.1.1 2.2.2.2 3.3.3.3 80/tcp Doug http 20M location 00:2b:1f 10

Session Data | 100% network accountability

Security EventsEvent Data Behavioral Analytics

Visibility

Firewall

PlanningSegmentation

Network

Operations

Network

Visualization

Internal User

MonitoringTrustSecInsider Threat

Use Cases

Stitching Context to Provide User Transaction Visibility

BRKDGT-2601

Cisco StealthWatch


IP SLA Overview

30

In-Line Bear Path Probes

Cisco IOS IP SLAs

VoIPUDP

Jitter

UDP

Echo

ICMP

PathJitterICMP

PathEchoHTTP

DNS,

DHCP

TCP

ConnectFTP

Cisco IOS IP SLAs Operations

Delay Packet Loss JitterPacket

SequenceConnectivity Path

Download

Time

Cisco IOS IP SLAs Metrics

IP SLA MonitoringNetwork Performance

Monitoring

Network Health

Assessment

Edge-to-Edge

Network AvailabilityTroubleshooting

Cisco IOS SLAs Functions

ERP/CRM VoIP VideoWeb

PortalsWeb Conf.

Client-

ServerVPN CoS/QoS

Applications and Solutions

BRKDGT-2601


Use IPSLA to Measure Key Performance Metrics

31

Core

Distribution

Access

Control

Datacentre

PLC

Historian SCADA

PLC

HMI

IP SLAs can send SNMP traps that are triggered by event such as:

• Connection loss

• Timeout

• Round-trip time threshold

• Average jitter threshold

• One-way packet loss

• One-way mean opinion score

(MOS – for VoIP flows)

• One-way latency

BRKDGT-2601

DesigningWithFabrics


Time over Network 101

To transfer time we need:

• Timing source(s)

• Ranging / Transfer mechanism : TWTT

• Client/Slave Servo

• Client/Slave Clock

Clock / Servo

TWTT : Two-Way Time Transfer

BRKDGT-2601


Time over Network 102

Client time accuracy degraded by:

• Time source accuracy

• Path delay jitter

• Quality of servo

• Accuracy of server & client timestamp

• Asymmetrical paths

Clock / Servo

Packet delay variation (PDV) in both directions.

Asymmetry: the TWTT assumption never happens naturally.

BRKDGT-2601


Industrial Network Future → Fabric

Fabric

Users or Devices

Secure Industrial Fabric

Controller Management

• Policies based on User, Device or Application Group

• Traffic Visibility and Fabric Orchestration

• Single User Interface for Network Management

Programmable Overlay

• Dynamic Path Setup and Client Mobility

• Network Segmentation via Virtual Networks (VNs)

• User/Device Segmentation via Segments (Groups)

Prescriptive Underlay

• Topology and Protocol Independent

• Leverage Standards-based Network Infrastructure

• Optimized Forwarding for Time Distribution & Scale

MES

LIMS

ERP

APO

SCADAHISTORIAN

Where clock

synchronisation

is imperative,

design for

physical

symmetry of the

underlying routed

fabric and routing

protocol

configurationBRKDGT-2601

OT Systems

Opaque IP

Underlay

Network


SD-Access Extended Node

• DNA Center Automation and Assurance for Extended Nodes

• Consistent Segmentation and Policy-based Access for IoT Endpoints

• Fabric Benefits (e.g. stretch subnet) for IoT Endpoints

Key Benefits

Fabric Enabled Wireless

REP Ring

Extended NodesIE5000IE4010IE4000

Catalyst Digital Building 3560-CX Compact

Extended Node Portfolio

DNA Center

AnalyticsPolicy Automation

IEEE 802.1Q

VLAN to SGT

Group Mapping

Closing Words


Closing Words

• Continually Challenge your ”My Precious” designs

• Understand where “the Dragons” (Limitations) are

o Control Systems

o Switching Systems

o Organisational Demarcations/Requirements

o Network Equipment/Technology Capabilities

o New Architecture Alternatives

• Keep abreast of technology and product changes

• Design to allow for Evolution

• Focus on people/thing workflows – APIs may be your new best friend

38BRKDGT-2601

Dragon Icon Source: http://icons.mysitemyway.com

http://icons.mysitemyway.com/


• Give us your feedback and receive a Cisco Live 2018 Cap by completing the overall event evaluation and 5 session evaluations.

• All evaluations can be completed via the Cisco Live Mobile App.

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Global.

http://ciscolive.com/Online

http://www.ciscolive.com/Global

Thank you

Reference Information


Background Presentations – CiscoLive! On Demand

• CiscoLive! On Demand Library www.ciscolive.com

• New to Industrial Networking or would like a refresher?:

o BRKIOT-2130 – Anatomy of Modern Process Control Networking Infrastructure

• For a deep dive on Clocking:

o BRKSPG-2170 - Synchronization in Packet-based Networks

43BRKDGT-2601

http://www.ciscolive.com/


IP Device Tracking (IPDT)

• IP Device Tracking (IPDT) Overview


• Cisco Troubleshooting Technotes - Troubleshoot "Duplicate IP Address 0.0.0.0" Error Messages

https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/8021x/116529-problemsolution-product-00.html

• External Link – NAT Overload: Cisco Switch causes duplicate IP address conflict errors on Windows 7

https://www.alfredtong.com/cisco/cisco-switch-causes-duplicate-ip-address-conflict-errors-windows-7/2/

44BRKDGT-2601


https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/8021x/116529-problemsolution-product-00.html

https://www.alfredtong.com/cisco/cisco-switch-causes-duplicate-ip-address-conflict-errors-windows-7/2/


IP SLA

Cisco Industrial Ethernet 4000, 4010 and 5000 Switch Software Configuration Guide

• https://www.cisco.com/c/en/us/td/docs/switches/lan/cisco_ie4010/software/release/15-2_4_EC/configuration/guide/scg-ie4010_5000/swipsla.html

45BRKDGT-2601

https://www.cisco.com/c/en/us/td/docs/switches/lan/cisco_ie4010/software/release/15-2_4_EC/configuration/guide/scg-ie4010_5000/swipsla.html


Time Synchronization

Cisco Whitepaper - What Time Is IT? The Importance of Time on the Network

• https://www.cisco.com/c/en/us/products/collateral/ios-nx-os-software/ios-ip-service-level-agreements-slas/prod_white_paper0900aecd806dab49.html

46BRKDGT-2601

https://www.cisco.com/c/en/us/products/collateral/ios-nx-os-software/ios-ip-service-level-agreements-slas/prod_white_paper0900aecd806dab49.html


Acronyms

ACD – Address Collision Detection

API – Application Programming Interface

APO – Advanced Planner and Optimizer

BACnet – Building Automation and Control Net

CIP – Common Industrial Protocol (ODVA)

COS – Class Of Service

CTS – Cisco TrustSec

CVSS - Common Vulnerability Scoring System

DHCP – Dynamic Host Configuration Protocol

BRKDGT-2601


Acronyms

DLR – Device Level Ring

DNS – Domain Name Service

DNA – Digital Network Architecture

ERP – Enterprise Resource Planning

ERSPAN – Encapsulated Remote Switched Port Analyzer

FTP – File Transfer Protocol

HMI – Human Machine Interface

HTTP – Hypertext Transfer Protocol

ICMP – Internet Control Message Protocol

BRKDGT-2601


Acronyms

IE – Industrial Ethernet

IETF – Internet Engineering Task Force

IND – Industrial Network Director (Cisco)

IOS – Internet Protocol

IOC – Indications Of Compromise

BRKDGT-2601


Acronyms

IP – Internet Protocol

IPDT – IP Device Tracking

IPSG – IP Services Gateway

ISE – Identity Services Engine (Cisco)

IT – Internet Technology

LAN – Local Area Network

LIMS – Laboratory Information Management System

BRKDGT-2601


Acronyms

NAT – Network Address Translation

MAC – Medium Access Control

MAB – MAC Authentication Bypass

MES – Manufacturing Execution System

Modbus – Modicon Bus

MOS – Mean Opinion Score

NMSP – Network Mobility Services Protocol

ODVA – Open DeviceNet Vendor Association

BRKDGT-2601


Acronyms

PDV – Packet Delay Variation

PLC – Programmable Logic Controller

PNI – Prime Network Infrastructure

PRP – Parallel Redundancy Protocol

PROFINET – Process Field Net

PROFINET RT – PROFINET Real-Time

PROFINET IRT – PROFINET Isochronous Real-Time

QoS – Quality of Service

BRKDGT-2601


Acronyms

RFC – Request For Comment

REP – Resilient Ethernet Protocol

RSPAN – Remote Switch Port ANalyzer

SCADA – Supervisory Control And Data Acquisition

SDA – Software Defined Access

SGT – Scalable Group Tag

SLA – Service Level Agreement

SNMP – Simple Network Management Protocol

SPAN – Switch Port ANalyzer

BRKDGT-2601


Acronyms

TCP – Transport Control Protocol

TTL – Time To Live

TWTT – Two Way Time Transfer

UDP – User Datagram Protocol

VLAN – Virtual Local Area Network

VN – Virtual Network

VoIP – Voice Over IP

BRKDGT-2601

BRKDGT-2601 From Theory to Reality - Cisco Live

Documents

Transcript of BRKDGT-2601 From Theory to Reality - Cisco Live