BRKSPG-2900.pdf - Cisco Live

NCS5500Deepdive in the Merchant SiliconHigh-end SP Routers

Nicolas Fevrier, Technical Leader Engineering

BRKSPG-2900

@CiscoIOSXR

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKSPG-2900


What We Hope To Achieve With This Session• Getting familiar with the NCS 5500 portfolio

• Understand the implementation differences compared to traditional XR products

• Introduction

• Products Portfolio

• Fixed / Modular Platforms / Optics

• VOQ and Life of a Packet

• NCS 5500 Internals

• Memory Structure

• Features: ACL / QoS

• Conclusion

Agenda

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 6BRKSPG-2900

Another 170+ Pages Slidedeck ?

• 120 Minutes

• “Back Up Slides” section

• Use of “For Reference” logo

For Reference

Introduction


IOS XR Family

BRKSPG-2900 8

Carrier GradeUltra-high DensityProgrammability

and AutomationCost OptimizedElastic

Virtual RR/PE/DC Forwarder

Cisco IOS-XRv 9000 Cisco NCS 5500 & NCS 5000 Cisco ASR 9000, CRS & NCS 6000

Cisco XR Software


NCS 5500 and NCS 5000

• Both based on Merchant Silicon forwarding ASICs and running IOS XR 64-bit• Still they are very different in nature and in position in networks

• NCS 5500• High scale routing and features• Exists in Fixed and Modular form factors• Deep buffers

• NCS 5000• Lower scale and small buffers• Cost optimized• Can be used as a nV Satellite for ASR 9000 and NCS 6000

Two Very Different Platforms

NCS5500 Portfolio


NCS 5500 Products Family

• 1RU to 23RU systems in fixed of modular form factors

• 3 modular chassis, 7 line cards

• 7 fixed-form systems

11BRKSPG-2900

NCS 5516NCS 5508NCS 5504

NCS 5502

NCS 5501

NCS 55A1-36H

NCS 55A1-24H


NCS 5500 Product Family

• Base and Scale options for modular and fixed systems

• Base• On-chip FIB and small TCAM for ACLs / QoS

• Scale (-SE) have increased FIB and ACL• off-chip TCAM• Fewer forwarding ASICs per line card

• External TCAM is a shared resource• IPv4 & IPv6 route scale• Ingress ACL / QoS matching scale

Base and Scale

CPUDRAM

QSFP28QSFP28QSFP28QSFP28QSFP28QSFP28

ForwardingASIC

Optics x 6 FA

Optics x 6 FA

Optics x 6 FA

Optics x 6 FA

Optics x 6 FA

Buffers

CPUDRAM

QSFP28QSFP28QSFP28QSFP28QSFP28QSFP28

ForwardingASIC

Optics x 6 FA

Optics x 6 FA

Optics x 6 FA

BuffersTCAM

External TCAM

Base System / LCScale System / LC


Custom and MerchantComponents On Cisco Platforms

NCS 6000

NCS 5000 ASR 9000

CRS NCS 5500


Components

• Merchant• Available on the open market to any router vendor or network operator

• Proprietary• Designed or acquired by a router vendor and not available to others

• Custom• Designed in concert with a specific router in mind

Merchant/Commodity, Proprietary, Custom


Simplification is KeyFewer Components: Cost Optimization and Lower Power Consumption

Optics NPUASR9900 FIA Fabric ASIC

Line Card / Slice Line Card / Slice

Fabric ASIC OpticsNPUFIAFabric

ASIC

Optics ForwardingASICNCS 5501 Optics

Optics+ OTNPHYCRS-3/X

IngressQ

Fabric ASIC Inbar PSE

PLIM MSC-X SliceFabric Card

FabricQ

PSE

PLA

PSEEgressQ

Optics+ OTNPHY

PLIMMSC-140 Slice

Optics ForwardingASIC Fabric ASIC Forwarding

ASIC Optics

Line Card / Slice Line Card / Slice

NCS 5502/4NCS 5508/16


NCS 5500 Forwarding ASIC

• Integrated Forwarding and Fabric Interface (28nm technology)

• Two packet processing cores (PP)

Jericho / Qumran-MX / Jericho+ ASICs

Off-chipBuffers

Fabric Interface

Network Interface

TCAM

TCAM

Ingress Egress

LPM

LEM

TCAM

STAT

FECPP TM

PP TM

On-chip Buffer

PP TM

PP TM

OTMOff-chipBuffers

Network Interface

TCAM

TCAM

Ingress Egress

LPM

LEM

TCAM

STAT

FECPP TM

PP TM

On-chip Buffer

PP TM

PP TM

OTMOff-chipBuffers

Fabric Interface

Network Interface

TCAM

TCAM

Ingress Egress

LPM

LEM

TCAM

STAT

FECPP TM

PP TM

On-chip Buffer

PP TM

PP TM

OTM

Jericho600/720Mpps

Jericho+835Mpps

Qumran-MX600/720Mpps

720G 800G 900G

900G 1200G


• Integrated Forwarding and Fabric Interface

• 2 cores, Ingress and egress Pipeline

• On-chip resources• Small internal buffers & TCAM• Route table memory

• Expansion via off-chip resources• Deep GDDR5 packet buffers external packet buffers• Optional TCAMs for route/ACL scale

• Ingress/Egress Traffic Managers• 96k Virtual Output Queues• WRED, Hierarchical scheduling, shaping, policing

NCS 5500 Forwarding ASICJericho / Qumran-MX / Jericho+ ASICs

Off-chipBuffers

Fabric Interface

Network Interface

TCAM

TCAM

Ingress Egress

LPM

LEM

TCAM

STAT

FECPP TM

PP TM

On-chip Buffer

PP TM

PP TM

OTM

Fixed Platforms


NCS 5500 Fixed Chassis

• Single 800 Gbps FA, 4GB packet buffer• 600 Mpps

• No Oversubscription, total interfaces: 800G• 40x 1/10G SFP ports

• 4x 40/100G QSFP ports

• Support of Timing and DWDM interfaces

NCS 5501-SE

SF

P+

SF

P+

SF

P+

SF

P+

SF

P+

SF

P+

ForwardingASIC

QS

FP

28

QS

FP

28

QS

FP

28

Buffers

40x10G 4x100G

TC

AM

CP

UD

RA

M

QS

FP

28

24 ports DWDM/ZR capable (ports 16 to 39)

16 regular ports(ports 0 to 15)



• Single 800 Gbps forwarding ASIC, 4GB packet buffer

• 720 Mpps

• Oversubscribed design, total bandwidth of 1.08 Tbps

• 48x 1/10G SFP ports

• 6x 40/100G QSFP ports

• No DWDM support

• No timing support

NCS 5501

SF

P+

SF

P+

SF

P+

SF

P+

ForwardingASIC

Buffers

48x10G 6x100G

CP

UD

RA

M

QS

FP

28

QS

FP

28

QS

FP

28

QS

FP

28

QS

FP

28

QS

FP

28

SF

P+


NCS 5501100Mbps / 1Gbps Limitations

• NCS 5501-SE ports 0/8 to 0/15• Don’t support 100Mbps copper SFP modules (GLC-T)• Don’t support auto-neg for 1G optical SFP

• NCS 5501-SE other SFP ports• Support 1G and 100M speeds • Support 1G Auto Neg (Clause 37)

• No limitation on the 48 ports of NCS 5501



• 4.8 Tbps line-rate 100G < 2000W (SR optics)

• 48x 100G QSFP28 (or QSFP+)

• 8x 600 Gbps Forwarding ASICs(Common FA with modular chassis)• 600 Mpps per FA

NCS 5502-SE

QS

FP28

QS

FP28

QS

FP28

QS

FP28

QS

FP28

QS

FP28

ForwardingASIC

QS

FP x

6F

A

QS

FP x

6F

A

QS

FP x

6F

ABuf

fers

48x100G

Fabric

CP

U

DR

AM

QS

FP x

6F

A

QS

FP x

6F

A

QS

FP x

6F

A

QS

FP x

6F

A

SwitchSwitch

LCcores

TCA

M



• 4.8 Tbps line-rate 100G < 2000W (SR optics)

• 48x 100G QSFP28 (or QSFP+)

• De-pop’d version without external TCAM

• 8x 600 Gbps Forwarding ASICs• 720 Mpps per FA

NCS 5502

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

ForwardingASIC

QSF

P x

6FA

QSF

P x

6FA

QSF

P x

6FA

Buffe

rs

48x100G

Fabric

CPU

D

RAM

QSF

P x

6FA

QSF

P x

6FA

QSF

P x

6FA

QSF

P x

6FA

SwitchSwitch

LCcores


NCS 5500 Fixed ChassisNCS 5502 Internal Architecture

ForwardingASIC

CPU

DRAM

Fabric Element 018x8x25G=3600G

Fabric Element 118x8x25G=3600G

18

ForwardingASIC

ForwardingASIC

ForwardingASIC

ForwardingASIC

ForwardingASIC

ForwardingASIC

ForwardingASIC

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

QSF

P28

8 Forwarding ASICs2 Fabric ASICs


NCS 5500 Fixed ChassisPowers and Fans

For Reference


NCS-55A1-36H-S / NCS-55A1-36H-SE-S• 1 Rack Unit Fixed System: 36x QSFP28 or QSFP+ optics

• NCS-55A1-36H-S / NCS-55A1-36H-SE-S

• Base and Scale (-SE) versions• Scale extension with eTCAM of 4M entries

• 4x 900 Gbps Jericho+ Forwarding ASICs

• 1x 3.6Tbps Fabric ASIC

• 1588 / Sync-E Capable

• MACSEC Capable

26BRKSPG-2900


NCS-55A1-36H-S / NCS-55A1-36H-SE-S• Single Intel Broadwell-DE D1577 CPU

• 8-core @ 1.6GHz

• 32GB RAM, 64GB SSD

• 2 Redundant Power Modules: 2kW AC or DC• PID: NC55-2KW-ACFW / NC552KW-DCFW• Base system: Typical= 1000W / Max Power= 1450W• Scale system: Typical= 1300W / Max Power= 1700W

• 3 Redundant (N+1) Fan Modules• Front to Back (B2F planned)

• Dimension: 1RU / depth: 30 inches

27BRKSPG-2900

Product LEM LPM eTCAM55A1-36H-S 784k 256k-350k -

55A1-36H-SE-S 784k 256k-350k 4M

For Reference


NCS-55A1-36H-S

28BRKSPG-2900

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

CPU

DRAM

Fabric ASIC4x36x25G=3.6T

36x25G=900G

MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec MACsec

Forwarding ASIC Forwarding ASIC Forwarding ASIC Forwarding ASIC

eTCA

M

eTCA

M

eTCA

M

eTCA

M

• Scale version: with eTCAM• Base version: without eTCAM

eTCAM

36 links @25G from 4 FA to 1 FE= 3.6 Tbps Fabric Capacity / System


NCS-55A1-24H• 1 Rack Unit Fixed System: 24x QSFP28 ports

• Base version only and no MACSEC capability

• 1588 / Sync-E Capable

• 2x 900 Gbps Forwarding ASICs

• No Fabric ASIC, Forwarding ASICs are directly connected

29BRKSPG-2900

Product LEM LPM eTCAMNCS-55A1-24H 784k 1M-1.3M -


NCS-55A1-24H• Single Intel Broadwell-DE D1577 CPU

• 8-core @ 1.6GHz

• 32GB RAM, 128GB SSD

• Dimension: 1RU / Depth: 21 inches

• 2 Redundant Power Modules: AC or DC• PID: NCS-1100W-ACFW / NCS-950W-DCFW • Typical= 600W / Max Power= 800W

• 2 Redundant (N+1) Fan Modules: Front to Back (B2F planned)

30BRKSPG-2900

For Reference


NCS-55A1-24H

31BRKSPG-2900

CPU

DRAM

Forwarding ASIC Forwarding ASIC

48x25G

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

QSFP

4x 25G

Oversubscription of 12x100G ports on 900G Forwarding ASIC


NCS 5500 Fixed Systems Comparison

32BRKSPG-2900

ASIC 100G 10G eTCAM SystemCapacity

Forwarding Capacity

NCS 5501 QMx 6 48 - 1.08 Tbps 800 Gbps

NCS 5501 SE QMx 4 40 Yes 800 Gbps 800 Gbps

NCS 5502 8x J 48 - - 4.8 Tbps 4.8 Tbps

NCS 5502 SE 8x J 48 - Yes 4.8 Tbps 4.8 Tbps

NCS 55A1 24H 2x J+ 24 - - 2.4 Tbps 1.8 Tbps

NCS 55A1 36H 4x J+ 36 - - 3.6 Tbps 3.6 Tbps

NCS 55A1 36H SE 4x J+ 36 - Yes 3.6Tbps 3.6 Tbps

For Reference

Modular Chassis


“Orthogonal Direct” Design• No backplane/midplane for data path

• Direct connection between LC to fabric cards at 90 degrees

• Air inlets above and between optics

• Air inlets on RP & power supplies

34BRKSPG-2900

AIR INLETS


3 Chassis Types

• Common parts

• RP

• SC

• Line Cards

• Power Supply Modules

• Specific

• Chassis

• 3x Fan Tray Modules

• 6x Fabric Line Cards

35BRKSPG-2900


NCS 5500 Modular ChassisMechanical Layout

Fans Fans

RP RP

Power

Line Cards

Controller

Fabric

Controller

Fabric Behind Fans

Air Intake

Front View Rear View Side View w/ Airflow


NCS 5504 Chassis

• Dimensions – 7RU• H x W x D: 12.25 x 17.5 x 31.7“ • (31.1 x 44.50 x 84.20 cm)

• Power Supplies• 4 supplies• AC or DC

Up to 14.4 Tbps

For Reference


NCS 5508 ChassisUp to 28.8 Tbps

• Dimensions – 13RU (1/3 rack)• H x W x D: 22.7 x 17.5 x 31.7” • 57.78 x 44.50 x 80.67 cm• Depth: 34.78 in / 88.34 cm

(from linecard ejector to fantray handles)

• Power Supplies• 8 supplies• NEBS via air filter door and enclosure

• 28.8 Tbps @ 6920 W = 0.24 W/Gbps

• 288 QSFP28 or QSFP+ ports


NCS 5516 Chassis

• Dimensions – 21 RU (1/2 rack)• H x W x D: 36.7 x 17.5 x 31.7” • 93.41 x 44.50 x 80.67 cm• Depth: 34.78 in / 88.34cm

(from LC ejector to FT handles)

• Power supplies• 10 power supplies AC or DC

• 57.6 Tbps @ ~18000W = 0.31 W/Gbps

• 576 QSFP28 or QSFP+ ports

Up to 57.6 Tbps

For Reference


Switch Fabric Cards• Cell-based fabric

• FE3600 fabric ASIC

• 6 Fabric Cards per chassis

• Same Switch Fabric Cards for both FA generations

• Single-stage full mesh utilizing• 1 Fabric ASICs / card for NCS 5504• 2 Fabric ASICs / card for NCS 5508• 6 Fabric ASICs / card for NCS 5516

40BRKSPG-2900


NCS 5500 Modular Chassis36x 100G Line Card Bandwidth Example

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

QSF

P

ForwardingASIC

ForwardingASIC

ForwardingASIC

ForwardingASIC

ForwardingASIC

ForwardingASIC

CPU

DRAM

Fabric Card 06x6x25G=900G

Fabric Card 1900G

Fabric Card 2900G

Fabric Card 3900G

Fabric Card 4900G

Fabric Card 5900G

6


NCS 5500 Modular ChassisCommon System Controller and Route Processor

• Route Processor• Ivy Bridge with 24GB RAM • Routing and management tasks

• System Controller• Chassis control and monitoring

• Fan trays / Power supply• Ethernet Out-of-Band Channel (EOBC)• Ethernet Protocol Channel (EPC)


Chassis Line Cards

43BRKSPG-2900

36 Port 100GE no eTCAM (QSFP)NC55-36X100G-BA

24 Port 100GE External TCAM (QSFP)NC55-24X100G-SB

18 Port 100GE & 18 Port 40GE No eTCAM (QSFP) - NC55-18H18F-BA

24 Port 100GE & 12 Port 40GE External TCAM (QSFP) - NC55-24H12F-SB

36 Port 100GE with MACsecNo eTCAM (QSFP) - NC55-36X100G-BM

6 Port 100/150/200GE with MACsecNo eTCAM (CFP2) - NC55-6x200-DWDM-S

36 Port 100GE External TCAM (QSFP)NC55-36X100G-A-SB


NCS 5500 Line Card Comparison

44BRKSPG-2900

ASIC 100G 40G eTCAM MACSEC

Line CardCapacity

Forwarding Capacity

NC55-36X100G 6x J 36 - - - 3.6 Tbps 3.6 Tbps

NC55-36X100G-S 6x J 36 - - Yes 3.6 Tbps 3.6 Tbps

NC55-18H18F 3x J 18 18 - - 2.52 Tbps 2.16 Tbps

NC55-24X100G-SE 4x J 24 - Yes - 2.4 Tbps 2.4 Tbps

NC55-24H12F-SS 4x J 24 12 Yes - 2.88 Tbps 2.88 Tbps

NC55-36X100G-A-SE 4x J+ 36 - Yes - 3.6 Tbps 3.6 Tbps

NC55-6X2H-DWDM-S 2x J6x

100/150/

200

- - Yes 1.2 Tbps 1.2 Tbps

For Reference


NCS 5500 Modular ChassisSlice-based Architecture

RP/0/RP0/CPU0:NCS5508#sh plat Node Type State Config state -------------------------------------------------------------------------------- 0/6/CPU0 NC55-24H12F-SE IOS XR RUN NSHUT 0/6/NPU0 Slice UP 0/6/NPU1 Slice UP 0/6/NPU2 Slice UP 0/6/NPU3 Slice UP 0/7/CPU0 NC55-24X100G-SE IOS XR RUN NSHUT 0/7/NPU0 Slice UP 0/7/NPU1 Slice UP 0/7/NPU2 Slice UP 0/7/NPU3 Slice UP 0/RP0/CPU0 NC55-RP(Active) IOS XR RUN NSHUT 0/RP1/CPU0 NC55-RP(Standby) IOS XR RUN NSHUT 0/FC0 NC55-5508-FC OPERATIONAL NSHUT 0/FC1 NC55-5508-FC OPERATIONAL NSHUT 0/FC3 NC55-5508-FC OPERATIONAL NSHUT 0/FC5 NC55-5508-FC OPERATIONAL NSHUT 0/FT0 NC55-5508-FAN OPERATIONAL NSHUT RP/0/RP0/CPU0:NCS5508#

NCS 5500 Optics


NCS 5500 / NCS 5500 InterfacesEthernet Only Platforms

• SFP optics slot: offering 1G or 10G (with SFP+) on the following platforms• NCS 5501: 40/48 ports

• QSFP optics slot: offering 100G (with QSFP28), 40G (with QSFP+)and 4x 10G (with QSFP+ with break-out cables) on the following platforms or LC• NCS 5502: 48 ports• Line Cards


NCS 5500 / NCS 5500 InterfacesEthernet Only Platforms

• CFP2 optics slot: only on the 6 ports 100/150/200GE DWDM Line Card

• 25GE only supported on J+ Platforms with 4x25G break-out

NCS 5500 Positioning


NCS 5500 Position in Network

• The position decision of a platform should be based on the following:

• Ports types / density requirement for X years

• Scale requirements

• Buffering capability

• Supported features

• Power consumption

• Network OS preference (IOS XR)

• No simple rule of thumb

Multi-dimensional Equation


Platforms Comparison

NCS 5501 NCS 5501-SE

NCS5502/-SE NCS 5504 NCS 5508 NCS 5516 NCS55A1

-36H/-SENCS55A1

-24H10G 48+6x4 40+4x4 48x4 4x36x4 8x36x4 16x36x4 36x4 24x4

40G 6 4 48 4x36 8x36 16x36 36 24

100G 6 4 48 4x36 8x36 16x36 36 24

BW Gbps 800 800 4,800 14,400 28,800 57,600 3,600 2,400

Total Mpps 720 600 5,760* 17,280 34,560 69,120 3,340 1,679

Power W 240 260 1,850 3,990 7,980 17,100 1,300 600

Pfx scale 1.1M+ 2.75M Up to 2.75M IPv4 4M+ 2M+

Buffer 4GB per Forwarding ASIC

100G 6 4 48 4x 36 8x 36 16x 36 36 24

Queues 96k

For Reference

VOQ and Life of a Unicast Packet


NCS 5500 ArchitectureLocal Routing

Optics

NPUASR9900 FIA Fabric ASIC

Slice Fabric Card

Optics

OpticsForwarding

ASIC Fabric ASIC

Slice

NCS 5502NCS 5508

Fabric Card

Optics

• Local traffic on NCS 5500 series can be routed by the FA without going through the fabric: lower latency


NCS 5500 Architecture

• Two-lookup architecture on traditional XR platforms

Comparison with Traditional XR Platforms

Optics NPUASR9900 FIA Fabric ASIC

Optics+ OTNPHY

CRS-3/XIngressQ

Fabric ASIC Inbar PSEFabricQ

PSE

PLA

PSEEgressQ

Optics+ OTNPHY

Fabric ASIC OpticsNPUFIAFabric

ASIC

Lookup #2Egress to identifyInterface, VLAN,

adjacency

Lookup #1Ingress to identify

destination LC


NCS 5500 Architecture

• Single-lookup architecture at ingress on NCS 5500• VOQ-only Model

Comparison with Traditional XR Platforms

Optics ForwardingASIC

Fabric ASIC ForwardingASIC Optics

NCS 5502NCS 5508

Optics ForwardingASIC

NCS 5501 Optics

Single lookup in ingress FARelevant info set in internal headers


NCS 5500 System Architecture

• All queues of any egress interface are replicated to all ingress devices which want to send packet to that interface. These queues are called VOQs.

VOQ-Only Architecture (Virtual Output Queues)

Fabric Card 0

Fabric Card 1

Fabric Card 2

Fabric Card 3

Fabric Card 4

Fabric Card 5

IngressInterface

EgressInterfaceHu0/7/0/0

Egress Port Queues

Net

Egress VOQ Scheduler

IngressInterface

LC7

VOQ

0/7/0/0 Queue00/7/0/0 Queue10/7/0/0 Queue20/7/0/0 Queue30/7/0/0 Queue40/7/0/0 Queue50/7/0/0 Queue60/7/0/0 Queue7

VOQ

Net

Ingress VOQ SchedulerLC0

0/7/0/0 VOQ00/7/0/0 VOQ10/7/0/0 VOQ20/7/0/0 VOQ30/7/0/0 VOQ40/7/0/0 VOQ50/7/0/0 VOQ60/7/0/0 VOQ7

0/7/0/x VOQ00/7/0/x VOQ10/7/0/x VOQ20/7/0/x VOQ30/7/0/x VOQ40/7/0/x VOQ50/7/0/x VOQ60/7/0/x VOQ7

VOQ

Net

Ingress VOQ SchedulerLC0





Connector



• CLI illustration: Local and Remote visibility of the Output Queues

VOQ-Only Architecture (Virtual Output Queues)

RP/0/RP0/CPU0:NCS5508-1_PE1#sh contr npu voq-usage interface all instance 0 location 0/0/CPU0

-------------------------------------------------------------------Node ID: 0/0/CPU0Intf Intf NPU NPU PP Sys VOQ Flow VOQ Portname handle # core Port Port base base port speed

(hex) type (Gbps)----------------------------------------------------------------------Hu0/3/0/5 1800100 0 0 1 1537 1072 10280 remote 100Hu0/0/0/26 200 4 1 17 273 1424 4136 local 100Hu0/3/0/6 1800108 1 1 21 1621 1080 1064 remote 100Hu0/0/0/27 208 4 0 9 265 1432 5416 local 100Hu0/3/0/7 1800110 1 1 13 1613 1088 2344 remote 100Hu0/0/0/28 210 4 0 5 261 1440 7208 local 100Hu0/3/0/8 1800118 1 1 17 1617 1096 4136 remote 100Hu0/0/0/29 218 4 0 1 257 1448 8488 local 100Hu0/3/0/9 1800120 1 0 9 1609 1104 5416 remote 100Hu0/0/0/30 220 5 1 21 341 1456 2344 local 100


NCS 5500 Forwarding ASIC Detail

• Expansion via off-chip resources• Deep GDDR5 packet buffers external packet buffers

• In normal conditions• Packets are stored in On-Chip Buffers only

• In case of egress congestion• Packets are moved to the Off-Chip Buffer in Virtual Output Queues• Packets are identified by packet descriptors• Each ASIC can manage 3M of these descriptors

• A single queue can take up to 25% of the 1.5M descriptors of a core

• Decision to move packets from on-chip to off-chip buffer is made• When a queue exceeds 200kB• When a queue exceeds 6000 packets

Deep Buffer

Off-chipBuffers

Ingress Egress

LPM

LEM

TCAM

STAT

FECPP TM

PP TM

On-chip Buffer

PP TM

PP TM

OTM


NCS 5500 VOQ-Only Architecture

• Packet is received on ingress interface, classified, and stored in an internal buffer• Single lookup• Queuing is based on credit request and grant scheme• Actual buffering happens on ingress devices

Fabric Card 0

Fabric Card 1

Fabric Card 2

Fabric Card 3

Fabric Card 4

Fabric Card 5

VirtualOutputQueues

IngressInterface

EgressInterface

Net Fab

Egress PortQueues

Fab Net

Ingress VOQ Scheduler




• Ingress VOQ scheduler polls Egress scheduler (maintaining a local VOQ DB)• Egress answers with a credit-message (or not, in our example)• Egress device decides how much traffic can be sent by granting credits to any ingress

requesting Forwarding ASIC

Fabric Card 0

Fabric Card 1

Fabric Card 2

Fabric Card 3

Fabric Card 4

Fabric Card 5

VirtualOutputQueues

IngressInterface

EgressInterface

Net Fab

Egress PortQueues

Fab Net



NO Credit

Queue-Status ?



• Egress device has actual scheduler, supporting priority, min, max and excess. It provides credits based on these attributes

• Scheduling attributes of interface are configured at egress • Congestion control and congestion avoidance values are configured at ingress

Fabric Card 0

Fabric Card 1

Fabric Card 2

Fabric Card 3

Fabric Card 4

Fabric Card 5

VirtualOutputQueues

IngressInterface

EgressInterface

Net Fab

Egress PortQueues

Fab Net



NO Credit

Queue-Status ?



• Finally, the egress schedule grants the credit for packet transmission

Fabric Card 0

Fabric Card 1

Fabric Card 2

Fabric Card 3

Fabric Card 4

Fabric Card 5

VirtualOutputQueues

IngressInterface

EgressInterface

Net Fab

Egress PortQueues

Fab Net



Queue-Status ?

Credit



• Packet is split in cells and load balanced among the fabric cards• Cells are transported to the egress line card

Fabric Card 1

Fabric Card 2

Fabric Card 3

Fabric Card 4

Fabric Card 5

Fabric Card 6

VirtualOutputQueues

ingressInterface

egressInterface

Net Fab

Egress PortQueues

Fab Net





256B

256B

256B

256B

188B

188B

• Let’s take the example of a 1400B packet

• If the last part is between 256B and 512B, we divide by 21400 – 4 x 256 = 376 = 2 x 188



• Cells are collected and packet re-assembled• Packet is stored in the port queue• Finally packet is transmitted through the egress interface

Fabric Card 1

Fabric Card 2

Fabric Card 3

Fabric Card 4

Fabric Card 5

Fabric Card 6

VirtualOutputQueues

ingressInterface

egressInterface

Net Fab

Egress PortQueues

Fab Net




NCS 5500 VOQ-Only Architecture in NCS 5501

• Packet is received on ingress interface, classified, and stored in internal buffer• Ingress VOQ scheduler polls Egress scheduler (maintaining a local VOQ DB)• Egress answers with a credit-message

VirtualOutputQueues

IngressInterface

EgressInterface

Net

Egress PortQueues

Net



Queue-Status ?

Credit


NCS 5500 VOQ-Only Architecture in NCS 5501

• Packet is stored in the port queue• Finally packet is transmitted through the egress interface

VirtualOutputQueues

IngressInterface

EgressInterface

Net

Egress PortQueues

Net



FMQ and Life of the Multicast Packets


Multicast in NCS 5500

• Multicast support introduced in IOS XR 6.1.1

• Initially limited to Source Specific Protocols and IPv4 only (IGMPv3 and PIM SSM)

• ASM protocols (RP discovery, etc) added later

• mLDP Core soon

• (S,G) information stored in LPM

• MC-ID Database: 64k entries

• MC-ID is an index in the table and each entry is a bitmaprepresenting the forwarding ASICs

• Each multicast group is using also one entry in the FEC table

• Replication performed at two levels

• Fabric level

• egress Forwarding ASIC level

69BRKSPG-2900


Multicast in NCS 5500• Based on Fabric Multicast Queues

• Pairs of Traffic Class mapped into FMQ• TC 0 and 1 à FMQ 0• TC 2 and 3 à FMQ 1• TC 4 and 5 à FMQ 2• TC 6 and 7 à FMQ 3

• Not scheduled• Back pressure mechanism needed• Tie-break rule in case of egress congestion

70BRKSPG-2900

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 71

Fabric Card 0

Fabric Card 1

Fabric Card 2

Fabric Card 3

Fabric Card 4

Fabric Card 5

FabricMulticastQueues

IngressInterface

EgressInterface

Net Fab

Egress PortQueuesFab Net

Fab Net

FMQ3 – Traffic Class 6, 7FMQ2 – Traffic Class 4, 5FMQ1 – Traffic Class 2, 3FMQ0 – Traffic Class 0, 1

• Input policy-map sets traffic class• Traffic Class mapped in one of the 4 FMQs

EgressInterfaces

Egress PortQueues

NCS 5500 Life of the Multicast Packets

BRKSPG-2900



Fabric Card 0

Fabric Card 1

Fabric Card 2

Fabric Card 3

Fabric Card 4

Fabric Card 5

Fabric

Multicast

Queues

IngressInterface

EgressInterface

Net Fab

Egress Port

QueuesFab Net

FabEgress Port

Queues Net

Input policy-map

EgressInterfaces

• Ingress Interface receives packet, applies input policy-map

• Then it makes forwarding decision and selects FMQ based on traffic class value

BRKSPG-2900



Fabric Card 0

Fabric Card 1

Fabric Card 2

Fabric Card 3

Fabric Card 4

Fabric Card 5


IngressInterface

Net Fab

FabEgress Port

Queues

Not controlled by output

policy-map

EgressInterface


NetEgressInterfaces

• Ingress Traffic Manager selects packet from an FMQ and gives it to Ingress Fab

BRKSPG-2900



Fabric Card 0

Fabric Card 1

Fabric Card 2

Fabric Card 3

Fabric Card 4

Fabric Card 5


IngressInterface

Net Fab

FabEgress Port

Queues

EgressInterface


NetEgressInterfaces

• Ingress Fab splits packet into cells and load balances them across the fabric cards

BRKSPG-2900



Fabric Card 3

Fabric Card 0

Fabric Card 1

Fabric Card 2

Fabric Card 4

Fabric Card 5


IngressInterface

Net Fab

Egress PortQueues

Egress PortQueues

Fab

Fab Net EgressInterface

NetEgressInterfaces

• Fabric cards replicate cells to each egress card• Egress Fab reassembles and replicates to each interface’s egress queues

BRKSPG-2900



Fabric Card 3

Fabric Card 0

Fabric Card 1

Fabric Card 2

Fabric Card 4

Fabric Card 5


IngressInterface

Net Fab

EgressInterfaceQueues

EgressInterfaceQueues

Fab

Fab

Net

Net EgressInterface

Not controlled by output

policy-map

EgressInterfaces

• Egress Traffic Manager selects packets from egress interface queues

• Egress Net transmits packets

• No ingress replication (one at the fabric, one at the egress NPU level)

BRKSPG-2900



Fabric Card 0

Fabric Card 1

Fabric Card 2

Fabric Card 3

Fabric Card 4

Fabric Card 5

IngressInterface

EgressInterface

Net Fab Fab Net

Fabric Multicast Queues

VirtualOutputQueues

Unicast LP

Multicast LP

HP High PriorityLP Low Priority

Unicast HP

Multicast HP

• Four shallow egress port queues per interface

• They hold the packets before they are put on the wire

BRKSPG-2900

NCS 5500Internals



• EOBC and EPC: two isolated networks

• EOBC network: Ethernet Out-of-Band Channel• Used for inter-process communication (IPC)

• EPC network: Ethernet Protocol Channel• Used for packet punt (all “for-us packets”)

• EMON• Kernel process running on all cards and managing the path• Replaces spanning tree to offer loop free topology• HeartBeat (HB) every 40ms, 5 misses à failure

• System Controller• All these messages are going through the SC cards in NCS 5508 chassis

Intra-Chassis Communication


NCS 5500 Internals

• Ethernet Out-of-Band Channel• Intra-system management

communication• EOBC channel is provided via a switch

chipset on the System Controllers that inter-connects all modules together, including RPs, Fabric Cards and Line Cards

EOBC in Modular ChassisGMAC1GMAC0 GMAC1GMAC0

EOBCSwitch

EOBCSwitch

GMAC0

EOBCSwitch

GMAC0

EOBCSwitch

RP0 RP1

SC0 SC1

LC0-7 FC0-5


NCS 5500 Internals

• Ethernet Protocol Channel• Intra-system data plane protocol

communication• EPC switch only connects Fabric Cards

to RPs• If protocol packets need to be sent to

RP, line cards utilize the internal data path to transfer packets to Fabric Cards first, Fabric Cards then redirect them via the EPC channel to supervisor engines

• Uses different VLAN for different traffic types (one VLAN per NPU for Netflowsampled packets)

EPC in Modular Chassis GMAC1GMAC0 GMAC1GMAC0

EPCSwitch

EPCSwitch

GMAC0

EPCSwitch

EPCSwitch

LC0-7

FC0-5

SC1SC0

RP0 RP1

BRKSPG-2900


NCS 5500 InternalsInternal Switches in Modular Chassis

BRKSPG-2900

sysadmin-vm:0_RP0# show controller switch reachable

Rack Card Switch---------------------0 SC0 SC-SW0 SC0 EPC-SW0 SC0 EOBC-SW0 SC1 SC-SW0 SC1 EPC-SW0 SC1 EOBC-SW0 LC0 LC-SW0 LC1 LC-SW0 LC3 LC-SW0 FC0 FC-SW0 FC1 FC-SW0 FC2 FC-SW0 FC3 FC-SW0 FC4 FC-SW0 FC5 FC-SW

sysadmin-vm:0_RP0#

EPC switchEOBC swithBoth EOBC and EPC

For Reference


NCS 5500 InternalsEPC/EOBC Switches

BRKSPG-2900

• In Line Cards, switchesare shared for EPC/EOBC

• Different bandwidthdepending on the LC type(1G, 2.5G)

• Only one Fabric Card linkis forwarding EPC

SwitchEOBCSwitch

NPU

0

NPU

1

NPU

2

NPU

3

NPU

4

NPU

5LC

CPU

SC0EOBCSwitch

SC1EOBCSwitch

EPCSwitch

FC0-5 SC0 SC1

EOBC EPCdown EOBC+EPC


Example: EPC/EOBC in 24x100G Line Cardssysadmin-vm:0_RP0# show controller switch summary location 0/LC7/LC-SWRack Card Switch Rack Serial Number--------------------------------------0 LC7 LC-SW FGE194714QQ

Phys Admin Port Protocol ForwardPort State State Speed State State Connects To--------------------------------------------------------------------4 Up Up 2.5-Gbps - Forwarding LC CPU (EPC 0)5 Up Up 2.5-Gbps - Forwarding LC CPU (EPC 1)6 Up Up 2.5-Gbps - Forwarding LC CPU (EPC 2)7 Up Up 2.5-Gbps - Forwarding LC CPU (EOBC)8 Up Up 2.5-Gbps - Forwarding NPU29 Up Up 2.5-Gbps - Forwarding NPU110 Up Up 2.5-Gbps - Forwarding NPU011 Up Up 2.5-Gbps - Forwarding NPU312 Up Up 1-Gbps - Forwarding FC013 Down Down 1-Gbps - - FC114 Down Down 1-Gbps - - FC215 Down Down 1-Gbps - - FC316 Down Down 1-Gbps - - FC417 Down Down 1-Gbps - - FC518 Up Up 1-Gbps - Forwarding SC0 EOBC-SW19 Down Down 1-Gbps - - SC1 EOBC-SWsysadmin-vm:0_RP0#

EPCSwitch

EOBCSwitch

NP

U0

NP

U1

NP

U2

NP

U3

LCCPU

SC0EOBCSwitch

SC1EOBCSwitch

EPCSwitch

FC0-5 SC0 SC1

EOBC EPC

down EOBC+EPC

NCS 5500Memory Structure


Hardware ScaleNCS5501 1.1M pfx

NCS5501-SE 2.75M pfxNCS5502 1.1M pfx

NCS5502-SE 2.75M pfxNCS55A1-36H-S 1.1M pfx

NCS55A1-36H-SE-S 4M pfxNCS55A1-24H 2M pfxNC55-36X100G 1.1M pfx

NC55-24X100G-SE 2.75M pfxNC55-18H18F 1.1M pfx

NC55-24H12F-SE 2.75M pfxNC55-36X100G-S 1.1M pfx

NC55-6x200-DWDM-S 1.1M pfxNC55-36X100G-A-SE 4M pfx


NCS 5500 Forwarding ASIC DetailsMemory / Databases

BRKSPG-2900

• Longest Prefix Match Database (LPM or KAPS)• Used to store IPv4 and IPv6 prefixes• Algorithmic memory: worst case 256k entries (IPv6 uses 2 entries)

• Large Exact Match Database (LEM)• Used to store MAC addresses, MPLS labels and IPv4 host prefix

(but also /24, /23, /20… Database size: 784k entries)

• Internal TCAM (iTCAM)• Packet classification (ACL, QoS, VLAN ranges, tunnels. Database size: 48k entries)

• External TCAMs (eTCAM, not on all line cards / systems)• Used for unicast route scale up to 2M or 4M IPv4 Routes• Used to extend ACL and classification


NCS 5500 Forwarding ASIC DetailsAlgorithmic Database (LPM) ?

BRKSPG-2900

• LPM memory is qualified for 256k IPv4 or 128k IPv6 addresses worst case

• Algorithmic memory scaling higher: around 350k with Internet v4 distribution and 160k with Internet v6 distribution

RP/0/RP0/CPU0:Router#show contr fia diagshell 0 "kbp kaps_db_stats" location 0/0/CPU0

Node ID: 0/0/CPU0

Table Configuration

Table-ID Table-Name Size Table Width AD Width Entry Count ~Capacity8 - Public FLP IPv4 UC KAPS 256000 50 20 308390 3425308 - Private FLP IPv4 UC KAPS 256000 50 20 308390 342530

<SNIP>

53 - Public FLP IPv4 UC SCALE SHORT KAPS 256000 42 20 308390 34253053 - Private FLP IPv4 UC SCALE SHORT KAPS 256000 42 20 308390 34253054 - Public FLP IPv4 UC SCALE LONG KAPS 256000 50 20 308390 34253054 - Private FLP IPv4 UC SCALE LONG KAPS 256000 50 20 308390 342530RP/0/RP0/CPU0:Router#


NCS 5500 Forwarding ASIC Details

Algorithmic Database – Specific Case of NCS55A1-24H

BRKSPG-2900

• The 1RU 24x100G system uses a specific flavor of Forwarding ASIC

• LPM is algorithmic memory too but is qualified for minimum of 1M IPv4

prefixes and could scale up to 1.3M


NCS 5500 Forwarding ASIC DetailsMemory / Databases

BRKSPG-2900

• FEC• Used for NextHop and ECMP (128k entries)• Contains the FEC ECMP (4k entries)

• Egress Encapsulation DB (EEDB)• Used for egress rewrites (96k entries)

• Link Local – ARP, ND• Tunnel – MPLS label, GRE, etc

• Ingress/Egress Small Exact Match (ISEM/ESEM)• Used for tunnel termination and egress VLAN translation

• Statistics• Used to store all counters (256k entries)


NCS 5500 DatabasesFor Packet Lookup

BRKSPG-2900

LEM

LPM

eTCAM

FEC

ECMPFEC

EEDB

Ingress Pipeline Egress Pipeline

• Prefix lookup points to FEC Entry• FEC Entry contains VOQ / Egress Interface and EEDB (encapsulation entry)• EEDB indicates the encapsulation for the packet (ARP, ND or GRE, MPLS, …)

Next-HopLoad-balancingPrefixes

Forwarding FEC Resolution Header Editor

FAB

RIC

Encap Editor


Memory Structure for Non-eTCAM Systems / LCPublic Internet View: 655815 v4 routes

RP/0/RP1/CPU0:NCS5508-1-631#sh dpa resources iproute loc 0/6/CPU0 "iproute" DPA Table (Id: 21, Scope: Global) -------------------------------------------------- IPv4 Prefix len distribution Prefix Actual Capacity Prefix Actual Capacity /0 3 16 /1 0 16 /2 0 16 /3 0 16 /4 3 16 /5 0 16 /6 0 16 /7 0 16 /8 17 16 /9 14 16 /10 36 164 /11 107 328 /12 286 655 /13 552 1309 /14 1053 2621 /15 1861 4586 /16 13349 33905 /17 7614 20310 /18 13369 34068 /19 24663 69283 /20 37743 101879 /21 41822 113344 /22 79671 185575 /23 64538 165739 /24 366481 884473 /25 156 3440 /26 126 3604 /27 140 2621 /28 198 2293 /29 395 5569 /30 704 2293 /31 36 164 /32 904 16

NPU ID: NPU-0 NPU-1 NPU-2 NPU-3 In Use: 655841 655841 655841 655841


Memory Structure for Non-eTCAM Systems / LC

• Hiding IPv6 with 0 entry

Public Internet View: 58966 v6 routes

RP/0/RP1/CPU0:NCS5508-1-631#sh dpa resourcesip6route loc 0/6/CPU0 "ip6route" DPA Table (Id: 22, Scope: Global) -------------------------------------------------- IPv6 Prefix len distribution Prefix Actual Prefix Actual /0 3 /1 0 /10 3 /11 0 /16 10 /17 0 /18 0 /19 2 /20 9 /21 3 /22 4 /23 4 /24 19 /25 6 /26 15 /27 17 /28 78 /29 1848 /30 153 /31 127 /32 9279 /33 487 /34 345 /35 357

/36 1436 /37 199 /38 673 /39 154 /40 2239 /41 206 /42 369 /43 113 /44 2213 /45 178 /46 1451 /47 356 /48 19222 /49 0 /50 0 /51 1 /52 1 /53 0 /56 11540 /57 16 /64 5038 /65 0 /96 1 /97 0 /104 3 /105 0 /114 0 /115 4 /122 71 /123 0 /126 0 /127 18 /128 718


Memory Structure for Non-eTCAM Systems / LCIn IOS XR 6.1.4 Onwards

BRKSPG-2900

• Two routes distribution options• Host-optimized - /32 & /24 into LEM with /23 prefix expansion• Internet-optimized - /24, /23 & /20 into LEM

• CLI in config mode (requires LC reload):

# hw-module fib ipv4 scale [host-optimized (default) | internet-optimized]# hw-module fib ipv6 scale internet-optimized (default)


Memory Structure for Non-eTCAM Systems / LCHost Optimized Mode

LEMLookup 1

LPMLookup 1

LEMLookup 2

LPMLookup 2

/32 /31 à /25 /24 /23 à /0

z

LPM

256k-350Kentries

LEM

786kentries

IPv4 prefixes (/32s and /24s)IPv6 prefixes (/48s)MPLS labelsMAC addresses

IPv4

IPv6 LPMLookup 1

LEMLookup

LPMLookup 2

/128 à /49 /48 /47 à /0

LEMLookup

MPLSMAC

IPv4 prefixes (except those in LEM)IPv6 prefixes (non-/48s)Multicast groups v4


Non-eTCAM Systems / LC with Host Optimized ModeIllustration with Public Internet View: 655815 v4 and 58966 v6 real routes

HW Resource Information Name : lem Current Usage

NPU-0 Total In-Use : 386610 (49 %) iproute : 367385 (47 %)

ip6route : 19222 (2 %) mplslabel : 5 (0 %) HW Resource Information Name : lpm Current Usage


ip6route : 39767 (10 %) ipmcroute : 0 (0 %)

v4/32 and v4/24

v6/48

Other v4 routes

Other v6 routes


Memory Structure for Jericho non-eTCAMInternet Optimized Mode

LPMLookup 1

LEMLookup 1

LEMLookup 2

LPMLookup 2

/32 à /25 /24 and /23 /20 /22, /21, /20/19 à /0

z

LPM

256k-350Kentries

LEM

786kentries

IPv4 prefixes (except those in LEM)IPv6 prefixes (non-/48s)Multicast groups v4

IPv4 prefixes (/20s, /23s - /24s)IPv6 prefixes (/48s)MPLS labelsMAC addresses

IPv4

IPv6 LPMLookup 1

LEMLookup

LPMLookup 2

/128 à /49 /48 /47 à /0

LEMLookup

MPLSMAC

• /23s get expanded 2x

• /20s go into LEM if no /22 or /21 overlaps


Non-eTCAM Systems / LC with Internet Optimized ModeIllustration with Public Internet View: 655815 v4 and 58966 v6 real routes


NPU-0 Total In-Use : 530670 (67 %) iproute : 518495 (66 %) ip6route : 19222 (2 %) mplslabel : 5 (0 %) HW Resource Information Name : lpm Current Usage

NPU-0 Total In-Use : 231172 (51 %) iproute : 194021 (43 %) ip6route : 39768 (9 %) ipmcroute : 0 (0 %)

v4/24, v4/23 expandedv4/20

v6/48

Other v4 routesv4/20 with overlaps

Other v6 routes


Memory Structure for eTCAM Systems / LCDefault Distribution

BRKSPG-2900

eTCAM2M/4Mentries

IPv4 pfx (non /32s)

IPv4 /32sIPv6 /48sMPLS labelsMAC addresses

z

LPM

256k-350kentries

LEM

786k entries

LEMLookup

MPLSMAC

64k-160k IPv6 pfxexcept /48sIPv4 Multicast Groups

IPv6 LPMLookup 1

LEMLookup

LPMLookup 2

/128 à /49 /48 /47 à /0

LEMLookup

eTCAMLookupIPv4

/32 /31 à /0


Memory Structure for J w/ eTCAMIllustration with Public Internet View: 655815 v4 and 58966 v6 real routes


NPU-0 Total In-Use : 20132 (3 %) iproute : 904 (0 %) ip6route : 19222 (2 %) mplslabel : 5 (0 %) HW Resource Information Name : lpm Current Usage


ip6route : 39767 (10 %) ipmcroute : 0 (0 %)

HW Resource Information Name : ext_tcam_ipv4 Current Usage

NPU-0 Total In-Use : 654937 (40 %) iproute : 654937 (40 %) ipmcroute : 0 (0 %)

v4/32

v6/48

No v4 routes in LPM

Other v6 routes

All v4 routes except v4/32


Memory Structure for eTCAM and non-eTCAM Systems / LCRecorded Demos

BRKSPG-2900

• http://bit.ly/ncs5500-base

• http://bit.ly/ncs5500-scale

Publicly available Youtube videos details with example how to support internet table and much larger one on Base and Scale NCS5500 systems.

Also contains example with very large host table and projection of internet growth up to 2025.

NCS 5500TCAM Carving


Default eTCAM CarvingJericho w/ eTCAM

80% IPv4 pfxexcept /32s

eTCAM1.6M

entries

20% hybrid ACLs

IPv4 pfxexcept /32s

eTCAM2M

entriesIOS XR 6.2.x

IOS XR 6.1.xIOS XR 6.3.2

RP/0/RP0/CPU0:TME-5508-6.2.3#sh contr npu externaltcam loc 0/6/CPU0

External TCAM Resource Information=============================================================NPU Bank Entry Owner Free Per-DB DB DB

Id Size Entries Entry ID Name=============================================================0 0 80b FLP 498950 1139450 15 IPV4 DC0 1 80b FLP 28672 0 76 INGRESS_IPV4_SRC_IP_EXT0 2 80b FLP 28672 0 77 INGRESS_IPV4_DST_IP_EXT0 3 160b FLP 26624 0 78 INGRESS_IPV6_SRC_IP_EXT0 4 160b FLP 26624 0 79 INGRESS_IPV6_DST_IP_EXT0 5 80b FLP 28672 0 80 INGRESS_IP_SRC_PORT_EXT0 6 80b FLP 28672 0 81 INGRESS_IPV6_SRC_PORT_EXT...

RP/0/RP0/CPU0:NCS5508-6.3.2#sh contr npu ext loc 0/6/CPU0

External TCAM Resource Information=============================================================NPU Bank Entry Owner Free Per-DB DB DB

Id Size Entries Entry ID Name=============================================================0 0 80b FLP 2047993 7 15 IPV4 DC1 0 80b FLP 2047993 7 15 IPV4 DC2 0 80b FLP 2047993 7 15 IPV4 DC3 0 80b FLP 2047993 7 15 IPV4 DC

RP/0/RP0/CPU0:NCS5508-6.3.2#


Default eTCAM CarvingJericho w/ eTCAM with uRPF Loose

IOS XR 6.2.xIOS XR 6.1.xIOS XR 6.3.2

• Activating uRPF requires to disable the eTCAM dual capacity mode

• It effectively reduces the eTCAM size by half

80% IPv4 pfxexcept /32s

eTCAM800k

entries20% hybrid ACLs

disabled

IPv4 pfxexcept /32s

eTCAM1M

entries

disabled


• In 6.3.2, the system is validated for 4M v4 routes (with or without uRPF)

• Hybrid ACL objects are stored in a different zone and don’t impact the scale

Default eTCAM CarvingJericho+ w/ eTCAM

IPv4 pfxexcept /32s

eTCAM4M

entries


• It’s advised to configure a total of 100% for predictable results

• After reload of the line cards

RP/0/RP0/CPU0:R(config)#hw-module profile tcam fib ipv4 unicast percent 50RP/0/RP0/CPU0:R(config)#hw-module profile tcam fib ipv6 unicast percent 50RP/0/RP0/CPU0:R(config)#commit

RP/0/RP0/CPU0:R#show controllers npu diag kbp dbstats instance 0 location 0/7/CPU0

Statistics Rack: 0, Slot: 7, Asic instance: 0

Table Configuration

Tbl-ID Tbl-Name Size Width AD Width Num ent. ~Capacity Shuffles-------------------------------------------------------3 IPv6 UC 256000 160 64 7 51200 04 IPv6 RPF 256000 160 32 0 51200 015 IPV4 DC 1024000 80 48 5 1024000 0

Modifying eTCAM CarvingJericho w/ eTCAM


Modifying eTCAM CarvingJericho and Jericho+ w/ eTCAM

eTCAM2M/4Mentries

x% IPv4 pfx (non /32s)y% IPv6 pfxx+y=100

IPv4 /32sMPLS labelsMAC addresses

z

LPM

LEM

786k entries

LEMLookup

MPLSMAC

IPv4 Multicast Groups

IPv6 eTCAMLookup

/128 à /0

LEMLookup

eTCAMLookup

IPv4

/32 /31 à /0

Only v4/32s are programmed in LEMAll other v4/v6 routes go to eTCAM exceptif x=100 / y=0, IPv6 will be moved to LEM/LPM

Configuring 100% IPv6 in eTCAM is not possible,but 1% / 99% is accepted


• No hard carving on new OP eTCAM• Following configuration only forces routes to be stored in this database:

• Example: Injecting 2M IPv6 routes, we can verify we are not limited to 20%

RP/0/RP0/CPU0:NCS5508-1-631(config)#hw-module profile tcam fib ipv4 unicast percent 80RP/0/RP0/CPU0:NCS5508-1-631(config)#hw-module profile tcam fib ipv6 unicast percent 20RP/0/RP0/CPU0:NCS5508-1-631(config)#commit

RP/0/RP0/CPU0:NCS5508-1-631#sh contr npu externaltcam loc 0/1/CPU0 External TCAM Resource Information ============================================================= NPU Bank Entry Owner Free Per-DB DB DB Id Size Entries Entry ID Name ============================================================= 0 0 80b FLP 0 37 0 IPv4 UC 0 1 80b FLP 0 0 1 IPv4 RPF 0 2 160b FLP 22598 2000113 3 IPv6 UC 0 3 160b FLP 0 0 4 IPv6 RPF

Modifying eTCAM CarvingJericho+ w/ eTCAM


Monitoring Memory Resources

• For both base and scale systems

• Hardware programming is done through an abstraction layer: DPA

• Each database is using two thresholds: yellow at 80% and red at 95%

Thresholds Yellow / Red

LC/0/0/CPU0:Jan 18 23:41:56.750 : fia_driver[279]: %PLATFORM-DPA-1-OOR_RED : NPU 0, Table iprouteLC/0/0/CPU0:Jan 18 23:41:56.750 : fia_driver[279]: %PLATFORM-DPA-4-OOR_YELLOW : NPU 0, Table iprouteLC/0/0/CPU0:Jan 18 23:41:56.750 : fia_driver[279]: %PLATFORM-DPA-1-OOR_RED : NPU 0, Table iprouteLC/0/0/CPU0:Jan 18 23:42:00.336 : fia_driver[279]: %PLATFORM-DPA-1-OOR_RED : NPU 2, Table iprouteLC/0/0/CPU0:Jan 18 23:42:00.418 : fia_driver[279]: %PLATFORM-DPA-1-OOR_RED : NPU 4, Table iprouteLC/0/0/CPU0:Jan 18 23:42:00.438 : fia_driver[279]: %PLATFORM-DPA-4-OOR_YELLOW : NPU 4, Table iprouteLC/0/0/CPU0:Jan 18 23:42:00.439 : fia_driver[279]: %PLATFORM-DPA-1-OOR_RED : NPU 4, Table iproute

RoutingProtocols

Data PlaneAgentRIB Hardware

Resources


Monitoring Memory Resources

• DPA will not program new prefixes and “Hw failures” counter will increment

• Example: advertising 800k IPv4 /24s (in LEM database):• 784k prefixes are actually programmed and 16k are generating failures

Exceeding a Database Capacity

RP/0/RP0/CPU0:NCS5508#sh dpa resources iproute location 0/0/CPU0

<SNIP>NPU ID: NPU-0 NPU-1 NPU-2 NPU-3 NPU-4 NPU-5

<SNIP> Errors

HW Failures: 16131 16131 16131 16132 16131 16131 Resolve Failures: 0 0 0 0 0 0 No memory in DB: 0 0 0 0 0 0 Not found in DB: 0 0 0 0 0 0 Exists in DB: 0 0 0 0 0 0RP/0/RP0/CPU0:NCS5508#RP/0/RP0/CPU0:NCS5508#sh contr npu resources lem location 0/0/CPU0<SNIP>Current Usage

NPU-0 Total In-Use : 783898 (100 %) iproute : 783898 (100 %) (Prefix Count: 783898) mplslabel : 0 (0 %) (Prefix Count: 0) <SNIP>


Monitoring Memory ResourcesCLI to Check LEM Database Usage

RP/0/RP0/CPU0:NCS5508#sh contr npu resources all loc 0/0/CPU0 HW Resource Information Name : lem OOR Information

NPU-0 Estimated Max Entries : 786432 Red Threshold : 95 Yellow Threshold : 80

OOR State : Green OOR State Change Time : 2017.Jan.19 01:49:56 UTC



<...>



<...> Current Usage

NPU-0 Total In-Use : 350029 (46 %) iproute : 350029 (46 %) (Prefix Count: 350029) mplslabel : 0 (0 %) (Prefix Count: 0)


<...>


<...>


Monitoring Memory ResourcesCLI to Check LPM Database Usage

HW Resource InformationName : lpm

OOR InformationNPU-0

Estimated Max Entries : 256000Red Threshold : 95Yellow Threshold : 80OOR State : Green

NPU-1Estimated Max Entries : 256000Red Threshold : 95Yellow Threshold : 80OOR State : Green

<...>


<...>

<...>

Current Usage NPU-0

Total In-Use : 195029 (76 %) iproute : 45003 (17 %) (Prefix Count: 45003)

ip6route : 150026 (58 %) (Prefix Count: 75013) ipmcroute : 0 (0 %) (Prefix Count: 0)

NPU-1 Total In-Use : 195029 (76 %) iproute : 45003 (17 %) (Prefix Count: 45003)


<...>

NPU-5 Total In-Use : 195029 (76 %) iproute : 45003 (17 %) (Prefix Count: 45003)


<...>

For Reference


Monitoring Memory ResourcesCLI to Check EEDB/Encap Database Usage

<...>

HW Resource InformationName : encap

OOR InformationNPU-0

Estimated Max Entries : 80000Red Threshold : 95Yellow Threshold : 80OOR State : Green


<...>


<...>

Current UsageNPU-0

Total In-Use : 3 (0 %)ipnh : 1 (0 %) (Prefix Count: 1)ip6nh : 2 (0 %) (Prefix Count: 2)mplsnh : 0 (0 %) (Prefix Count: 0)

NPU-1Total In-Use : 0 (0 %)ipnh : 0 (0 %) (Prefix Count: 0)ip6nh : 0 (0 %) (Prefix Count: 0)mplsnh : 0 (0 %) (Prefix Count: 0)

<...>

NPU-5Total In-Use : 0 (0 %)ipnh : 0 (0 %) (Prefix Count: 0)ip6nh : 0 (0 %) (Prefix Count: 0)mplsnh : 0 (0 %) (Prefix Count: 0)

RP/0/RP0/CPU0:NCS5508#

For Reference


Monitoring Memory ResourcesCLI to Check eTCAM Usage

RP/0/RP0/CPU0:NCS5508#sh contr npu resources exttcamipv4 location 0/3/CPU0

HW Resource Information Name : ext_tcam_ipv4 OOR Information


OOR State : Green <...> Current Usage

NPU-0 Total In-Use : 395012 (19 %) iproute : 395012 (19 %) (Prefix Count: 395012) ipmcroute : 0 (0 %) (Prefix Count: 0) <...> RP/0/RP0/CPU0:NCS5508#

For Reference


Monitoring Memory ResourcesCLI to Check ECMP FEC Database Usage

RP/0/RP0/CPU0:NCS5508-1-631#sh contr npu resources ecmpfec location 0/7/CPU0 HW Resource Information Name : ecmp_fec OOR Information


OOR State : Green ... Current Usage

NPU-0 Total In-Use : 0 (0 %) ipnhgroup : 0 (0 %)

ip6nhgroup : 0 (0 %) RP/0/RP0/CPU0:NCS5508-1-631#

For Reference


Monitoring Memory ResourcesCLI to Check ECMP FEC Database Usage before 6.3.15

RP/0/RP0/CPU0:ios#show contr npu diag alloc all instance 0 location 0/7/CPU0

Node ID: 0/7/CPU0<SNIP>Pool FECs for global use Total number of entries: 126976 Used entries 14 Lowest entry ID is: 4096(0x1000)Pool VLAN translation ingress usage is unavalible.Pool VLAN translation egress usage is unavalible.Pool VSIs for TB VLANS Total number of entries: 4096 Used entries 0 Lowest entry ID is: 1(0x1)Pool VSIs for MSTP Total number of entries: 28672 Used entries 1 Lowest entry ID is: 4096(0x1000)Pool FEC Failover id (Jericho) Total number of entries: 65533 Used entries 1 Lowest entry ID is: 1(0x1)Pool Ingress Failover id (Jericho) Total number of entries: 32767 Used entries 0 Lowest entry ID is: 1(0x1)Pool Egress Failover id (Jericho) Total number of entries: 32767 Used entries 0 Lowest entry ID is: 1(0x1)Pool Failover id (Arad+ and below) is unavalible.Pool QOS INGRESS LABEL MAP ID Total number of entries: 1 Used entries 0 Lowest entry ID is: 0(0x0)Pool QOS INGRESS LIF/COS IDs Total number of entries: 63 Used entries 0 Lowest entry ID is: 1(0x1)Pool QOS INGRESS PCP PROFILE IDs Total number of entries: 15 Used entries 0 Lowest entry ID is: 1(0x1)Pool QOS INGRESS COS OPCODE IDs Total number of entries: 7 Used entries 0 Lowest entry ID is: 0(0x0)Pool QOS EGRESS REMARK QOS IDs Total number of entries: 15 Used entries 0 Lowest entry ID is: 1(0x1)Pool QOS EGRESS MPLS PHP QOS IDs Total number of entries: 3 Used entries 0 Lowest entry ID is: 1(0x1)Pool number of meters in processor A Total number of entries: 65536 Used entries 442 Lowest entry ID is: 0(0x0)Pool number of meters in processor B Total number of entries: 65536 Used entries 12 Lowest entry ID is: 0(0x0)Pool SW handles of policer Total number of entries: 7 Used entries 0 Lowest entry ID is: 1(0x1)Pool ECMP id Total number of entries: 4095 Used entries 0 Lowest entry ID is: 1(0x1)Pool QOS EGRESS L2 I TAG PROFILE IDs Total number of entries: 1 Used entries 0 Lowest entry ID is: 0(0x0)Pool QOS EGRESS DSCP/EXP MARKING PROFILE ID,s Total number of entries: 4 Used entries 0 Lowest entry ID is: 0(0x0)<SNIP>

For Reference


Monitoring Memory ResourcesAlternative CLI to Check eTCAM Database Usage

RP/0/RP0/CPU0:NCS5508-1-631#show controllers npu diag kbp dbstats instance 0 location 0/1/CPU0 ... Table Configuration Tbl-ID Tbl-Name Size Width AD Width Num ent. ~Capacity Shuffles -------------------------------------------------------------------------------- 0 IPv4 UC 1024000 80 64 37 75591 0 1 IPv4 RPF 1024000 80 32 0 0 0 18 IPV4 UC DUMMY 0 80 32 0 0 0 ... RP/0/RP0/CPU0:NCS5508-1-631#show controllers npu diag kbp dbstats instance 0 location 0/6/CPU0 ... Table Configuration Tbl-ID Tbl-Name Size Width AD Width Num ent. ~Capacity Shuffles -------------------------------------------------------------------------------- 15 IPV4 DC 2048000 80 24 8 2048000 0 20 IPV4 DC DUMMY 0 80 32 0 0 0 ...

For Reference


Monitoring Memory ResourcesCLI to Check Statistics Database Usage in 6.3.x

RP/0/RP0/CPU0:NCS5508-1-631#sh contr npu resources stats instance 0 loc 0/7/CPU0

System information for NPU 0:Counter processor configuration profile: DefaultNext available counter processor: 4

Counter processor: 0 | Counter processor: 1State: In use | State: In use

|Application: In use Total | Application: In use TotalTrap 97 300 | Trap 97 300Policer (QoS) 0 6976 | Policer (QoS) 0 6976ACL RX, LPTS 171 915 | ACL RX, LPTS 171 915

||


|Application: In use Total | Application: In use TotalVOQ 104 8191 | VOQ 104 8191

||

Counter processor: 4 | Counter processor: 5State: Free | State: Free

||


For Reference


Monitoring Memory ResourcesCLI to Check Statistics Database Usage in 6.3.x


||


|Application: In use Total | Application: In use TotalL3 RX 0 8191 | L3 RX 7 8191L2 RX 0 8192 | L2 RX 0 8192

||


|Application: In use Total | Application: In use TotalInterface TX 0 16383 | Interface TX 14 16383

||


|Application: In use Total | Application: In use TotalInterface TX 0 16384 | Interface TX 0 16384

||

RP/0/RP0/CPU0:NCS5508-1-631#

For Reference


Monitoring Memory Resources via YANG

120BRKSPG-2900

<?xml version="1.0"?><rpc-reply message-id="urn:uuid:4883a370-4115-4779-ac18-636371bb7bef" xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><data><dpa xmlns="http://cisco.com/ns/yang/Cisco-IOS-XR-fretta-bcm-dpa-hw-resources-oper"><stats><nodes><node><node-name>0/0/CPU0</node-name><hw-resources-datas><hw-resources-data><resource>lem</resource><resource-id>0</resource-id><name>lem</name><num-npus>6</num-npus><npu-hwr><max-allowed>0</max-allowed><npu-id>0</npu-id><max-entries>750000</max-entries><red-oor-threshold>712500</red-oor-threshold><red-oor-threshold-percent>0</red-oor-threshold-percent><yellow-oor-threshold>600000</yellow-oor-threshold><yellow-oor-threshold-percent>0</yellow-oor-threshold-percent><inuse-objects>13</inuse-objects><num-lt>2</num-lt><oor-change-count>0</oor-change-count><oor-state-change-time1>N/A</oor-state-change-time1><oor-state-change-time2>N/A</oor-state-change-time2><oor-state>Green</oor-state>

...

Mixing Scale and Base LineCards


Selective Route Download Feature

• eTCAM and non-eTCAM can co-exist in the same chassis

• It’s possible to select routes that will be programmed in scale line cards only

• In BGP configuration• using a table-policy and a specific path-color “external-reach”

• With this feature• IGP routes will be programmed in both LC types

• BGP routes with path-color external-reach will be programmedin Scale LC only

• Other BGP routes will programmed in both LC types

122BRKSPG-2900

z

LPM

256k-350Kentries

LEM

786kentries

eTCAM2M

entries

z

LPM

256k-350Kentries

LEM

786kentries


Selective Route Download Configuration

123BRKSPG-2900

route-policy PEER-EXTset community PEER-EXT-comm

end-policy!route-policy HILO-FIBif community matches-any PEER-EXT-comm thenset path-color external-reachpass

elsepass

endifend-policy!

router bgp 100address-family ipv4 unicasttable-policy HILO-FIB

!!neighbor 192.168.100.151address-family ipv4 unicastroute-policy PEER-EXT inmaximum-prefix 8000000 75route-policy PERMIT-ANY out

!


Selective Route Download Verification

• Check a route

124BRKSPG-2900

RP/0/RP0/CPU0:NCS5508-1-631#sh route 1.0.144.0/20

Routing entry for 1.0.144.0/20Known via "bgp 100", distance 200, metric 0, external-reach-lc-onlyTag 2914, type internalInstalled Nov 27 22:48:56.925 for 00:00:45Routing Descriptor Blocks192.168.100.151, from 192.168.100.151Route metric is 0

No advertising protos.RP/0/RP0/CPU0:NCS5508-1-631#


Selective Route Download Verification

125BRKSPG-2900

RP/0/RP0/CPU0:NCS5508-1-631#sh cef 1.0.144.0/20 detail

1.0.144.0/20, version 25081094, external-reach-lc-only, internal 0x5000001 0x0 (ptr 0x8f485390) [1], 0x0 (0x0), 0x0 (0x0)Updated Nov 27 22:48:56.929local adjacency 192.168.100.151Prefix Len 20, traffic index 0, precedence n/a, priority 4gateway array (0x8e0e9250) reference count 655801, flags 0x2010, source rib (7), 0 backups

[1 type 3 flags 0x48501 (0x8e18f758) ext 0x0 (0x0)]LW-LDI[type=0, refc=0, ptr=0x0, sh-ldi=0x0]gateway array update type-time 1 Nov 27 22:48:56.929LDI Update time Nov 27 22:48:56.929via 192.168.100.151/32, 2 dependencies, recursive [flags 0x6000]path-idx 0 NHID 0x0 [0x8e0bf1b0 0x0]next hop 192.168.100.151/32 via 192.168.100.151/32

Load distribution: 0 (refcount 1)

Hash OK Interface Address0 Y MgmtEth0/RP0/CPU0/0 192.168.100.151

RP/0/RP0/CPU0:NCS5508-1-631#


Selective Route Download Use-Case• Lookup executed in ingress only

• Position of the Base and Scale line card is opposite than ASR9k or CRS

• Internet-facing interface could be DWDM card or MACsec card

126BRKSPG-2900

AllInternetRoutes

ScaleLC

BaseLC Internet

OnlyInternalRoutes

MPLSCore

ScaleLC

BaseLC

ContentServers DC role Peering role

Internal+ all

InternetRoutes

MPLSand

CustomerRoutes


Selective Route Download Limitations

• Bundle members

• Can’t span across Hi-FIB and Lo-FIB cards

• Bundle membership configuration will block this combination

• Route coloring

• Not propagated to peers

• Each router must have it’s own policy to filter routes.

127BRKSPG-2900

NCS 5500Access-Lists


Using Access-Lists

• Traditional ACLs• Supported on systems with or without eTCAM• ACEs are stored in iTCAM only

• Hybrid / Scale ACLs• Supported on scale systems only (with eTCAM)• Part of the ACE will be stored and compress on eTCAM• Other part of the ACE will be in iTCAM (2-step look-up mechanism)

With Jericho and Jericho+ LC / Systems


Traditional ACLs

• 12 large banks (0-11): 2k entries each

• 4 small banks (12-15): 128 entries each• Shared between ingress and egress features configured. First come, first served

• Same ACL used on several ingress interfaces are counted once

• Same ACL used on X egress interfaces are counted X times

• Support of 32 ingress and 32/255 egress ACLs per NPU

• Support 4000 IPv4 or 2000 IPv6 ACEs per NPU• Smaller than potential 12k entries (bundles spread among multiple NPUs)

Using Only Internal TCAM (iTCAM)


Traditional ACLs

• IOS XR 6.1.x CLI

Checking Internal TCAM (iTCAM) in 6.1.x

RP/0/RP0/CPU0:NCS5508-1-614#sh contr npu internaltcam loc 0/7/CPU0 | i "(size|NPU|==|Id)" NPU 0: ================================================================================== Bank Id Key Entry Size Free InUse Nof DBs Owner

DB Id DB InUse Prefix ================================================================================== 0 size_160_bits 2043 5 8 pmf-0 1 size_160_bits 2047 1 1 pmf-1 2\3 size_320_bits 1972 76 3 pmf-0 4\5 size_320_bits 2020 28 1 pmf-0 12 size_160_bits 126 2 1 pmf-1 13 size_160_bits 115 13 1 pmf-0 14 size_160_bits 118 10 1 egress_acl...

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSPG-2900

Traditional ACLsChecking Internal TCAM (iTCAM) in 6.2.2 Onwards

RP/0/RP0/CPU0:NCS5508-2-622#sh contr npu internaltcam location 0/7/CPU0 Internal TCAM Resource Information NPU Bank Entry Owner Free Per-DB DB DB Id Size Entries Entry ID Name ============================================================= 0 0\1 320b pmf-0 2006 36 7 INGRESS_LPTS_IPV4 0 0\1 320b pmf-0 2006 2 12 INGRESS_RX_ISIS 0 0\1 320b pmf-0 2006 2 32 INGRESS_QOS_IPV6 0 0\1 320b pmf-0 2006 2 34 INGRESS_QOS_L2 0 2 160b pmf-0 2044 2 31 INGRESS_QOS_IPV4 0 2 160b pmf-0 2044 1 33 INGRESS_QOS_MPLS 0 2 160b pmf-0 2044 1 42 INGRESS_ACL_L2 0 3 160b egress_acl 2022 10 3 EGRESS_RECEIVE 0 3 160b egress_acl 2022 16 4 EGRESS_QOS_MAP 0 4\5 320b pmf-0 2024 24 8 INGRESS_LPTS_IPV6 0 6 160b Free 2048 0 0 0 7 160b Free 2048 0 0 0 8 160b Free 2048 0 0 0 9 160b Free 2048 0 0 0 10 160b Free 2048 0 0 0 11 160b Free 2048 0 0 0 12 160b pmf-1 90 37 11 INGRESS_RX_L2 0 12 160b pmf-1 90 1 13 INGRESS_MCAST_IPV4_ASM 0 13 160b pmf-0 112 2 10 INGRESS_DHCP 0 13 160b pmf-0 112 13 26 INGRESS_MPLS 0 13 160b pmf-0 112 1 41 INGRESS_EVPN_AA_ESI_TO_FBN_DB 0 14 160b Free 128 0 0 0 15 160b Free 128 0 0

Free SpaceNo ACL configured


Traditional ACLsChecking Internal TCAM (iTCAM) in 6.2.2 Onwards

RP/0/RP0/CPU0:NCS5508-2-622#sh contr npu internaltcam location 0/7/CPU0 Internal TCAM Resource Information NPU Bank Entry Owner Free Per-DB DB DB Id Size Entries Entry ID Name ============================================================= 0 0\1 320b pmf-0 2005 37 7 INGRESS_LPTS_IPV4 0 0\1 320b pmf-0 2005 2 12 INGRESS_RX_ISIS 0 0\1 320b pmf-0 2005 2 32 INGRESS_QOS_IPV6 0 0\1 320b pmf-0 2005 2 34 INGRESS_QOS_L2 0 2 160b pmf-0 2044 2 31 INGRESS_QOS_IPV4 0 2 160b pmf-0 2044 1 33 INGRESS_QOS_MPLS 0 2 160b pmf-0 2044 1 42 INGRESS_ACL_L2 0 3 160b egress_acl 2022 10 3 EGRESS_RECEIVE 0 3 160b egress_acl 2022 16 4 EGRESS_QOS_MAP 0 4\5 320b pmf-0 2024 24 8 INGRESS_LPTS_IPV6 0 6 160b pmf-0 997 1051 16 INGRESS_ACL_L3_IPV4 0 7 160b Free 2048 0 0 0 8 160b Free 2048 0 0 0 9 160b Free 2048 0 0 0 10 160b Free 2048 0 0 0 11 160b Free 2048 0 0 0 12 160b pmf-1 90 37 11 INGRESS_RX_L2 0 12 160b pmf-1 90 1 13 INGRESS_MCAST_IPV4_ASM 0 13 160b pmf-0 112 2 10 INGRESS_DHCP 0 13 160b pmf-0 112 13 26 INGRESS_MPLS 0 13 160b pmf-0 112 1 41 INGRESS_EVPN_AA_ESI_TO_FBN_DB 0 14 160b Free 128 0 0 0 15 160b Free 128 0 0

1000 ACEs configured


Traditional ACLs

• Limitations with packets targeted to the router

• For-us packets matching deny ACE

• Counted and dropped

• For-us packets matching permit ACE

• Punted and not counted

Counters


Traditional ACLs

• By default only deny ACEs are allocated counters

• Permit entries can be allocated counters via configuration

• Requires a reload of the line card to be activated

Counting with permit ACEs

RP/0/RP0/CPU0:NCS5508-1-631(config)#hw-module profile stats acl-permitRP/0/RP0/CPU0:NCS5508-1-631(config)#commit

RP/0/RP0/CPU0:NCS5508-1-631#sh access-lists ipv4 PERMIT-TEST hardware ingress location 0/7/CPU0Mon Nov 6 20:40:42.043 UTCipv4 access-list PERMIT-TEST 10 permit icmp any host 1.1.1.1 15 permit icmp any host 1.1.1.3 16 permit tcp any any eq telnet (2 matches) ßßß17 permit tcp any eq telnet any 20 permit udp any any 30 permit tcp any any 40 deny ipv4 any any (1169 matches)RP/0/RP0/CPU0:NCS5508-1-631#


Hybrid ACLs

• In 6.3.2, requires a carving

• IPv4 and IPv6

• Ingress only

• Two-step look-up• First in eTCAM• Second in iTCAM

Only on eTCAM Systems

object-group network ipv4 netobj1description my-network-objecthost 10.1.1.110.2.1.0 255.255.255.0range 10.3.1.10 10.3.1.50!object-group port portobj1description my-port-objecteq bgprange 100 200!ipv4 access-list network-object-acl10 deny tcp net-group netobj1 port-group portobj1 any20 permit ipv4 net-group netobj1 any!interface hu0/7/0/0ipv4 access-group network-object-acl ingress compress level 3

80%

eTCAM

20%

iTCAMv4 Pfx

ACL


Hybrid ACLs

• CLI to display anexpanded version of theaccess-list

Example

RP/0/RP0/CPU0:R1#sh access-lists ipv4 network-object-aclipv4 access-list network-object-acl10 deny tcp net-group netobj1 port-group portobj1 any20 permit ipv4 net-group netobj1 any

RP/0/RP0/CPU0:R1#sh access-lists ipv4 network-object-acl expandedipv4 access-list network-object-acl10 deny tcp 10.2.1.0 0.0.0.255 eq telnet any10 deny tcp 10.2.1.0 0.0.0.255 eq bgp any10 deny tcp 10.2.1.0 0.0.0.255 range 100 200 any10 deny tcp host 1.11.111.1 eq telnet any10 deny tcp host 1.11.111.1 eq bgp any10 deny tcp host 1.11.111.1 range 100 200 any10 deny tcp host 1.3.5.7 eq telnet any10 deny tcp host 1.3.5.7 eq bgp any10 deny tcp host 1.3.5.7 range 100 200 any20 permit ipv4 10.2.1.0 0.0.0.255 any20 permit ipv4 host 1.11.111.1 any20 permit ipv4 host 1.3.5.7 anyRP/0/RP0/CPU0:R1#

object-group network ipv4 netobj110.2.1.0/24host 1.3.5.7host 1.11.111.1!object-group port portobj1eq telneteq bgprange 100 200!


Hybrid ACLsMonitoring Resource: 1- On eTCAM

RP/0/RP1/CPU0:NCS5508-1-631#sh contr npu externaltcam loc 0/7/CPU0 External TCAM Resource Information ============================================================= NPU Bank Entry Owner Free Per-DB DB DB Id Size Entries Entry ID Name ============================================================= 0 0 80b FLP 983784 654616 15 IPV4 DC 0 1 80b FLP 28634 38 81 INGRESS_IPV4_SRC_IP_EXT 0 2 80b FLP 28671 1 82 INGRESS_IPV4_DST_IP_EXT 0 3 160b FLP 26624 0 83 INGRESS_IPV6_SRC_IP_EXT 0 4 160b FLP 26624 0 84 INGRESS_IPV6_DST_IP_EXT 0 5 80b FLP 28664 8 85 INGRESS_IP_SRC_PORT_EXT 0 6 80b FLP 28672 0 86 INGRESS_IPV6_SRC_PORT_EXT ...


Hybrid ACLsMonitoring Resource: 2- On iTCAM

RP/0/RP1/CPU0:NCS5508-1-631#sh contr npu internaltcam loc 0/7/CPU0 Internal TCAM Resource Information ============================================================= NPU Bank Entry Owner Free Per-DB DB DB Id Size Entries Entry ID Name ============================================================= 0 0\1 320b pmf-0 1963 49 7 INGRESS_LPTS_IPV4 0 0\1 320b pmf-0 1963 2 12 INGRESS_RX_ISIS 0 0\1 320b pmf-0 1963 11 32 INGRESS_QOS_IPV6 0 0\1 320b pmf-0 1963 23 34 INGRESS_QOS_L2 0 2 160b pmf-0 2030 11 31 INGRESS_QOS_IPV4 0 2 160b pmf-0 2030 6 33 INGRESS_QOS_MPLS 0 2 160b pmf-0 2030 1 42 INGRESS_ACL_L2 0 3 160b egress_acl 2032 16 4 EGRESS_QOS_MAP 0 4\5 320b pmf-0 2021 27 8 INGRESS_LPTS_IPV6 0 6\7 320b pmf-1 2045 3 49 INGRESS_HYBRID_ACL 0 8 160b Free 2048 0 0 0 9 160b Free 2048 0 0 0 10 160b Free 2048 0 0 0 11 160b Free 2048 0 0 0 12 160b pmf-1 88 40 11 INGRESS_RX_L2 0 13 160b pmf-0 84 3 10 INGRESS_DHCP 0 13 160b pmf-0 84 1 13 INGRESS_MCAST_IPV4_ASM 0 13 160b pmf-0 84 13 26 INGRESS_MPLS 0 13 160b pmf-0 84 1 41 INGRESS_EVPN_AA_ESI_TO_FBN_DB 0 13 160b pmf-0 84 26 79 INGRESS_BFD_IPV4_NO_DESC_TCAM_T 0 14 160b Free 128 0 00 15 160b Free 128 0 0

`

NCS 5500 QoS


Quality of Service

• Ingress direction supports classification and remarking

• Ingress direction supports only policing

• Egress direction supports only shaping

Ingress EgressPolicingconfig

Queueingconfig


Quality of Service

• We use internal markers at ingress to take egress actions

Internal Markers

Ingress Egress

match xxxset qos-group

match yyyset traffic-class

match zzzset discard-class

match qos-group

match traffic-class

random-detect discard-class 1 x ms y msrandom-detect discard-class 2 x ms y ms

Egress Remarking

Queueing / Shaping

WRED


Configuring Quality of ServicePolicer Configuration

Class-MapMatch criterias

set qos-group

Policer

Ingress Egress

class-map classify1match precedence 1

policy-map Pol1class classify1set qos-group 1set dscp efpolice rate percent 10

interface hu 0/0/0/0service-policy input Pol1

(optional)

set dscp/… (optional)


Configuring Quality of ServicePolicer Configuration




policy-map ingress-policyclass classify1

set qos-group 1police rate percent 10 peak-rate percent 20

class classify2set qos-group 2

class classify3set qos-group 3

interface hu 0/0/0/0service-policy input ingress-policy

10Gbps Prec 2

10Gbps Prec 3

10Gbps Prec 4

30Gbps Prec 1

10Gbps Prec 2

10Gbps Prec 3

10Gbps Prec 4

20Gbps Prec 1

qos-group 2

qos-group 3

qos-group 0

qos-group 1

For Reference


Configuring Quality of ServiceShaper Configuration

Class-MapMatch criterias

Set traffic-class

Class-MapMatch traffic-class

Ingress Egress

Shaper

class-map match-any classify1match precedence 1end-class-map!class-map match-any classify2match precedence 2end-class-map!class-map match-any classify3match precedence 3end-class-map!policy-map Pol1class classify1set traffic-class 1!class classify2set traffic-class 2!class classify3set traffic-class 3!class class-defaultset traffic-class 7!end-policy-mapinterface bundle-ether 1service-policy input Pol1

class-map match-any tc1match traffic-class 1end-class-map!class-map match-any tc2match traffic-class 2end-class-map!class-map match-any tc3match traffic-class 3end-class-map!policy-map Pol1class tc1priority level 1shape average percent 20!class tc2shape average percent 50!class tc3shape average percent 30!class class-default!end-policy-map!interface hu 0/0/0/0service-policy output Pol1

40Gbps Prec 2

20Gbps Prec 3

10Gbps Prec 4

30Gbps Prec 1

qos-group 2

qos-group 3

qos-group 0

qos-group 1


Configuring Quality of ServiceComing Soon: Egress Dual-Policy

class-map match-any cos1match cos 1

end-class-map! class-map match-any cos2

match cos 2end-class-map! policy-map ingress-classify

class cos1set qos-group 1set traffic-class 3

!class cos2

set qos-group 2set traffic-class 5

! class class-default!

class-map match-any qos1match qos-group 1

end-class-map!class-map match-any qos2

match qos-group 2end-class-map!policy-map egress-marking class qos1 set cos 1 ! class qos2 set cos 2 set dei 1 ! class class-default set cos 7 ! end-policy-map

class-map match-any tc3match traffic-class 3

end-class-map!class-map match-any tc5

match traffic-class 5end-class-map!policy-map egress-queuing

class tc3priority level 1 shape average 10 mbps

!class tc5

bandwidth remaining <>!class class-default!

end-policy-map!

interface TenGigE0/0/1/0/0service-policy input ingress-classifservice-policy output egress-markingservice-policy output egress-queuing!

For Reference

BRKSPG-2900 146



• All QoS rules applied to a bundle are applied to all members

BE100

Hu0/0/0/0

Priority1: 10%

Queue2: 50%

Queue3:25%Default: 15%

Hu0/1/0/0

Priority1: 10%

Queue2: 50%


Priority1: 10%

Queue2: 50%

Queue3:25%

Default: 15%

BE100

Hu0/0/0/0

Priority1: 10%

Queue2: 50%


Priority1: 10%

Queue2: 50%

Queue3:25%

Default: 15%

Hu0/1/0/0goes down



• If we use absolute values, they are applied to each member too

à Use percent

BE100

Hu0/0/0/0

Priority1: 5G

Queue2: 25G

Queue3: 12G

Default: 7G

Hu0/1/0/0

Priority1: 5G

Queue2: 25G

Queue3: 12G

Default: 7G

Priority1: 5G

Queue2: 25G

Queue3: 12G

Default: 7G

BE100

Hu0/0/0/0

Priority1: 5G

Queue2: 25G

Queue3: 12G

Default: 7G

Priority1: 5G

Queue2: 25G

Queue3: 12G

Default: 7G

Hu0/1/0/0

goes down


Key Differences with Traditional XR Platforms

• In case of egress interface congestion

• If unicast or multicast is high priority, it will take full precedence over the other

• If same priority (HP/HP or LP/LP), then the forwarding will be 80% ucast / 20% mcast

Unicast is Scheduled but Multicast Traffic doesn’t Follow VOQ-only Model

HunG

10G10G

HunG

10G10G

HunG

10G10G

10G 10G 10G 10G 10G 10G

HP LP LP LP LP HP

TenG TenG TenG

8G 2G10G 10G

NCS 5500 Software Architecture


Software Architecture

RouteProcessor

IOS XRLxC System

AdminLxC

Host OS (Linux)

Classic and Containerized OS

Host OS (Linux)

IOS XRLxC

AdminLxC Line Cards

Containerized IOS XR

IOS XR

QNX

Classic IOS XR

QNX

IOS XR

RoutingApps

System Admin

RoutingApps

• IOS XR Exists in two flavours

• 32-bit in XR12k, CRS, ASR9000• QNX-based• No virtualization

• 64-bit in ASR9000, NCS 5500,NCS 5000, NCS 1000 and in NCS 6000• Linux based• Larger addressable memory• Separation Networking OS and

System Management• Third party Containers


Understanding the 64-bit OS StructurePrompt indicates if you are connected to an LXC or Guest/Host OS

RP/0/RP0/CPU0:LAB#RP/0/RP0/CPU0:LAB#runMon Nov 16 21:27:01.148 UTC

[xr-vm_node0_RP0_CPU0:~]$[xr-vm_node0_RP0_CPU0:~]$exitexitRP/0/RP0/CPU0:LAB#adminMon Nov 16 21:27:05.958 UTC

root connected from 127.0.0.1 using console on xr-vm_node0_RP0_CPU0sysadmin-vm:0_RP0#sysadmin-vm:0_RP0# runMon Nov 16 21:27:08.336 UTC

[sysadmin-vm:0_RP0:~]$[sysadmin-vm:0_RP0:~]$ssh 10.0.2.16[host:0_RP0:~]$[host:0_RP0:~]$

IOS XR LXC

IOS XRLxC

AdminLxC

(calvados)

Host OSWind River 7, linux kernel 3.14

RP0

IOS XRGuest OS

AdminGuest OS

XR Guest OS

admin Guest OSadmin Host OS

admin LXC (calvados)


Understanding the 64-bit OS StructureHost OS: WR7

[sysadmin-vm:0_RP0:~]$arpAddress HWtype HWaddress Flags Mask Iface192.0.96.1 ether b0:aa:77:e6:fe:0a C eth-vf1192.0.120.1 ether 04:15:7b:14:1e:00 C eth-vf1192.0.16.1 ether a4:6c:2a:2b:53:00 C eth-vf1192.0.104.1 ether b0:aa:77:e7:b7:b6 C eth-vf1192.0.16.3 ether a4:6c:2a:2b:53:02 C eth-vf1192.0.108.4 ether 64:f6:9d:06:9d:53 C eth-vf1192.0.116.1 ether 04:15:7b:14:1d:00 C eth-vf110.0.2.16 ether 36:19:1c:08:0f:d5 C eth-vf0192.0.100.1 ether b0:aa:77:e7:6b:c2 C eth-vf1[sysadmin-vm:0_RP0:~]$ssh 10.0.2.16[host:0_RP0:~]$[host:0_RP0:~]$ virsh list --allId Name State----------------------------------------------------4778 sysadmin running10135 default-sdr--1 running

[host:0_RP0:~]$

IOS XRLXC

adminLXC

(calvados)


For Reference


OS Structure on Modular PlatformsNCS 5508: Based on LXCs and not VMs

IOS XRLXC

adminLXC

(calvados)


RP0/1

IOS XRGuest OS

adminGuest OS

IOS XRLXC

adminLXC

(calvados)


LC0-7

IOS XRGuest OS

adminGuest OS

adminLXC

(calvados)


SC0/1

adminGuest OS

adminLXC

(calvados)


FC0-7

adminGuest OS

3rd PartyLXC

IOS XRGuest OS


OS Structure on Modular PlatformsCLI Illustration

sysadmin-vm:0_RP0# sh vm

Location: 0/0Id Status IP Address HB Sent/Recv-------------------------------------------------------------sysadmin running 192.0.4.1 NA/NAdefault-sdr running 192.0.4.3 30867/30867

Location: 0/3Id Status IP Address HB Sent/Recv-------------------------------------------------------------sysadmin running 192.0.16.1 NA/NAdefault-sdr running 192.0.16.3 290117/290117

Location: 0/FC0Id Status IP Address HB Sent/Recv-------------------------------------------------------------sysadmin running 192.0.84.1 NA/NA


<...SNIP...>


Location: 0/RP0Id Status IP Address HB Sent/Recv-------------------------------------------------------------sysadmin running 192.0.108.1 NA/NAdefault-sdr running 192.0.108.4 5802352/5802352

Location: 0/RP1Id Status IP Address HB Sent/Recv-------------------------------------------------------------sysadmin running 192.0.112.1 NA/NAdefault-sdr running 192.0.112.4 5802356/5802356

Location: 0/SC0Id Status IP Address HB Sent/Recv-------------------------------------------------------------sysadmin running 192.0.116.1 NA/NA

Location: 0/SC1Id Status IP Address HB Sent/Recv-------------------------------------------------------------sysadmin running 192.0.120.1 NA/NAsysadmin-vm:0_RP0#

For Reference


OS Structure on Fixed Platforms

• NCS 5501 is a SoC architecture, everything operates in one CPU

• NCS 5502 is actually a distributed platform, we will use a container for the LC

NCS 5501 and NCS 5502

IOS XRLXC

adminLXC

(calvados)


NCS 5501 / NCS 5502

IOS XRGuest OS

adminGuest OS

LineCardIOS XR

LXC

IOS XRGuest OS

3rd PartyLXC

IOS XRGuest OS

Conclusion


Conclusion• Merchant silicon is not something new in SP portfolio

• Port density and features

• NCS 5500 can be used in multiple roles in Networks such as• Core, Peering, SP DC, Aggregation and potentially Edge

• Architecture based on VOQ-only

• Compared to traditional IOS XR platforms• Resources needs to be monitored differently• Features can have a different implementation

158BRKSPG-2900


Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKSPG-2900


• Please complete your Online Session Evaluations after each session

• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online.

Complete Your Online Session Evaluation


Continue Your Education• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

161BRKSPG-2900

Thank you


NCS 5500 CEF Implementation• Prefix lookup points to FEC Entry• FEC Entry contains VOQ / Egress Interface and EEDB (encapsulation entry)• EEDB indicates the encapsulation for the packet (ARP/ND or GRE, MPLS,…)

LEM

LPM

eTCAM

FEC

ECMPFEC

EEDB

Ingress Pipeline Egress Pipeline

Next-HopLoad-balancingPrefixes

Forwarding FEC Resolution Header Editor

FAB

RIC

Encap Editor

BRKSPG-2900 165


RP/0/RP0/CPU0:R1#sh cef 4.4.4.4 hardware egress location 0/0/CPU0 4.4.4.4/32, version 304, internal 0x1000001 0x0 (ptr 0x8908a698) [3], 0x0 (0x88f74868), 0x0 (0x0)Updated Jan 11 14:31:40.966 local adjacency 192.1.2.2Prefix Len 32, traffic index 0, precedence n/a, priority 1via 192.1.2.2/32, HundredGigE0/0/0/46, 5 dependencies, weight 0, class 0 [flags 0x0]path-idx 0 NHID 0x0 [0x89d760a0 0x0]next hop 192.1.2.2/32local adjacency

LEAF - HAL pd context : sub-type : IPV4, ecd_marked:0, has_collapsed_ldi:0collapse_bwalk_required:0, ecdv2_marked:0HW Walk:LEAF:

PI:0x308908a698 PD:0x308908a730 Rev:1198 type: 0 FEC handle: 0x308936b248

LWLDI:PI:0x3088f74868 PD:0x3088f748a8 rev:1197 p-rev:1187 ldi type:3FEC hdl: 0x308936b248 fec index: 0x0(0) num paths:1, bkup: 0

SHLDI:PI:0x3088df2068 PD:0x3088df20e8 rev:1187 dpa-rev:6925889 flag:0x0FEC hdl: 0x308936b248 fec index: 0x2000100a(4106) num paths: 1 bkup paths: 0p-rev:1129 Path:0 fec index: 0x2000100a(4106) DSP:0x570 Dest fec index: 0x0(0)

TX-NHINFO:PD: 0x3089d76118 rev: 1129 dpa-rev: 6919550 Encap hdl: 0x3089e3b0e8Encap id: 0x40010002 Remote: 0 L3 int: 1048 npu_mask: 80

RP/0/RP0/CPU0:R1#

FEC Handle

Since it is a non-recursive lookup, the FEC location is in SHLDI.

FEC Index and the number of paths available as next hop.

FEC Index points to DSP which is the pointer to the next hop. In this example, the VOQ 0x570 = 1392.

Encap id used in next page

NCS 5500 CEF ImplementationCase 1: Non-Recursive / No-ECMP


RP/0/RP0/CPU0:R1#show controllers fia diagshell 7 "diag pp LIF_show type=out id=0x40010002 gl=1" location 0/0/CPU0

Node ID: 0/0/CPU0

************************************************ Warning: Core 0: Packet diagnostics refers to OLP packet ************************************************

Global Out_LIF:0x40010002 -> Local Out_LIF:0x00004008 -> Type:(null) Bank:2 Offset:4

LL Encapsulation:^Mdest_mac:0c:11:67:46:ac:30^Mout_vid_valid: 1^Mout_vid: 1048^Mpcp_dei_valid: 0^Mpcp_dei: 0^Mtpid_index: 0^Mll_remark_profile: 0^Mout_ac_valid: 0^Mout_ac_lsb: 0^Moam_lif_set: 0^Moutlif_profile: 0x10^MNext_eep: 0x0

Encap pointer to the location in EEDB.

ASIC number where to which the outgoing port belongs to.

Pointer to the outgoing interface to which the int-VoQis associated that was received in the FTMH header.

Destination MAC address.

RP/0/RP0/CPU0:R1#sh contr npu voq-usage interface all instance all location 0/0/CPU0 | i 1392

Hu0/0/0/46 300 7 0 5 453 1392 1336 local 100

RP/0/RP0/CPU0:R1#

Local: location 0/0/cpu0Port 46: NPU number 7



RP/0/RP0/CPU0:Router#sh cef 4.4.4.4 hardware egress location 0/0/CPU0

4.4.4.4/32, version 304, internal 0x1000001 0x0 (ptr 0x8908a698) [3], 0x0 (0x88f74868), 0x0 (0x0)Updated Jan 11 14:31:40.966 local adjacency 192.1.2.2Prefix Len 32, traffic index 0, precedence n/a, priority 1via 192.1.2.2/32, HundredGigE0/0/0/46, 5 dependencies, weight 0, class 0 [flags 0x0]path-idx 0 NHID 0x0 [0x89d760a0 0x0]next hop 192.1.2.2/32

local adjacency

LEAF - HAL pd context : sub-type : IPV4, ecd_marked:0, has_collapsed_ldi:0collapse_bwalk_required:0, ecdv2_marked:0HW Walk:LEAF:

Hardware WalkLet’s focus on this partof the show cef CLI



HW Walk: LEAF:

PI:0x308908a698 PD:0x308908a730 Rev:1198 type: 0 FEC handle: 0x308936b248

LWLDI:

PI:0x3088f74868 PD:0x3088f748a8 rev:1197 p-rev:1187 ldi type:3 FEC hdl: 0x308936b248 fec index: 0x0(0) num paths:1, bkup: 0

SHLDI:

PI:0x3088df2068 PD:0x3088df20e8 rev:1187 dpa-rev:6925889 flag:0x0 FEC hdl: 0x308936b248 fec index: 0x2000100a(4106) num paths: 1 bkup paths: 0 p-rev:1129 Path:0 fec index: 0x2000100a(4106) DSP:0x570 Dest fec index: 0x0(0)

TX-NHINFO:

PD: 0x3089d76118 rev: 1129 dpa-rev: 6919550 Encap hdl: 0x3089e3b0e8 Encap id: 0x40010002 Remote: 0 L3 int: 1048 npu_mask: 80 RP/0/RP0/CPU0:Router#

FEC Handle Index:0Non-recursive à SHLDI

Paths: 1No ECMP

FEC Index

DSPPointer to NH

Encap idIn EEDB

Egress intf



RP/0/RP0/CPU0:R1#show controllers fia diagshell 7 "diag pp LIF_show type=out id=0x40010002 gl=1" location 0/0/CPU0

Node ID: 0/0/CPU0************************************************ Warning: Core 0: Packet diagnostics refers to OLP packet ************************************************

Global Out_LIF:0x40010002 -> Local Out_LIF:0x00004008 -> Type:(null) Bank:2 Offset:4

LL Encapsulation:^Mdest_mac:0c:11:67:46:ac:30^Mout_vid_valid: 1^Mout_vid: 1048^Mpcp_dei_valid: 0^Mpcp_dei: 0^Mtpid_index: 0^Mll_remark_profile: 0^Mout_ac_valid: 0^Mout_ac_lsb: 0^Moam_lif_set: 0^Moutlif_profile: 0x10^MNext_eep: 0x0

Encap idIn EEDB

ASIC InstanceIdentified with egress intf

Local Out-LIF

DestinationMAC AddressLinked to this encap ID



ECMP FEC FEC DB

Forwarding FEC Resolution – L3

ECMP FECLevel L2

Level L1

IRPP ETPP

ITM Header Editor

DRAM

EEDB

SHLDI (non-recursive)

RSHLDI (recursive)

FEC DBLEM

FTMH

4.4.4.4/32

0x400100020x570

FEC handle: 0x308936b248

Since lookup is non-recursive, the location will be in SHLDIFEC index 0x2000100aNum of path=1

@ 0x2000100a0x570 + 0x40010002

DMAC:0c:11:67:46:ac:30

SMAC of Local Out_LIF

Packet Out

Header

Payload

@ 0x40010002

NCS 5500 CEF Implementation• Case 1: Non-Recursive / No-ECMP

BRKSPG-2900 171


Generic Comments on uRPF• Only uRPF Loose mode supported (not Strict mode)

• Loose mode is still useful in 2017 or later?• How many packets are actually dropped in loose mode?• Source-Based RTBH is a valid use-case

• Enabled at interface level

• Deactivating uRPF• Involves disabling it for both IPv4 and IPv6 on the interface

• allow-default is not supported

• allow-self-ping is default mode

BRKSPG-2900 173


hw-module fib ipv4 scale host-optimized-disablehw-module fib ipv6 scale internet-optimized-disable

Using uRPF Jericho w/o eTCAM• uRPF features requires 2 LEM lookup

• You need to disable optimization modes• Host optimized is enabled by default • Needs to be explicitly deactivated for both IPv4 and IPv6

• Line card should be reloaded to active these new profiles• “In order to activate this new scale, you must manually reload the chassis/all line cards”

• Current Internet view will no longer fit in these Jericho-based non-SE systems

LEMLookup

LPMLookup

/32 /31 à /0

IPv4

IPv6 LPMLookup

LEMLookup

MPLSMAC

BRKSPG-2900 174


Configuring uRPF Jericho w/o eTCAM• Configuration will be:

hw-module fib ipv4 scale host-optimized-disablehw-module fib ipv6 scale internet-optimized-disable!interface HundredGigE0/7/0/0cdpipv4 address 192.168.1.1 255.255.255.252ipv4 verify unicast source reachable-via anyipv6 verify unicast source reachable-via anyipv6 address 2001:10:1::1/64!

BRKSPG-2900 175


RP/0/RP0/CPU0:NCS5508-1-631(config)#hw-module tcam fib ipv4 scaledisableRP/0/RP0/CPU0:NCS5508-1-631(config)#commit

IPv4 Route IPv4 Route












80b

Double capacity mode disabled

Double capacity mode

Using uRPF Jericho w/ eTCAM• eTCAM Double Capacity mode must be disabled explicitly

• 80b database where each line is split in two blocks of 40b by default• We need 2 IP lookup for uRPF (source/dest), Jericho cannot do it• We need to disable double capacity mode:

BRKSPG-2900 176

BRKSPG-2900.pdf - Cisco Live

Documents

Transcript of BRKSPG-2900.pdf - Cisco Live