#CLUS
David Roten, Technical Marketing Engineer
BRKARC-2031
QoS Configuration Migrations From Classic IOS to IOS XE
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Cisco Spark
Questions? Use Cisco Spark to chat with the speaker after the session
Find this session in the Cisco Events App
Click “Join the Discussion”
Install Spark or go directly to the space
Enter messages/questions in the space
How
Cisco Spark spaces will be available until June 28, 2018.
cs.co/ciscolivebot#BRKACC-2031
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
1
2
3
4
BRKARC-2031 3
Agenda
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
• Introduction
• What is new with QoS in IOS XE
• IOS XE vs IOS Classic behavior differences
• Platform specific caveats for IOS XE
• Specific use case migrations from IOS classic to IOS XE
• Conclusion
BRKARC-2031 4
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Coverage and prerequisites
• We will be covering QoS on the routing platforms. We will not be covering switching platforms.
Please be familiar with the concepts of shaping, queuing, policing and MQC style configuration.
BRKARC-2031 5
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Introduction
• Multicore forwarding
• More forwarding horsepower prevents masking of problems by linear processing
• More scheduling parameters for detailed control
• Identical configurations give different behavior on different platforms
• More rules for constructing hierarchies
• Added flexibility for non-traditional hierarchies in QoS
• Etherchannel QoS combination
• No longer are you restricted to VLAN balancing or member link QoS
• Multiple platforms behave the same way now
• Branch and headend routers now run the same code with the same behavior
BRKARC-2031 6
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
QoS data path forwarding
TCAMResource memory
Policer assist
BQS pkt buffers
Dispatcher / buffers
QFP forwarding complex
QFP schedulingcomplex
Interconnect
ASR1000 QFP hardware engine handles packet forwarding. In ISR4000 routers this is done via software using the same microcode.
A. Available core is allocated for packet processing (ingress QoS, police, WRED drops, mark, classify)
B. Based on default and user configurations, packets are scheduled for transmission based on the egress physical interface. All scheduling and queuing happens in this second phase.
A
B
BRKARC-2031 7
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
What’s new with QoS in IOS XE
• 2 versus 3 parameter scheduler
• Queue-limit options
• 2 levels of priority
• Aggregate Etherchannel QoS
• Tunnel QoS configurations
• Performance licenses
• Service-fragments
• Service groups
• Shaping parameters ignored
• Priority versus PAK_PRI
• Fair-queue scaling
BRKARC-2031 9
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
IOS XE QoS – 3 parameter scheduler
• IOS XE provides an advanced 3 parameter scheduler
• Minimum - bandwidth
• Excess - bandwidth remaining
• Maximum- shape
• 3 parameter schedulers share excess bandwidth equally in default configuration
• bandwidth and bandwidth remaining may not be configured in the same policy-map
BRKARC-2031 11
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
What are the three parameters?
2 Mb/sec
6 Mb/sec
5 Mb/sec
25
Mb
/s
6 Mb/sec
policy-map child
class voice
priority level 1
police cir 2000000 (bit/sec)class critical_services
bandwidth 5000
class internal_services
shape average percent 100
class class-default
!
policy-map parent
class class-default
shape average 25000000
service-policy child
Maximum is implemented by shapers. Traffic rates beyond the shaper rates will be held in queues.
Excess is defined by the bandwidth remaining
command. Excess is the amount of bandwidth available once all the minimums guarantees are satisfied. By default, classes have a remaining ratio of 1 even if bandwidth remaining is not configured.
Minimum is defined by the bandwidth
command or priority classes with policers. Classes with these directives are guaranteed to receive at least and maybe more bandwidth.
Minimums
Excess
Maxim
um
Minimum is defined by the bandwidth
command or priority classes with policers. Classes with these directives are guaranteed to receive at least and maybe more bandwidth.
6 Mb/sec
BRKARC-2031 12
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
• 2 parameter schedulers share excess bandwidth proportionally
if the above policy-map is overdriven, the 80% of the remaining bandwidth is proportionally split amongst all three queues
12% + (12 / (12+6+2) * 80%) = 60%
3 versus 2 parameter scheduler behavior
0
10
20
30
40
50
60
70
80
90
100
3 param 2 param
c-d excess
D2 excess
D1 excess
class-default
D2
D1
Minimumguarantee
3 parameter schedulers share excess bandwidth equally
if the above policy-map is overdriven, the 80% of the remaining bandwidth is equally split amongst all three queues
12% + (80% / 3) = 12% + 26.67% = 38.67%
policy-map test
class D1bandwidth percent 12 class D2bandwidth percent 6
class class-default
bandwidth percent 2
Exc
ess
bandw
idth
BRKARC-2031 13
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
2 ways to manage bandwidth remaining (excess)
• The bandwidth remaining (BR) adjust sharing of excess bandwidth
• Two options are available:
• bandwidth remaining ratio X, where X ranges from 1 to 1000, with variable base
• bandwidth remaining percent Y, where Y ranges from 1 to 100, with fixed base of 100
• bandwidth remaining percent (BR%) based allocations remain the same as classes are added to a configuration
• bandwidth remaining ratio (BRR) based allocations adjust as more queuing classes are added to a configuration with or without BRR configured
• base changes as new classes are added with their own ratios defined or with a default of 1
• By default, all classes have a bandwidth remaining ratio or percent value of 1
BRKARC-2031 14
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
• 3 parameter schedulers share excess bandwidth equally after satisfying minimum requirements
• 2 parameter schedulers share excess bandwidth proportionally based on the minimum rate configured
Gives even more of an advantage to traffic that was given a heavy weight in the first place
• To get 2 parameter behavior from IOS XE scheduler, use BRR configuration
This grants no minimums but the excess sharing will provide the same type of behavior and be proportional from the satisfied minimum to max rate which is essentially the same as what the 2 parameter schedule provides with minimum + proportional excess.
3 versus 2 parameter scheduler behavior
BRKARC-2031 15
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Required minimums
• 1st tier guarantees
• priority classes (with policers that are a hard limit on bandwidth)
• bandwidth classes
• 2nd tier guarantees (aka excess sharing)
• Only applies after minimum requirements from 1st tier have been satisfied
• By default equal sharing
• Sharing can be modified with bandwidth remaining command
• In isolation acts like Classic IOS
BRKARC-2031 16
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
QoS 3 parameter scheduler – minimumpolicy-map child
class voice
priority level 1
police cir 1000000 (bit/sec)
class critical_services
bandwidth 10000 (kbit/sec)
class internal_services
bandwidth 10000 (kbit/sec)
class class-default
bandwidth 1000 (kbit/sec)
!
policy-map parent
class class-default
shape average 25000000
service-policy child
10 Mb/sec
10 Mb/sec
10 Mb/sec
10 Mb/sec
1 Mb/sec
3 Mb/sec
10 Mb/sec
10 Mb/sec
1 Mb/sec
Inje
cte
d t
raff
ic
Satisfy minsCalculate
excess sharing9 Mb/s
6 Mb/s
25
Mb
/s
dro
pped
dro
pped
Required m
inim
um
s
BRKARC-2031 17
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
QoS 3 parameter scheduler – excess ratiopolicy-map child
class voice
priority level 1
police cir 1000000 (bit/sec)
class critical_services
bandwidth remaining ratio 4
class internal_services
bandwidth remaining ratio 1
class class-default
bandwidth remaining ratio 1
!
policy-map parent
class class-default
shape average 25000000
service-policy child
10 Mb/sec
5 Mb/sec
15 Mb/sec
10 Mb/sec
1 Mb/sec
9.5 Mb/sec
5 Mb/sec
0.5 Mb/sec
5.5 Mb/sec
9 Mb/sec
25
Mb
/s
dro
pped
dro
pped
dro
pp
ed
9.5 Mb/secIn
jecte
d t
raff
ic
Satisfy minsCalculate
excess sharing
Required m
inim
um
s
4:1:1 ratio on 24 Mb/sec16:4:4 Mb/sec
BRKARC-2031 18
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
QoS 3 parameter scheduler – excess %policy-map child
class voice
priority level 1
police cir 1000000 (bit/sec)
class critical_services
bandwidth remain percent 40
class internal_services
bandwidth remain percent 10
class class-default
!
!
policy-map parent
class class-default
shape average 25000000
service-policy child
10 Mb/sec
5 Mb/sec
15 Mb/sec
10 Mb/sec
1 Mb/sec
9 Mb/sec
5 Mb/sec
6 Mb/sec
9 Mb/sec
25
Mb
/s
dro
pped
dro
pp
ed
10 Mb/sec
Inje
cte
d t
raff
ic
Satisfy minsCalculate
excess sharing
Required m
inim
um
s
40: 10:50 ratio on 24 Mb/sec9.6:2.4:12 Mb/sec
BRKARC-2031 19
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
BR% vs BRR behavior
• Each class is driven with a packet stream which exceeds the parent shaper rate
• BR% offers class-default 80% since it is based on a total pool of 100
• BRR offers class-default only 4.5% since it is based on a pool of 21 (12+6+2+1)
policy-map test
class D1
bandwidth remaining percent 12
class D2
bandwidth remaining percent 6
class D3
bandwidth remaining percent 2
D1 D2 D3 c-d
BR percent 12% 6% 2% 80%
BR ratio57% 29% 9.5% 4.5%
12 / 21 6 / 21 2 / 21 1 / 21
policy-map test
class D1
bandwidth remaining ratio 12
class D2
bandwidth remaining ratio 6
class D3
bandwidth remaining ratio 2
BRKARC-2031 20
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Queue-limit options
• Classic IOS offered only units of packets for defining queue limits
• IOS XE on ASR1000 offers packets and also offers time and byte based
• ISR4000 and CSR1000V allows packet based configuration only
• Time and byte based are essentially the same thing since time is simply changed into bytes based on the speed of the interface (or parent shaper)
• 150 ms latency on GigabitEthernet interface and OC3 POS
0.150 𝑠𝑒𝑐 ×1𝐸9 𝑏𝑖𝑡𝑠
𝑠𝑒𝑐×
𝑏𝑦𝑡𝑒
8 𝑏𝑖𝑡𝑠= 18,750,000 𝑏𝑦𝑡𝑒𝑠
0.150 𝑠𝑒𝑐 ×155𝐸6 𝑏𝑖𝑡𝑠
𝑠𝑒𝑐×
𝑏𝑦𝑡𝑒
8 𝑏𝑖𝑡𝑠= 2,960,250 𝑏𝑦𝑡𝑒𝑠
BRKARC-2031 22
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Queue-limit options
• Time units allows flexibility to have a single policy-map work for multiple interfaces at different speeds
• Using time / bytes based units gives a consistent latency profile
• The latency associated with 4000 packets is much different if those packets are 64 or 1500 bytes
• With a GigE interface the difference is 2 msec vs. 48 msec (64 vs. 1500 byte packets)
4000 𝑝𝑎𝑐𝑘𝑒𝑡𝑠 × 64𝑏𝑦𝑡𝑒𝑠
𝑝𝑎𝑐𝑘𝑒𝑡× 8
𝑏𝑖𝑡𝑠
𝑏𝑦𝑡𝑒÷ 1𝐸9
𝑏𝑖𝑡𝑠
𝑠𝑒𝑐𝑜𝑛𝑑= 2 𝑚𝑠𝑒𝑐
4000 𝑝𝑎𝑐𝑘𝑒𝑡𝑠 × 1500𝑏𝑦𝑡𝑒𝑠
𝑝𝑎𝑐𝑘𝑒𝑡× 8
𝑏𝑖𝑡𝑠
𝑏𝑦𝑡𝑒÷ 1𝐸9
𝑏𝑖𝑡𝑠
𝑠𝑒𝑐𝑜𝑛𝑑= 48 𝑚𝑠𝑒𝑐
BRKARC-2031 23
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Queue-limit optionspolicy-map queue-limit
class class-default
queue-limit 150 ms
Rtr#show policy-map int GigabitEthernet0/3/0
GigabitEthernet0/3/0
Service-policy output: queue-limit
Class-map: class-default (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: any
queue limit 150 ms/ 18750000 bytes
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 0/0
Rtr#show policy-map int Serial1/1/0
Serial1/1/0
Service-policy output: queue-limit
Class-map: class-default (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: any
queue limit 150 ms/ 828937 bytes
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 0/0
• All queue-limit units in a policy and its related hierarchy must be the same units.
• Precludes modification on the fly, must remove, modify and reapply
BRKARC-2031 24
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
IOS XE – queuing memory management
• Packet Buffer DRAM treated as one large buffer pool
• QFP has specific pool of memory for packet buffering, ISR4400 platforms use shared FP memory, ISR4300 platforms use common memory for control and data planes
• Two main building blocks for packet buffer DRAM on ASR1000
Block: each queue gets 1 Kbyte blocks of memory
Particle: packets are divided into 16 byte particles and linked together
• Advantages to such an implementation:
Less complex than buffer carving schemes
Fragmentation is minimal & predictable due to small sized blocks & particles
• Thresholds exist to protect internal control traffic and priority traffic
BRKARC-2031 25
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Default queue-limit
• Classic IOS used 64 packets as the default queue-limit for all queues
• IOS XE uses 512 packets for priority queues and 50ms (with one exception of 25ms on ESP-40) of MTU sized packets for all other queues, with a strict minimum of 64 packets
• Don’t adjust priority queue limits unless you have a really good reason
• IOS XE platforms ignore the following buffering parameters for interface QoS
• interface hold-queue
• global IOS buffers configuration
• IOS displays the affects of the commands, but no effect on traffic forwarding or QoS behavior
BRKARC-2031 26
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
• Without MQC QoS configuration, interface default queues have 50 ms of buffering in a packets based configuration (except on ASR1000 with ESP-40 which uses 25 ms)
𝑞𝑢𝑒𝑢𝑒_𝑙𝑖𝑚𝑖𝑡𝑝𝑎𝑐𝑘𝑒𝑡𝑠 =𝑖𝑛𝑡𝑒𝑟𝑓𝑎𝑐𝑒_𝑠𝑝𝑒𝑒𝑑𝑏𝑖𝑡𝑠/𝑠𝑒𝑐 × 0.050𝑠𝑒𝑐
𝑖𝑛𝑡𝑒𝑟𝑓𝑎𝑐𝑒_𝑚𝑡𝑢𝑏𝑦𝑡𝑒𝑠/𝑝𝑎𝑐𝑘𝑒𝑡 × 8𝑏𝑖𝑡𝑠𝑏𝑦𝑡𝑒
• MQC created queues have 50 ms of buffering by default in a packets based configuration (on all IOS XE platforms). The speed used however depends…
𝑞𝑢𝑒𝑢𝑒_𝑙𝑖𝑚𝑖𝑡𝑝𝑎𝑐𝑘𝑒𝑡𝑠 =𝑠𝑝𝑒𝑒𝑑𝑏𝑖𝑡𝑠/𝑠𝑒𝑐 × 0.050𝑠𝑒𝑐
𝑖𝑛𝑡𝑒𝑟𝑓𝑎𝑐𝑒_𝑚𝑡𝑢𝑏𝑦𝑡𝑒𝑠/𝑝𝑎𝑐𝑘𝑒𝑡 × 8𝑏𝑖𝑡𝑠𝑏𝑦𝑡𝑒
IOS XE – queue limit management
BRKARC-2031 27
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
IOS XE – queue limit management
• Default queue limits are based on 50ms (except for ASR1000 ESP-40 at 25ms)
• If bandwidth parameter is configured, then that rate is used as the speed value
• If bandwidth percent parameter is configured, then that rate based on the percentage from the parent is used as the speed value
• If a shape parameter is configured, then that rate is used as the speed value
• If only bandwidth remaining is configured, then the parent’s speed value is used
𝑞𝑢𝑒𝑢𝑒_𝑙𝑖𝑚𝑖𝑡𝑝𝑎𝑐𝑘𝑒𝑡𝑠 =𝑠𝑝𝑒𝑒𝑑𝑏𝑖𝑡𝑠/𝑠𝑒𝑐 × 0.050𝑠𝑒𝑐
𝑖𝑛𝑡𝑒𝑟𝑓𝑎𝑐𝑒_𝑚𝑡𝑢𝑏𝑦𝑡𝑒𝑠/𝑝𝑎𝑐𝑘𝑒𝑡 × 8𝑏𝑖𝑡𝑠𝑏𝑦𝑡𝑒
BRKARC-2031 28
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
IOS XE – queue limit managementpolicy-map child
class p7
priority
police cir percent 2
class p6
bandwidth 10000
shape average 200000000
class p5
shape average 200000000
class class-default
!
policy-map parent
class class-default
shape average 800000000
service-policy child
𝑞𝑢𝑒𝑢𝑒_𝑙𝑖𝑚𝑖𝑡𝑝𝑎𝑐𝑘𝑒𝑡𝑠 =𝑠𝑝𝑒𝑒𝑑𝑏𝑖𝑡𝑠/𝑠𝑒𝑐 × 0.050𝑠𝑒𝑐
𝑖𝑛𝑡𝑒𝑟𝑓𝑎𝑐𝑒_𝑚𝑡𝑢𝑏𝑦𝑡𝑒𝑠/𝑝𝑎𝑐𝑘𝑒𝑡 × 8𝑏𝑖𝑡𝑠𝑏𝑦𝑡𝑒
priority classes always have a default queue depth of
512 packets
bandwidth classes use the configured bandwidth value to
come up with Τ10𝐸6×0.0501500 ×8 = 41 → 64 𝑝𝑎𝑐𝑘𝑒𝑡𝑠
shape only classes use the configured shape value to
come up with Τ200𝐸6×0.0501500 ×8 = 832 𝑝𝑎𝑐𝑘𝑒𝑡𝑠
BRKARC-2031 29
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
IOS XE – queue limit managementpolicy-map child
class p7
priority
police cir percent 2
class p6
bandwidth remaining ratio 20
shape aver 30000000
class p5
bandwidth remaining ratio 10
class class-default
!
policy-map parent
class class-default
shape average 800000000
service-policy child
𝑞𝑢𝑒𝑢𝑒_𝑙𝑖𝑚𝑖𝑡𝑝𝑎𝑐𝑘𝑒𝑡𝑠 =𝑠𝑝𝑒𝑒𝑑𝑏𝑖𝑡𝑠/𝑠𝑒𝑐 × 0.050𝑠𝑒𝑐
𝑖𝑛𝑡𝑒𝑟𝑓𝑎𝑐𝑒_𝑚𝑡𝑢𝑏𝑦𝑡𝑒𝑠/𝑝𝑎𝑐𝑘𝑒𝑡 × 8𝑏𝑖𝑡𝑠𝑏𝑦𝑡𝑒
priority classes always have a default queue depth of
512 packets
Classes with shape use the configured shape value as
follows:Τ30𝐸6×0.0501500 ×8 = 125 𝑝𝑎𝑐𝑘𝑒𝑡𝑠
Bandwith remaining only classes use the parent shape value 8 Mbit/sec as follows:
Τ8𝐸6×0.0501500 ×8 = 33,333 𝑝𝑎𝑐𝑘𝑒𝑡𝑠
bandwidth remaining only classes use the parent
shape value 8 Mbit/sec as follows:Τ8𝐸6×0.0501500 ×8 = 33,333 𝑝𝑎𝑐𝑘𝑒𝑡𝑠
BRKARC-2031 30
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
2 levels of priority
• IOS Classic offers a single level of priority
• IOS XE offers 2 levels of low latency queuing
• IOS Classic priority configuration migrates forward without modification
• It is recommended to utilize priority level 1 for voice and level 2 for video
• Priority level 1 queues are always drained before priority level 2 queues.
BRKARC-2031 32
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Priority differences
• IOS XE: Naked / strict priority consumes ALL available bandwidth from the parent
• No other classes in that policy can use bandwidth command to reserve a minimum
• IOS Classic: Naked / strict priority consumes 99% of all available bandwidth from the parent, 1% reserved for all other classes in aggregate
• IOS Classic priority commands with policer percent, use the parent’s minimum rate for calculation. (IOS XE QoS uses the parent’s maximumrate.)
• IOS Classic limits priority throughput to the minimum guarantee of the parent during physical interface congestion.
BRKARC-2031 33
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Priority behavior differencesIOS Classic IOS XE
Strict Priority
Maximum throughput is minimum guaranteed for the particular child
Maximum throughput is up to the parent maximum
By default, minimum guaranteed is physical bandwidth / number of classes. Minimum can be configured in the parent.
Minimum cannot be configured in parent. Control with bandwidth remaining ratio instead.
If parent oversubscribed, each child will get up to its minimum of priority traffic
If parent oversubscribed, each child will get proportional bandwidth according to the parent excess configuration
Priority percent
Maximum throughput during congestion is calculated based on the parent minimum or maximum value, whichever is less.
Maximum congestion throughput is always calculated on the parent maximum
n/a If physical interface is oversubscribed, each child will get proportional bandwidth according to the parent excess configuration
Police percent
Determined by maximum rate Always determined by maximum rate
When used in conjunction with priority, the percentage of the minimum guaranteed is deducted from bandwidth available to other classes.
Minimum guaranteed for priority propagation uses rate determined from the maximum.
BRKARC-2031 34
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
IOS XE Etherchannel QoS support
• With VLAN based load balancing1. Egress MQC queuing configuration on Port-channel sub-interfaces
2. Egress MQC queuing configuration on Port-channel member link
3. Policy Aggregation – Egress MQC queuing on sub-interface
4. Ingress policing and marking on Port-channel sub-interface
5. Egress policing and marking on Port-channel member link
6. Policy Aggregation – Egress MQC queuing on main-interface (XE2.6 and higher)
• Active/standby with LACP (1+1)7. Egress MQC queuing configuration on Port-channel member link (XE2.4 and higher)
9. Egress MQC queuing configuration on PPPoE sessions, model D.2 (XE3.7 and higher)
10. Egress MQC queuing configuration on PPPoE sessions, model F (XE3.8 and higher)
• Etherchannel with LACP and load balancing (active/active)8. Egress MQC queuing configuration supported on Port-channel member link (XE2.5 and higher)
11. Aggregate Etherchannel QoS support on Port-channel main-interface (XE3.12 and higher)
BRKARC-2031 36
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
IOS XE Aggregate GEC QoS
• Without μcode gymnastics, all queuing QoS hierarchies must be rooted to a physical interface
• Runs in direct opposition to the concept for a logical Etherchannel interface since multiple interfaces need QoS in aggregate
platform qos port-channel-aggregate #
• Aggregate GEC QoS creates a new hierarchy that is rooted on a logical recycle interface
• After packets run through the aggregate schedule, they are recycled for an abbreviated run through the PPEs for Etherchannel processing. Queued again through internally defined member-link hierarchies.
BRKARC-2031 37
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
IOS XE Aggregate GEC QoS
• Policy-maps can be attached to Port-channel main-interface,sub-interfaces, and service-groups on the Port-channel for input and output directions
• No restrictions for contents of the policy-map other than what exist already for GigabitEthernet interfaces
• For example, all of the following are supported:
• 3 levels of hierarchy (queuing + nonqueuing)
• 2 levels of priority
• WRED
• fair-queue, plus other features
BRKARC-2031 38
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
IOS XE Aggregate GEC QoS
• Aggregate GEC member-links are configured internally with high and low priority queues.
• Traffic classified as high priority in the aggregate GEC policy, are enqueuedthrough the high priority queues on the member-link hierarchies.
• Priority levels 1 and 2 in the user-defined policy-map on the Port-channel main-interface will use the same priority queue on the physical interface
BRKARC-2031 39
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
IOS XE Aggregate GEC QoS
• If Port-channel does not have enough bandwidth to service all configured minimums, policy-map will go into suspended mode
• For example, there are more than 1Gb/sec of minimums and a 2 member Port-channel interfaces has one of the member links go down
• Once enough member links are available to provide the guaranteed minimums, service-policy will leave suspended mode and go into effect.
• Support for TenGigabitEthernet interfaces as ofXE 3.16.3 and XE 16.3
• Support for ISR4000 series as of IOS XE 16.6.1
BRKARC-2031 40
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
IOS XE Aggregate GEC QoS
• If one of the member links is overdriven, it will exert back pressure on the aggregate hierarchy
• Flow based load balancing could be lopsided and one of the member links is utilized at 100% and its egress queues fill up
• When this happens, backpressure is exerted on the aggregate hierarchy and all traffic will be queued even if it is destined for an underutilized member link
• This happens because the flow base load balancing happens after the queuing policy-map has been applied on the aggregate hierarchy
BRKARC-2031 41
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Aggregate GEC QoS backpressure
GigE
recycle
child
grandparent
parent
GigE
SIP
QFP PPEs
Etherchannelflow load balancing
QFP PPEs
Feature processing
Feedback from Ethernet to grandparent happens fast enough that the grandparent can slow down before Ethernet starts tail dropping.
Available bandwidth, but not available due to recycle queue under backpressure.
BRKARC-2031 42
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Basic IOS XE tunnel QoS rules
• IOS XE
• Tunnel hierarchies have to be rooted to a physical interface in the hardware
• This is the source of most of the restrictions with tunnel QoS
• In IOS XE, traffic can only be queued once. So if there isn’t magic glue to link together a queueing policy-map on the tunnel to the egress physical interface then the configuration is not supported.
• The only supported case is physical interface with class-default only shaper
• Classic IOS
• Supports a full queuing policy-map on the tunnel, and then another full queuing policy-map with user based queues on the physical interface.
• Queues the packet twice
• Classifies the packet twice
BRKARC-2031 44
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
IOS XE Egress GRE tunnel QoS Details
• Maximum of two level policy-map hierarchies allowed on GRE tunnels.
• Encryption is executed prior to egress queuing.
• In XE2.5 software and later, traffic is classified and prioritized by the tunnel QoS policy-map before being enqueued for cryptography. Packets marked as priority levels 1 and 2 will move through a high priority queue for the crypto hardware.
Interface and Tunnel Policy Combination Use Cases
physical interface QoSpolicy with queuing actions
physical interface QoSpolicy without queuing actions
tunnel policy-mapwith queuing actions
Class-default only policy-map on physical interface supported.Packets will be managed by tunnel policy then rate limited by the interface policy (class-default only).
Tunnel packets bypass interface policy-map
tunnel policy-mapwithout queuing actions
Tunnel packets go through tunnel policy-map fully and then through interface policy-map (class-default only)
Tunnel packets go through tunnel policy-map fully and then through interface policy-map (interface default queue). Interface policy-map has no effect. Traffic is only classified once in IOS XE.
BRKARC-2031 45
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Per SA QoS for DMVPN
• Single tunnel interface configuration with multiple QoS policies available
• Hub QoS hierarchies have the these
• 2 level queuing hierarchy
• Parent can have only class-default with shaper / bandwidth remaining
• Additional support for physical interface class-default only policy-map
• If egress interface changes, QoS on DMVPN tunnel stays intact after switching physical egress interfaces
• Supports up to 4000 DMVPN tunnels with QoS
• RP2 / RP3 / 1001-X / 1002-X / 1002-HX for this scale, other hardware varies
BRKARC-2031 46
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
policy-map child
class voice
police cir 100000
priority
class critical_data
bandwidth remaining ratio 10
class class-default
bandwidth remaining ratio 1
!
policy-map parent-15meg
class class-default
shape average 15000000
bandwidth remaining ratio 15
service-policy child
!
policy-map parent-10meg
class class-default
shape average 10000000
bandwidth remaining ratio 10
service-policy child
!
policy-map gr-parent
class class-default
shape average 10000000
Per SA QoS for DMVPN config
Client identifies a
specific NHRP group
and matching QoS
policy is applied.
Client is manually
configured with the
matching NHRP group.
Grandparent shaper is
allowed on egress
interface for DMVPN
tunnels.
interface Tunnel 1
ip nhrp map group 10meg service-policy output parent-10meg
ip nhrp map group 15meg service-policy output parent-15meg
ip nhrp map group 20meg service-policy output parent-20meg
!
interface GigabitEthernet0/0/0
service-policy output gr-parent
BRKARC-2031 47
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Adaptive QoS for DMVPN
• Gives you the ability to have tunnels that dynamically adjust their shaping rate. Ideal for scenarios where the transmit rate is limited by conditions receiving end of the tunnel (DSL conditions, saturated DOCSIS networks)
• Supported only on DMVPN tunnels
• Limited to configuration of class-default only parent of egress policy-map
• Must use NHRP group to apply policy-map as NHRP is used to communicate information about available bandwidth
BRKARC-2031 48
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Adaptive QoS
policy-map example
class class-default
shape adaptive upper-bound 50M lower-bound 30M
!
interface tunnel 0
ip address 192.168.1.1 255.255.255.0
ip nhrp authentication cisco123
ip nhrp map multicast dynamic
ip nhrp network-id 1
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel protection ipsec profile cisco
nhrp map group 1 service-policy output example
BRKARC-2031 49
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
DSCP Tunnel Marking examplepolicy-map change_marking
class class-default
set precedence tunnel 6
!
interface Tunnel 1
ip address 1.0.0.1 255.255.255.252
tunnel source 20.0.0.1
tunnel destination 30.0.0.2
service-policy output change_marking
!
interface GigabitEthernet0/0/0
ip address 30.0.0.1 255.255.255.0
L2
header
IP header
IP prec=3payload
L2
header
IP header
IP prec=6payload
GRE
header
IP header
IP prec=3
Packet at ingress
Packet at egress
BRKARC-2031 50
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Performance License
RootPerformance License
shaper node
External physical
interfaces
Internal “physical
Interfaces”
Queues
Punt to control plane
To service container
To front panel GigE intf
To UCS-E module
• Performance licenses on the ASR1000 and ISR4000 routers are implemented by additional high level QoS nodes inserted into the queueing hierarchies
• Performance licenses affect all external physical interfaces
• Includes switch modules and UCS-E modules in ISR4000s
• Does NOT include any traffic punted to the control plane or service containers
• Does NOT include traffic that stays within a switch module or UCS-E module
BRKARC-2031 52
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Verify license rate in the hardware
R#show plat hard qfp active infra bqs int GigabitEthernet0/0/0 hier detail | beg License
Index 1 (SID:0x0, Name: License Node)
Software Control Info:
sid: 0x3f011, parent_sid: 0x3fc03, obj_id: 0x6, parent_obj_id: 0x5
evfc_fc_id: 0xffff, fc_sid: 0x3f011, num_entries (act): 8, service_frag: False
num_children: 9, total_children (act/inact): 8, presize_hint: 0
debug_name: License Node
sw_flags: 0x080203ca, sw_state: 0x00000801, port_uidb: 0
orig_min : 0 , min: 5000000000
min_qos : 0 , min_dflt: 5000000000
orig_max : 0 , max: 5000000000
max_qos : 0 , max_dflt: 5000000000
share : 500
BRKARC-2031 53
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Performance licensing gotcha workarounds
• With SourceFire on a UCS-E module, traffic is routed to UCS-E module and then back out physical interface. This double counts against the license.
• Use policer based licensing instead. Policer can distinguish between the UCS-E generically and UCS-E configured as SourceFire.platform hardware throughput exceed-action drop
• With Smart Licensing on the CSR1000V, the router will configure queue-limits based on default values. After the SL server upgrades the throughput, default queue depths are not adjusted.
• Configure “bandwidth qos-reference X” on interfaces for the anticipated final allowed platform bandwidth
BRKARC-2031 54
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
ASR 1000 QoS service-fragments
• Typically, hierarchies are very rigid with strict parent – child relationships
• Service-fragments allow QoS hierarchies which allow queues to be parented by an entity outside of the strict hierarchy
• Model 4Allows queue conservation in scale configurations
• Model 3Allows aggregation shaping of some traffic while having per session / sub-interface limits on other classes of traffic
BRKARC-2031 56
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
ASR 1000 QoS service-fragments model 4
57BRKARC-2031
policy-map sub-int-mod4
class EF
police 1000000
class AF4
police 2000000
class class-default fragment BE
shape average 75000000
!
policy-map int-mod4
class data service-fragment BE
shape average 128000000
class EF
priority level 1
class AF4
priority level 2
class AF1
shape average 50000000
random-detect
!
interface GigabitEthernet0/0/0
service-policy output int-mod4
!
interface GigabitEthernet0/0/0.2
service-policy output sub-int-mod4
interfaceschedule
interface default queue
SIP root schedule
interface priority queues
service-fragment BE
multiple instances of sub-interface fragment BE
Useful for queue conservation in scaled session / service deployments.
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
ASR 1000 QoS service-fragments model 3
58BRKARC-2031
policy-map int-mod3
class data service-fragment BE
shape average 128000000
class class-default
shape average 100000000
!
policy-map sub-int-mod3
class EF
police 1000000
priority level 1
class AF4
police 2000000
priority level 2
class class-default fragment BE
shape average 115000000
service-policy sub-int-child-mod3
!
policy-map sub-int-child-mod3
class AF1
shape average 100000000
class class-default
shape average 25000000
interfaceschedule
interface class-default queue
SIP root schedule
service-fragment BE
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
ASR 1000 QoS service-fragments model 3
59BRKARC-2031
policy-map int-mod3
class data service-fragment BE
shape average 128000000
class class-default
shape average 100000000
!
policy-map sub-int-mod3
class EF
police 1000000
priority level 1
class AF4
police 2000000
priority level 2
class class-default fragment BE
shape average 115000000
service-policy sub-int-child-mod3
!
policy-map sub-int-child-mod3
class AF1
shape average 100000000
class class-default
shape average 25000000
interfaceschedule
SIP root schedule
sub-interface priority queues
service-fragment BE
fragment BE
interface class-default queue
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
ASR 1000 QoS service-fragments model 3
60BRKARC-2031
policy-map int-mod3
class data service-fragment BE
shape average 128000000
class class-default
shape average 100000000
!
policy-map sub-int-mod3
class EF
police 1000000
priority level 1
class AF4
police 2000000
priority level 2
class class-default fragment BE
shape average 115000000
service-policy sub-int-child-mod3
!
policy-map sub-int-child-mod3
class AF1
shape average 100000000
class class-default
shape average 25000000
interfaceschedule
interface class-default queue
SIP root schedule
multiple sub-interface priority queues
service-fragment BE
multiple instances of fragment BE
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
What are service-groups?
• Service-groups allow linking multiple L3 sub-interfaces and L2 service instances together for the purpose of aggregated QoS
• Before service-groups
• QoS policies could be applied to individual L3 sub-interfaces, individual L2 service instances, or to ethernet main interfaces
• In order to group multiple L3 or L2 entities together for QoS, a “mega-policy” on the main interface which classified multiple vlans in the topmost layer was required.
• If various groups of vlans on the same physical interface required QoS, the configuration quickly became unmanageable.
BRKARC-2031 62
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Support for service-groups
• Added in XE3.15, released March 2015
• Supported on ASR1000, CSR1000V, and ISR4000 series platforms
• Same functionality across all above platforms
• Supported on physical interfaces and Aggregate Port-channels
• Same scalability for all platforms
BRKARC-2031 63
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
New configuration commands for service-groupspolicy-map alpha
class-default
shape average 10000000
!
interface GigabitEthernet0/0/0
service instance 11 ethernet
encapsulation dot1q 11
group 10
service instance 12 ethernet
encapsulation dot1q 12
group 10
!
interface GigabitEthernet0/0/0.13
encapsulation dot1q 13
group 10
!
interface GigabitEthernet0/0/0.14
encapsulation dot1q 14
group 10
!
service-group 10service-policy alpha
Use the group keyword to put
service instances and sub-interfaces into a service-group.
Use the service-group
command as the application point for QoS policies.
BRKARC-2031 64
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Service-group configuration
• Ingress and egress policy-maps are supported on service-groups
• Up to three levels in policy-maps (ingress and egress)
• Same hierarchy restrictions as policy-maps applied to main-interfaces
• Support for all Ethernet interfaces and Aggregate Port-channels
• No support for non-ethernet interface types
• Statistics are collected on a per service-group level and will not be available per service-group member unless explicitly classified as part of the service policy
BRKARC-2031 65
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Restrictions
• All members of a given service-group must be on the same physical interface
• A sub-interface or service instance can belong to only one service-group at time
• Sub-interfaces and service instances in a service-group can not have a policy-map applied other than on the service-group
• Tunnels with QoS egressing through service-group not supported
BRKARC-2031 66
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Disallowed configuration examplespolicy-map alpha
class-default
shape average 10000000
!
policy-map betaclass-default
shape average 10000000
!
interface GigabitEthernet0/0/0
service instance 10 ethernet
encapsulation dot1q 10
service-policy alphagroup 10
!
service-group 10
service-policy beta
Policy-map is not allowed on sub-interface or service instance when included in a service-group.
policy-map alpha
class-default
shape average 10000000
!
policy-map betaclass-default
shape average 10000000
!
interface GigabitEthernet0/0/0.10encapsulation dot1q 10
group 10
!
interface GigabitEthernet0/0/1.20
encapsulation dot1q 20
group 10
!
service-group 10
service-policy beta
Service-group members can notspan across multiple physical interfaces.
BRKARC-2031 67
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Service groups use case
• Left side branch
• Traditional deployment with a single VLAN servicing the entire branch
• Right side branch
• serviced by a single downlink across the WAN but uses VLANs (blue and yellow) to differentiate business traffic and customer BYOD traffic.
• From the headend ASR1000, it is necessary to rate limit traffic to the CPE as a whole but also make guarantees to the business class traffic over the BYOD traffic via vlanclassification
BRKARC-2031 68
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Service groups use case
GigE
vlan101i
e
vlan106
i
e
voice
business
best effort
voice
business
best effort
vlan107marketing
best effort
se
rvic
e-g
rou
p
10
00
policy-map service-group-1000
class-default
shape average 10000000
service-policy right-branch
!
policy-map right-branch
class vlan106bandwidth remaining ratio 20
service-policy workersclass vlan107
bandwidth remaining ratio 1
service-policy guests!
policy-map workersclass voice
priority
police cir 1000000
class business
bandwidth 8000
class class-default
!
policy-map guests
class marketing
bandwidth remaining ratio 10
class class-default
bandwidth remaining ratio 1 vlanX classes are basic match vlan X class-maps.
BRKARC-2031 69
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Service groups use casevlanX classes used in the following slides are basic match vlan X class-
maps.
class-map match-any vlan101
match vlan 101
!
class-map match-any vlan106match vlan 106
!
class-map match-any vlan107
match vlan 107
!
class-map match-any vlans-106-107
match vlan 106
match vlan 107
GigE
vlan101i
e
vlan106
i
e
voice
business
best effort
voice
business
best effort
vlan107marketing
best effort
se
rvic
e-g
rou
p
10
00
BRKARC-2031 70
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Service groups use casepolicy-map service-group-1000
class-default
shape average 10000000
service-policy right-branch
!
policy-map right-branch
class-map vlan106bandwidth remaining ratio 20
service-policy workersclass-map vlan107
bandwidth remaining ratio 1
service-policy guests!
policy-map workersclass-map voice
priority
police cir 1000000
class-map business
bandwidth 8000
class-map class-default
!
policy-map guests
class-map marketing
bandwidth remaining ratio 10
class-map class-default
bandwidth remaining ratio 1
policy-map left-branch
class class-default
shape average 5000000
service-policy workers
!
interface Gig0/0/0.101
encapsulation dot1q 101
service-policy left-branch
!
!
!
!
!
!
interface Gig0/0/0.106
encapsulation dot1q 106
group 1000
!
interface Gig0/0/0.107
encapsulation dot1q 107
group 1000!
service-group 1000service-policy service-group-1000
leve
l1le
vel 2
level 3
vlanX classes are basic match vlan X class-maps.
GigE
vlan101i
e
vlan106
i
e
voice
business
best effort
voice
business
best effort
vlan107marketing
best effort
se
rvic
e-g
rou
p
10
00
BRKARC-2031 71
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Service groups use case
vlan 101Hierarchy
QQ2 levels of queuing
vlans 106 and 107Hierarchy
QQQ3 levels of queuing
service-group 1000
GigE
vlan101i
e
vlan106
i
e
voice
business
best effort
voice
business
best effort
vlan107marketing
best effort
se
rvic
e-g
rou
p
10
00
BRKARC-2031 72
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Same setup without service groups
• An identical configuration without service-groups is not possible
• In order to build a QoS mega-policy on the main interface, it would require:
• 3 levels of queuing
• grandparent level that classifies based on vlan
• previous two items items are mutually exclusive
• The following slide presents that invalid configuration.
• The permitted configuration would only allow 2 levels of queuing and then a third level that could mark or police traffic only
BRKARC-2031 73
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Same setup without service groupspolicy-map GigE
class vlans-106-107
shape average 10000000
service-policy right-branch
class vlan-101shape average 5000000
service-policy workers!
policy-map right-branchclass-map vlan106
bandwidth remaining ratio 20
service-policy workersclass-map vlan107
bandwidth remaining ratio 1
service-policy guests
!
policy-map workers
class-map voice
priority
police cir 1000000
class-map business
bandwidth 8000
class-map class-default
!
policy-map guestsclass-map marketing
bandwidth remaining ratio 10
class-map class-default
bandwidth remaining ratio 1
interface Gig0/0/0.106
encapsulation dot1q 106
!
interface Gig0/0/0.107
encapsulation dot1q 107
!
interface Gig0/0/0.101
encapsulation dot1q 101
!
interface Gig0/0/0
service-policy GigE
level1
level 2
level 3
Note that this is a invalid configuration because the grandparent level is not class-default only.
GigE
vlan101i
e
vlan106
i
e
voice
business
best effort
voice
business
best effort
vlan107marketing
best effort
se
rvic
e-g
rou
p
10
00
BRKARC-2031 74
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Multiple service groups
• Multiple service groups per physical interface are allowed
• A given service group is unique per chassis and can not be reused across multiple physical interfaces
• Service group numbers are only locally significant
GigE
vlan101i
e
vlan106
i
e
voice
business
best effort
voice
business
best effort
vlan107marketing
best effort
se
rvic
e-g
rou
p
10
00
BRKARC-2031 75
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Scalability
• Up to 1000 service-groups per system
• Up to 1000 members of a single service group
• Up to 8000 sub-interfaces and service instances can be members of a service group per system
BRKARC-2031 76
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
• show running-config service-group
• show service-group {service-group-identifier | all}
• show service-group interface type number
• show service-group stats
• show service-group state
• show service-group traffic-stats
• show policy-map target service-group {service-group-identifier}
• show ethernet service instance detail
• clear service-group traffic-stats
New commands available
BRKARC-2031 77
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
• debug service-group {all | error | feature | group | interface | ipc |
member | qos | stats}
• show platform software interface {rp | fp} active [brief]
• show platform hardware qfp active interface platform {service-group if
name}
• show platform hardware qfp active feature qos interface-string {service-
group if name}
Acquire the service-group if name with the show platform software interfacecommand.
New internal commands available
BRKARC-2031 78
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Shaper parameters ignoredIOSXE(config-pmap-c)#shape average ?
<8000-10000000000> Target Bit Rate (bits/sec).
percent % of interface bandwidth for Committed information rate
IOSXE(config-pmap-c)#shape average 200000000 ?
<32-800000000> bits per interval, sustained. Recommend not to configure, algo finds the best v
account Overhead Accounting
<cr>
IOSXE(config-pmap-c)#shape average 200000000 100000 ?
<0-154400000> bits per interval, excess.
account Overhead Accounting
<cr>
• IOS XE schedulers ignore the bc and be parameters given with the shape command. Any special tuning done for classic IOS shapers will be lost when brought to IOS XE.
• Policer bc and be parameters are used and behavior is the same as classic IOS.
BRKARC-2031 80
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
What is PAK_PRI
• Internally generated packets considered so important that they are always “no drop”
• Associated with protocols where reliable delivery is considered highly desirable
• Not all packets for a given protocol are considered PAK_PRI
BRKARC-2031 82
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
How is PAK_PRI handled
• PAK_PRI packets will show up in QoS class classification stats but will be queued in the interface default queues (non-ATM)
• Classification and queuing stats won’t match
• ATM interfaces have a default queue on a per VC basis
• PAK_PRI packets are never subject to dropping
• Only if physical packet memory is exhausted will PAK_PRI be dropped
• 5% of packet memory is reserved for PAK_PRI packet only
• PAK_PRI packets are not treated with LLQ unless classified into that class, and they will actually move through a low latency queue
BRKARC-2031 83
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Interface default queue for PAK_PRI
policy-map child
class AF4
bandwidth 20000
class class-default
!
policy-map parent
class class-default
shape average 30000000
service-policy child
!
interface GigabitEthernet0/0/0
!
interface GigabitEthernet0/0/0.2
service-policy output parent
interfaceschedule
interface class-default queue
PAK_PRI packets are always queued here
SIP root schedule
Sub-interface parent
Sub-interface child
PAK_PRI packets may classify in user classes and be counted but not queued
BRKARC-2031 84
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
PAK_PRI protocols
Layers 1 and 2• ATM Address Resolution Protocol Negative Acknowledgement (ARP NAK) o
ATM ARP requests• ATM host ping operations, administration and management cell(OA&M)• ATM Interim Local Management Interface (ILMI)• ATM OA&M• ATM ARP reply• Cisco Discovery Protocol• Dynamic Trunking Protocol (DTP)• Ethernet loopback packet• Frame Relay End2End Keepalive• Frame Relay inverse ARP• Frame Relay Link Access Procedure (LAPF)• Frame Relay Local Management Interface (LMI)• Hot standby Connection-to-Connection Control packets (HCCP)• High-Level Data Link Control (HDLC) keepalives• Link Aggregation Control Protocol (LACP) (802.3ad)• Port Aggregation Protocol (PAgP)• PPP keepalives• Link Control Protocol (LCP) Messages• PPP LZS-DCP• Serial Line Address Resolution Protocol (SLARP)• Some Multilink Point-to-Point Protocol (MLPP) control packets (LCP)
IPv4 Layer 3 • Protocol Independent Multicast (PIM) hellos • Interior Gateway Routing Protocol (IGRP) hellos • OSPF hellos • EIGRP hellos • Intermediate System-to-Intermediate System (IS-IS) hellos, complete
sequence number PDU (CSNP), PSNP, and label switched paths (LSPs) • ESIS hellos• Triggered Routing Information Protocol (RIP) Ack o TDP and LDP hellos• Resource Reservation Protocol (RSVP) • Some L2TP control packets o Some L2F control packets o GRE IP Keepalive• IGRP CLNS• Bidirectional Forwarding Protocol (BFD)
IPv6 Layer 3 • Miscellaneous protocols
This list is not considered to be complete nor exhaustive and is subject to change in various versions of software.
BRKARC-2031 85
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Scaled fair-queue configurations tax resources
• On ASR1000 platforms using the 2nd generation QFP ASIC, using scaled fair-queue configurations can tax a resource called LEAF:STEM connections.
• ESP100, ESP200, ASR1001-X, ASR1001-HX, ASR1002-X, ASR1002-HX
• The more classes you have with fair-queue in a given policy-map, the more of the resource will be consumed.
Total Classes (ASR1001-X)
Classes with fair-queue
Number of times applied
Percent of LEAF:STEM free
4 1 750 12%
4 2 400 24%
4 3 300 22 %
8 3 200 30%
8 4 200 24%
BRKARC-2031 87
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Scaled fair-queue configurations tax resources
• Target is to keep “remaining” resources above 20% in order to allow headroom during configuration changes.
• Usage can be observed with this command:
ASR1001-X# show plat hard qfp active infrastructure bqs sorter memory
available
CPP 0; Subdev 0
Level:Class Total Available Remaining Utilization
============= ====== ========= ========= ===========
ROOT:ROOT 16 16 100% 100%
ROOT:BRANCH 256 256 100% 100%
ROOT:STEM 1024 1006 98% 99%
HIER:ROOT 1024 1016 99% 100%
HIER:BRANCH 16384 16247 99% 100%
HIER:STEM:INT 27632 25616 92% 100%
HIER:STEM:INTL2 1024 1024 100% 100%
HIER:STEM:EXT 36848 28468 77% 99%
LEAF:ROOT 2304 2304 100% 100%
LEAF:BRANCH 36864 36864 100% 100%
LEAF:STEM 147440 18676 12% 99%
BRKARC-2031 88
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Changes in behavior – classic to IOS XE
90BRKARC-2031
Classic IOS IOS XE
Priority (naked)priority
Can consume up to 99% of parent bandwidth up to min guarantee of parent.
Can consume up to 100% of parent bandwidth. Class-default and user classes can be completely starved. No parent min guarantee limit.
Priority (naked)priority
Cannot be used in the same policy-map with bandwidth.
priority oversubsubscription Allowed via tunnel / sub-interface config, not in a given policy-map
Police (with or without priority)police cir percent X
Percent rate is based on parent’s maximum rate.
Conditional prioritypriority percent X
Rate is based on lesser of max or min for parent.Policer is activated if parent node or physical interface is congested.
Rate is based on max rate of parent. Policer is activated if parent node or physical interface is congested.
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Changes in behavior – classic to IOS XE
91BRKARC-2031
Classic IOS IOS XE
Unallocated bandwidth(min configuration)
All granted to class-default or spread proportionally if class-default has a minimum configured.
Equally spread amongst all non-priority classes
bandwidth Can be used at any levelCan only be used at the leaf level of a hierarchy.
bandwidth remaining percent X(excess configuration)
Rate is based on lesser of parent’s maximum or minimum rate.
Rate is based on parent’s maximum rate minus any priority class minimums satisfied.
Unallocated bandwidth remaining percent X
(excess configuration)
Allocated equally amongst classes with no excess configuration.
If all classes have excess config, granted proportionally amongst all excess configured classes.
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Changes in behavior – classic to IOS XE
92BRKARC-2031
Classic IOS IOS XE
Multiple policy-maps in egress path(MPOL)
Supported
Queuing policy-map with classification on Tunnel and then again on egress physical interface
Generally no.
Certain restricted configurations are supported with hierarchical policy-map on sub-interfaces, tunnels, etc. with class-default only shaper on main-interface.
shape burst parametersDefaults to 1 second of bursting. bc and be parameters are adjustable.
Defaults to 1 ms of burst time. bcand be parameters are accepted on the CLI but have no affect on operation.
fair-queue scalabilityUp to 4096 fair queues per class, configurable.
16 fair queues per class. Scalability not configurable.
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
QFP queue distribution for ESP100 and ESP200
QFP 0
QFP 1
QFP 2
QFP 3
ASR1006 or ASR1009-X
with ESP100 & SIP40 / ELC
ASR1013 with
ESP100 & SIP40 / ELC
ASR1013 with
ESP200 & SIP40 / ELC
BRKARC-2031 94
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
QFP queue distribution for ESP100 and ESP200
QFP 0
QFP 1
QFP 2
QFP 3
ASR1006-X with
ESP100 and SIP40
ASR1009-X with
ESP100 and SIP40
ASR1002-HX
BRKARC-2031 95
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
ASR1000 ESP specific QoS specifications
Card / Chassis Packet memory Max Queues TCAM
ASR 1001-X 512 MB 16,000 10 Mb
ASR 1002-X 512 MB 116,000 40 Mb
ASR1001-HX 512 MB 116,000 40 Mb
ASR 1002-HX 1G (512 MB x 2) 232,000 * 80 Mb (40 Mb x 2)
ESP-20 256 MB 128,000 40 Mb
ESP-40 256 MB 128,000 40 Mb
ESP-100 1G (512 MB x 2) 232,000 * 80 Mb (40 Mb x 2)
ESP-200 2G (512 MB x 4) 464,000 * 80 Mb (40 Mb x 2)
* Queues must be distributed amongst physical locations in the chassis.
BRKARC-2031 96
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
ISR4000 QoS specifications
Card / Chassis Packet memory Max Queues TCAM
ISR4321 Variable 8,000 Software lookup
ISR4331 Variable 16,000 * Software lookup
ISR4351 Variable 16,000 * Software lookup
ISR4431 Variable 16,000 * Software lookup
ISR4451 Variable 16,000 * Software lookup
* Current tested maximum, scale could reach 32,000 with additional validation .
BRKARC-2031 97
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
4000 node aggregation• On platforms using the 2nd generation and newer ASICs (1001-X, 1002-X, 1002-HX,
ESP100, ESP200), there is a limit of 4000 children per QoS node
• This difference does not impact 99% of enterprise configurations
• Broadband configurations most likely affected.
• Any location which has more than 4000 children will be split into two nodes that equally divide the children in a random fashion.
• No affect on rate limiting, but there is a impact on bandwidth sharing during congestion
BRKARC-2031 98
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
4000 node aggregation• If the bottommost node in each hierarchy is
congested sharing (excess) must be used to allocate bandwidth.
• The scenario on the right will give equal sharing amongst all.
• The left scenario will allow the two sessions in red to get up to 50% of the bandwidth causing multiple sessions on the right node to suffer.
• Workaround: distribute sessions across multiple ethernet sub-interfaces with bandwidth remaining ratio configuration
One or a few very fast sessions on the left can get 1:1 sharing with thousands of slower sessions on the right. This equal sharing happens at the intermediate node inserted.
Equal sharing amongst all 4000 child nodes
One or a few very fast sessions on the left can get 1:1 sharing with thousands of slower sessions on the right. This equal sharing happens at the intermediate node inserted.
BRKARC-2031 99
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
New unexplained drops
• Previously installed 7200s with QoS worked fine. When we replaced the boxes with ASR1000s we started to see tons of drops on the egress WAN interfaces. Average utilization via CLI and SNMP never shows interface utilization high enough to cause the parent shaper to drop traffic.
• With all this fancy QoS and packet buffer memory why do we start to have these problems now?
What’s with all these drops on a better / faster box?
BRKARC-2031 102
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
New unexplained drops
policy-map EGRESS
class VoIP
priority percent 15
class PRIORITY_DATA
bandwidth remaining percent 30
class MISSION_CRITICAL_DATA
bandwidth remaining percent 40
class SCAVENGER_DATA
bandwidth remaining percent 1
random-detect
class class-default
bandwidth remaining percent 3
!
policy-map SHAPE_730Mbps
class class-default
shape average 730000000
service-policy EGRESS
What’s with all these drops on a better / faster box?Class-map: MISSION_CRITICAL_DATA (match-any)
24718716075 packets, 21181619094739 bytes
30 sec offered rate 237687000 bps, drop rate 262000 bps
Match: ip dscp cs3
Queueing
queue limit 3041 packets
(queue depth/total drops/no-buffer drops) 0/324178/0
(pkts output/bytes output) 24718391897/21181154463832
bandwidth remaining 12%
MISSION_CRITICAL_DATA should always have access to at least 730E6*85%*40% = 248 Mbit/sec.
This class is running close to that on a 30 second average which almost promises that there are going to be microbursts higher than that.
Any 10GE interfaces that feed into this 730Mb/sec uplink would amplify that affect over GigE interfaces.
BRKARC-2031 103
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Microbursts
• Newer platforms can forward traffic truly at line rate
• No delays with ingress buffering or delays
• No delays with packet processing with multiple cores servicing traffic
• Older platforms were single threaded and processed packets more or less serially compared to in parallel on the newer boxes
• When bursty traffic arrived on older platforms, those bursts were smoothed in the forwarding phase by the ingress buffering and serial processing before hitting egress QoS queuing code.
• Newer platforms do not have those “smoothers” and need more egress buffering to compensate. Sometimes the default buffers of 50ms just are not enough.
What’s with all these drops on a better / faster box?
BRKARC-2031 104
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Microbursts
Fix is 3 fold
1. Realize that the function of QoS is not to preserve all packets.It is to prioritize traffic, rate limit, and provide minimum guarantees under congestion
2. Increase queue limits to deal with bursty conditions
Consider the use of time or byte based queue-limits to have true max latency protection and maximum capacity to absorb bursts
3. Use policer markdown with WRED to tamp down TCP bursting if applicable
still drops traffic but uses better management
What’s with all these drops on a better / faster box?
BRKARC-2031 105
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Microbursts – increase queue limitsWhat’s with all these drops on a better / faster box?
policy-map EGRESS
class VoIP
priority percent 15
class PRIORITY_DATA
bandwidth remaining percent 30
queue-limit 6083
class MISSION_CRITICAL_DATA
bandwidth remaining percent 40
queue-limit 9125
class SCAVENGER_DATA
bandwidth remaining percent 1
queue-limit 12166
random-detect
police cir 7300000 pir 14600000
conform-action transmit
exceed-action transmit-set-dscp af12
violate-action transmit-set-dscp af13
class class-default
bandwidth remaining percent 3
queue-limit 12166
policy-map EGRESS
class VoIP
priority percent 15
class PRIORITY_DATA
bandwidth remaining percent 30
queue-limit 100 ms -or- queue-limit 6083
class MISSION_CRITICAL_DATA
bandwidth remaining percent 40
queue-limit 150 ms -or- queue-limit 9125
class SCAVENGER_DATA
bandwidth remaining percent 1
random-detect
queue-limit 200 ms -or- queue-limit 12166
class class-default
bandwidth remaining percent 3
queue-limit 200 ms -or- queue-limit 12166
!
policy-map SHAPE_730Mbps
class class-default
shape average 730000000
service-policy EGRESS
If these deep queues are constantly full, you have a bandwidth problem and not a bursting problem.
BRKARC-2031 106
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Bandwidth configurations
• Due to the 2 vs 3 parameter scheduler changes, minor tweaks are necessary to keep the same behavior with excess sharing
policy-map child
class voice
priority percent 15
class BUSINESS_APP_1
bandwidth percent 30
class BUSINESS_APP_2
bandwidth percent 40
class SCAVENGER
bandwidth percent 1
random-detect
class class-default
bandwidth percent 3
!
policy-map PARENT
class class-default
shape average 50000000
service-policy child
policy-map child
class voice
priority percent 15
class BUSINESS_APP_1
bandwidth remaining percent 30
class BUSINESS_APP_2
bandwidth remaining percent 40
class SCAVENGER
bandwidth remaining percent 1
random-detect
class class-default
bandwidth remaining percent 3
!
policy-map PARENT
class class-default
shape average 50000000
service-policy child
BRKARC-2031 108
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Conclusion
• Multicore forwarding is great, but….
• It exposes your network to bursty traffic. Be armed with knowledge to explain and the tools to handle it.
• More is better, with parameters that is…
• Just be sure to understand how the excess bandwidth is shared evening vs. proportionally between configurations
• All these rules…
• With hardware forwarding comes a bit more rigidity. Know which use cases are covered, where the gaps are, and who to yell at to cover use cases that are missing
• Have QoS and Etherchannel make nice
• It is finally possible to do aggregate queueing across flow based load balanced interfaces.
BRKARC-2031 110
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Cisco Webex Teams
Questions? Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session
Find this session in the Cisco Events App
Click “Join the Discussion”
Install Webex Teams or go directly to the team space
Enter messages/questions in the team space
How
Webex Teams will be moderated by the speaker until June 18, 2018.
cs.co/ciscolivebot#BRKACC-2031
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
1
2
3
4
BRKARC-2031 111
Complete your online session evaluation
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Give us your feedback to be entered into a Daily Survey Drawing.
Complete your session surveys through the Cisco Live mobile app or on www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Online.
BRKARC-2031 112
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Demos in the Cisco campus
Walk-in self-paced
labs
Meet the engineer
1:1 meetings
Related sessions
Continue your education
BRKARC-2031 113
Top Related