#CLUS
Javier Contreras Albesa. Sr Technical LeadBRKEWN-3012
Performance Monitoring on AireOS and APs
Agenda
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
• Chinese Medicine - Prevention
• KPI - Health
• Translating KPI to AireOSIOSKPI
• AP-COS KPI
• Q & A
3Presentation ID
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Cisco Webex Teams
4
Questions? Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session
Find this session in the Cisco Live Mobile App
Click “Join the Discussion”
Install Webex Teams or go directly to the team space
Enter messages/questions in the team space
How
Webex Teams will be moderated by the speaker until June 18, 2018.
cs.co/ciscolivebot#BRKEWN-3012
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
1
2
3
4
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 6BRKEWN-3012
Chinese Medicine – Prevention
• We want to stay Healthy
• Focus on prevention and good habits
• Not on “fixing” after the crash/illness
• What to monitor to ensure the Wireless network remains Healthy
Performance Monitoring as Preventive Medicine
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 7BRKEWN-3012
Key Performance Indicators
• Key Performance Indicators
• Can be used to monitor success of an organization
• Define a set of values against which to measure
• Defining KPIs for network infrastructure subsystems
• KPI as health indicators: is the network device healthy?
Looking for Symptoms
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 8BRKEWN-3012
Doctor am I sick?
• Not good: going to doctor after 5 days with 42C fever
• Preventive Detection: monitoring health overall
• Good detection:
• Memory utilization increasing every day
• Task tending to high CPU
• Radio channel tending to high utilization
• Not optimal detection:
• Uptime
• Channel meltdown
Looking for trouble before it happens
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 9BRKEWN-3012
What are Good Health Indicators?
• Resources are Finite
• CPU time
• Memory
• Timers
• ARP entries
• Client entries
• Queues
• Longer Uptimes/Capwap
• You can always exhaust resources
• Utilization remains in expected range under intended load
Where to look
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 10BRKEWN-3012
What are Good Health Indicators?
• Negative Events
• Radio Resets (not all!)
• Crashes
• Drops
• Hangs
• Not servicing clients
Where to look
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 11BRKEWN-3012
How can I obtain this?
• PI will not provide it all
• It is a more “network wide status” tool
• This is a per device check
• For the moment, scripting tools can help
Getting the data
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
0 1
Service
Port
AP Client
13BRKEWN-3012
WLC Architecture What we need to Look At
Network
Multicore CPU
(Control)
6 7
PCIe Switch
Multicore CPU
(Data)
I/O Subsystem
Control Plane
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
0 1
Service
Port
AP Client
14BRKEWN-3012
WLC Architecture What we need to Look At
Network
Multicore CPU
(Control)
6 7
PCIe Switch
Multicore CPU
(Data)
I/O Subsystem
Data Plane
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
0 1
Service
Port
AP Client
15BRKEWN-3012
WLC Architecture What we need to Look At
Network
Multicore CPU
(Control)
6 7
PCIe Switch
Multicore CPU
(Data)
I/O Subsystem
AP
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
0 1
Service
Port
AP Client
16BRKEWN-3012
WLC Architecture What we need to Look At
Network
Multicore CPU
(Control)
6 7
PCIe Switch
Multicore CPU
(Data)
I/O Subsystem
RF
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 17BRKEWN-3012
AireOS KPIsIt is a complex world
APF
(802.11)
Dot1x
(Keys/Auth)
AAA
Subsystems
PEM
MSCB OS
IP Stack
Service
Port
Driver
Data Plane
RRM
SPAM
(CAPWAPP)DTLS
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 18BRKEWN-3012
AireOS Basic Health
• Where to Put the Thermometer?
• Control Plane
• Data Plane
Does it have Fever?
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 19BRKEWN-3012
AireOS KPIs
• Queue Utilization
• Per Process CPU
• CPU per core
• DHCP stats
• Timers
• ARP Switch/Kernel
• Memory Utilization
• Mbufs
• Client Count
• PMK Count
Let’s measure – Control Plane
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 20BRKEWN-3012
AireOS KPIs
• State monitoring
• DP-CP Drops
• Fragmentation
• Packet Pools
Let’s measure - Data Plane
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 22BRKEWN-3012
AireOS KPIsCPU Monitoring
(Cisco Controller) >show cpu
Current CPU(s) load: 4%
Individual CPU load: 0%/0%, 0%/0%, 0%/1%, 100%/0%, 0%/1%, 0%/1%, 0%/1%, 0%/1%, 0%/1%, 0%/1%, 0%/1%, 0%/1%, 0%/0%, 0%/0%, 0%/1%, 0%/0%, 0%/0%, 0%/0%, 0%/0%, 0%/1%, 0%/0%, 0%/0%, 0%/0%, 0%/0%
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 23BRKEWN-3012
AireOS KPIsTask-CPU Utilization
(WLC-5508-7) >show process cpu
Name Priority CPU Use (usr/sys)% hwm CPU Reaper
System Reset Task (240/ 7) 0 ( 0/ 0)% 0 9
reaperWatcher ( 3/ 96) 0 ( 0/ 0)% 0 7 I
osapiReaper ( 10/ 94) 0 ( 0/ 0)% 0 1 I
TempStatus (240/ 7) 0 ( 0/ 0)% 0 9 I
HAPeerToPeerCommTa ( 90/ 64) 0 ( 0/ 0)% 0 6
pktDebugSocketTask (255/ 1) 0 ( 0/ 0)% 0 6
webauthRedirect ( 95/ 62) 100 ( 95/ 4)% 100 9 T 300
emWeb (240/ 7) 0 ( 0/ 0)% 100 1 T 300
CSCuo86819 WLC crashes in 7.6.120.0 - memory corruption caused by Webauth
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 25BRKEWN-3012
AireOS KPIs
• Used for messaging between tasks
• Around 128-150 queues in 8.5/8.3
• Some are critical for the well being of the WLC
(5520-2) >show queue-info
Total message queue count = 128
* Column Breached shows the number of times the queue breached the 95% mark
Q-Id Queue Name Allocated MsgSize InUse MaxUsed Breached* Details
--------------------------------------------------------------------------------------------
1 PRINTF-Q 256 80 0 0 0 N
2 GRE Queue 100 8 0 1 0 N
3 dtlqueue 4096 2 0 3 0 N
4 dtlarpqueue 4096 24 0 8 0 Y
Queue Utilization
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 26BRKEWN-3012
AireOS KPIs
• Inuse: how many messages in the queue
• MaxUsed: top message count ever
• Breached: how many times we reached 95% utilization
Queue Utilization
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 27BRKEWN-3012
AireOS KPIsQueue Utilization
(WiSM-slot1-1) >show queue-info
Q-Id Queue Name Allocated MsgSize InUse MaxUsed Breached* Details
--------------------------------------------------------------------------------------------
1 PRINTF-Q 256 80 0 0 0 N
2 GRE Queue 100 8 0 5 0 N
3 dtlqueue 4096 2 0 3 0 N
4 dtlarpqueue 4096 16 0 75 0 Y
..
70 nfv9queue 4096 316 0 2 0 Y
71 RogueApInfoChangedDB 4000 4 4000 4000 68 N
72 RogueApMeasChangedDB 4000 4 4000 4000 1 N
73 RogueClientInfoChangedDB 5000 4 5000 5000 1 N
74 RogueClientMeasChangedDB 5000 4 5000 5000 1 N
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 28BRKEWN-3012
AireOS KPIsQueue Utilization
86 RRM-DCLNT-5_0-Q 375 0 4 0 N
87 RRM-MGR-5_0-Q 375 0 20 0 N
88 RRM-PERIODIC-TASK-Q 93 0 1 0 N
89 SNTP-Q 3 2 3 20010 N
90 CDP-Q 512 0 13 0 N
91 BCAST-Q 2048 0 2 0 Y
92 BCAST-DATA-Q 512 0 2 0 Y
93 IDS Track Event Msg Queue 1024 0 8 0 N
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 29BRKEWN-3012
AireOS KPIsQueue Decoder Ring
Dtlarpqueue: Data transformation layer ARP queue
SNMP-Q: SNMP requests
Debugger/logger-q: Debug and Syslog Messages
SPAM: Messages between AP and WLC
NMSP*: location
Dot1x: all authentication (PSK/dot1x)
APF*: 802.11 client state machine
BCAST*: multicast/broadcast
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 31BRKEWN-3012
AireOS KPIsRadius Queues
Radius messages between WLC and AAA server
8.0MR3 and lower: 255 standing requests (ID)
8.0MR4 and higher: 16 queues per server (ID/Source Port)
(5520-2) >show radius queue
Max Radius Queues Per Server..................... 16
Max Radius Buffers Available..................... 8128
Currently number of Buffers consumed............ 0
Radius Authentication Messages Stats
Total Auth Req sent(allocated).................. 140
Total Auth Resp rcvd(freed)..................... 140
Total Auth Req Pkts Dropped(no buffer).......... 0
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 32BRKEWN-3012
AireOS KPIsRadius Queues
(Cisco Controller) >show radius queue
Radius Queue
Auth Alloc : 122401616
Acct Alloc : 31288769
Auth Free : 122401615
Acct Free : 31288769
Auth Alloc Err : 0
Acct Alloc Err : 0
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 33BRKEWN-3012
AireOS KPIsDHCP
DHCP NAK Count:................................ 3573
Tx Fails:...................................... 26765
Last Rx Time:.................................. 1 seconds ago
Last Tx Time:.................................. 3 seconds ago
(CH-8510W-DC-1) show dhcp stats
DHCP Proxy Behaviour: enabled
DHCP Server IP Address:.......................... 10.5.0.10
DHCP RX DISCOVER Count:........................ 3117892
DHCP TX DISCOVER Count:........................ 3175288
DHCP ACK Count:................................ 4644105
DHCP REQUEST Count:............................ 7641156
DHCP INFORM Count:............................. 200006
DHCP DECLINE Count:............................ 0
DHCP RELEASE Count:............................ 987
DHCP REPLY Count:.............................. 0
DHCP OFFER Count:.............................. 1283401
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 35BRKEWN-3012
AireOS KPIs
Memory Monitoring Memory Utilization
Increases with load, remains steady on same load level
Most tasks use OSAPI Buffer based: time, owner, trailers
MBUFs
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 36BRKEWN-3012
AireOS KPIsMemory the easy way
show memory history (8.0.132.0 and higher)(5520-2) >show memory history
-------------------------- System Memory History -------------------------
System Name ..................................... 5520-2
Software Version ................................ 8.5.124.55
Current Time .................................... Fry May 11 17:59:44 2017
System UP Time .................................. 6 days 4 hrs 12 mins 48 secs
------------------------- System Memory at BOOT ------------------------
Memory Snapshot Time ............................ Fry May 5 14:17:18 2017
Total System Memory ............................. (32386132 KB) 31627 MB
Total System Free Memory......................... (29429840 KB) 28740 MB (90 %)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 37BRKEWN-3012
AireOS KPIsMemory the easy way
Hundreds of data points------------------------- System Memory Snapshot 1 Count: 289 -------
Memory Snapshot Time ............................ Sun Jan 29 14:54:31 2017
Total System Memory ............................. (32386132 KB) 31627 MB
Total System Free Memory......................... (29408732 KB) 28719 MB (90 %)
Total Memory in Buffers.......................... (9960 KB)
Total Memory in Cache............................ (703068 KB) 686 MB
Total Active Memory.............................. (2200724 KB) 2149 MB
Total InActive Memory............................ (597932 KB) 583 MB
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 38BRKEWN-3012
AireOS KPIsMemory the easy way
MBUF Utilization
------------------------- MBUF Information ------------------------------
Maximum number of Mbufs.......................... 24576
Number of Mbufs Free............................. 24560
Number of Mbufs In Use........................... 16
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 39BRKEWN-3012
AireOS KPIsMemory the easy way
Buffers in use and Top users------------------------- Pmalloc Pools Information --------------------
Index Pool-Size Chunks-In-Pool Chunks-In-Use Memory(Size/Used/Free)KB
0 16 200000 40881 21875 /19388 /2486
1 64 160000 6794 25000 /15424 /9575
2 128 93600 31317 20475 /12689 /7785
3 256 20000 6406 6875 /3476 /3398
..
-------------- TOP-20 Pools Consuming Tasks (Time: 465689 Secs) ------------
Index Name PID 16-byte 64-byte 128-byte 256-byte 384-byte 512-byte 1024-byte 2048-byte 4096-byte Raw-Pool Total-Pools
001 fp_main_task 2182 3011 4289 155 280 55 9 124058 60 24586 1767 158270
002 emWeb 2664 37373 657 31015 5987 30 4 21 2 7 9 75105
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 41BRKEWN-3012
AireOS KPIsHappy Uptime
(5520-2) >show ap uptime
Number of APs.................................... 2
Global AP User Name.............................. admin
Global AP Dot1x User Name........................ Not Configured
AP Name Ethernet MAC AP Up Time Association Up Time
------------------ ----------------- ----------------------- -----------------------
ap1800-3850-3-11 80:e8:6f:d0:12:08 5 days, 19 h 57 m 32 s 5 days, 19 h 55 m 54 s
barbados-beta-floor1 58:ac:78:dc:d3:a0 5 days, 19 h 58 m 24 s 1 days, 04 h 53 m 52 s
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 42BRKEWN-3012
AireOS KPIsCoredump Available
(5520-2) >show coredump summary
Core Dump upload is disabled
Core Dump file on flash:
SW Version....................................... 8.2.102.89
Time Stamp....................................... Tue Apr 26 15:01:08 2016
File Size........................................ 50431374
File Name Suffix................................. _8.2.102.89_5520-2_1461682868.gz
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 43BRKEWN-3012
AireOS KPIsTime(rs) are gold
(5500-2) >show system timers summary
Total Timers..................................... 150000
Allocated.................................... 2538
Free......................................... 147462
Peak Usage................................... 2842
Alloc Operations................................. 275207
Free Operations.................................. 272669
Alloc Failures................................... 0
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 44BRKEWN-3012
AireOS KPIs5520/8540 Chassis Monitoring
(5520-2) >show imm chassis fan
FAN1_TACH1 | 0Fh | ok | 29.1 | 8200 RPM
FAN1_TACH2 | 10h | ok | 29.1 | 9600 RPM
FAN2_TACH1 | 11h | ok | 29.2 | 8200 RPM
FAN2_TACH2 | 12h | ok | 29.2 | 9600 RPM
FAN3_TACH1 | 13h | ok | 29.3 | 8200 RPM
FAN3_TACH2 | 14h | ok | 29.3 | 10000 RPM
FAN4_TACH1 | 15h | ok | 29.4 | 8200 RPM
FAN4_TACH2 | 16h | ok | 29.4 | 10000 RPM
FAN5_TACH1 | 17h | ok | 29.5 | 8200 RPM
FAN5_TACH2 | 18h | ok | 29.5 | 10000 RPM
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 45BRKEWN-3012
AireOS KPIsMy Dear Clients
(Cisco Controller) >show client state summary
Client State Summary
====================
State Number of Clients
----- -----------------
START 8
8021X_REQD 25
DHCP_REQD 54
WEBAUTH_REQD 260
RUN 146
----- -----------------
Total 493
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 46BRKEWN-3012
AireOS KPIsMy Dear Clients
(Controller_2) >show pmk-cache all
Number of PMK Cache Entries: 156
Type Station Lifetime VLAN Override IP Override Audit-Session-ID
------ -------------- -------- ------------------ --------------- ----------------
RSN 04:54:53:4b:63:5c 86301 0.0.0.0
RSN 04:54:53:4c:22:d9 57521 0.0.0.0
RSN 08:70:45:92:7d:f0 86363 0.0.0.0
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 47BRKEWN-3012
AireOS KPIsARPs(5520-2) >show arp stats
ARP statistics:
Current Entries 236
Dup Additions 382947
Bad Additions 0
Bad Deletions 0
(5520-2) >show arp kernel
IP address HW type Flags HW address Mask Device
4.0.7.161 0x1 0x2 00:00:05:00:00:04 * dtl0.4
4.0.7.122 0x1 0x2 00:00:06:00:00:09 * dtl0.4
4.0.7.117 0x1 0x2 00:00:06:00:00:1d * dtl0.4
192.168.5.104 0x1 0x2 58:ac:78:dc:d3:a0 * dtl0.5
Max=client+1000
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 48BRKEWN-3012
AireOS KPIsLAG
(5500-4-82) >show lag summary
LAG Disable is in transition. Pls Reboot the switch
This won’t happen unless configured
If you change it, reboot
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 50BRKEWN-3012
Data PlaneHappy to forward your frames
Takes care of all frame forwarding for clients
Also: AVC, Encryption, Tunneling
Closely monitored by control plane
Its own memory, CPU, software
Its own stats and performance indicators
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 51BRKEWN-3012
Data PlaneHappy to forward your frames
Main command:
(5500-2) >debug fastpath dump fpapool
FP0.01:
FP0.01:(#)Buffer Pool = avail / size (in-use) min-avail (max in-use)
..
FP0.01:Buffer overload back-pressure limits:
FP0.01:Entered _GREENZONE 1 times. _GREENLINE is 27000 buffers available.
FP0.01:Entered YELLOWZONE 0 times. YELLOWLINE is 27000 buffers available.
FP0.01:Entered ORANGEZONE 0 times. ORANGELINE is 26000 buffers available.
FP0.01:Entered ___REDZONE 0 times. ___REDLINE is 10000 buffers available.
FP0.01:Entered _BLACKZONE 0 times. _BLACKLINE is 5000 buffers available.
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 52BRKEWN-3012
Data PlaneHappiness gone
FP0.02:Entered _GREENZONE 59575 times. _GREENLINE is 27000 buffers available.
FP0.02:Entered YELLOWZONE 59575 times. YELLOWLINE is 27000 buffers available.
FP0.02:Entered ORANGEZONE 1 times. ORANGELINE is 26000 buffers available.
FP0.02:Entered ___REDZONE 0 times. ___REDLINE is 10000 buffers available.
FP0.02:Entered _BLACKZONE 0 times. _BLACKLINE is 5000 buffers available.
Green Zone: Normal Scenario
Yellow Zone: Traffic Peaks
Orange Zone: Multicast is dropped, CAPWAP is top priority
Red/Black: Heavy packet drop -> recovery through crash
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 53BRKEWN-3012
Data Plane – Control PlaneLet’s Talk(5520-2) >debug fastpath dump dpcp-stats
FP0.20:Queue TxCount RateLmtDrop PktLmtDrop
FP0.20:Q0: Internal DP --- 85587 0 0
FP0.20:Q1: Mobility Ctrl --- 19 0 0
FP0.20:Q2: Capwap --- 237449 0 0
FP0.20:Q3: Client Traffic --- 1610 0 0
FP0.20:Q4: ARP, Client services --- 33729 0 0
FP0.20:Q5: IPv6 Client services --- 16634 0 0
FP0.20:Q6: Infra services --- 42932 0 0
FP0.20:Q7: Probe --- 276962 0 0
FP0.20:Q8: IAPP, Rogue --- 85139 1 0
FP0.20:Q9: HA --- 8714 0 0
FP0.20:Q10: WebAuth --- 0 0 0
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 55BRKEWN-3012
IOS APsSimpler
IOS as monolithic entity
Main things to monitor CPU Memory Interfaces Radios CAPWAP
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 56BRKEWN-3012
IOS APsRadio Interfaces
Show controller d0 | d1
Lot of data, not all human readable
Not intended for “general use”, but treasure of information
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 57BRKEWN-3012
IOS APsShow Controller - Monster
interface Dot11Radio0Radio AL 2.4GHz, Base Address b0aa.7733.bf10, BBlock version 0.00, Software version 4.18.2Serial number: FOC19066XZD Unused dynamic SQRAM memory: 0x0000C858 (50 KB)Unused dynamic SDRAM memory: 0x00082674 (521 KB)Spectrum FW version: 1.15.4Number of supported simultaneous BSSID on Dot11Radio0: 16Carrier Set: Spain (ES) (-E)Uniform Spreading Required: NoConfigured Frequency: 2412 MHz Channel 1Serving Frequency: 2412 MHz Channel 1 20MHzAllowed Frequencies: 2412( 1) 2417( 2) 2422( 3) 2427( 4) 2432( 5) 2437( 6) 2442( 7) 2447( 8) 2452( 9) 2457( 10) 2462( 11) 2467( 12) 2472( 13) Listen Frequencies: 2412( 1) 2417( 2) 2422( 3) 2427( 4) 2432( 5) 2437( 6) 2442( 7) 2447( 8) 2452( 9) 2457( 10) 2462( 11) 2467( 12) 2472( 13) 2484( 14) Beacon Flags: 0, Interface Flags 2020105, Interface Events 20, Mode 9; Beacons are enabled; Probes are enabledConfigured TxPower: 16 dBm (Level Index 1)Allowed Power Levels:
16 13 10 7 4 1 dBmAllowed Client Power Levels: 16 13 10 7 4 1 dBmAntenna:Rx[right-a left-b middle-c ]
Tx[right-a left-b middle-c cck all ofdm all]Internal Gain [Allowed 8, In
Use 8] (dBi x 2), In Use 8] (dBi x 2), In Use 8] (dBi x 2), In Use 8] (dBi x 2), Reported 0, Configured 0, In Use 8] (dBi x 2)Active Rates: 9.0 basic-12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2 m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Current Rates: 9.0 basic-12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2 m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Allowed Rates: 1.0 2.0 5.5 6.0 9.0 11.0 12.0 18.0 24.0 36.0 48.0 54.0All Rates: 1.0 2.0 5.5 11.0 6.0 9.0 12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2 m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Default Rates: basic-1.0 basic-2.0 basic-5.5 basic-11.0 6.0 9.0 12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2 m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Best Range Rates: basic-1.0 2.0 5.5 11.0 6.0 9.0 12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2 m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Best Throughput Rates: basic-1.0 basic-2.0 basic-5.5 basic-11.0 basic-6.0 basic-9.0 basic-12.0 basic-18.0 basic-24.0 basic-36.0 basic-48.0 basic-54.0 m0-2 m1-2 m2-2 m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Current Voice Rates: basic-5.5 basic-11.0 6.0 12.0 24.0 [disabled until voice packet-discard enabled]Default Voice Rates: basic-5.5 basic-11.0 6.0 12.0 24.0Managment Rates: 9.0 basic-12.0Multicast Rates: basic-12.0 Multicast Active rate: 12.0RTS Rates: basic-12.0RTS Active rate 12.0RTS Max rate 12.0 Local Packet Rate Probing is disabled : Packet Size 20Data Rate Sensitivity (rate, SNR dB, Contention dBm)( 1.0, 10, -93) ( 2.0, 10, -92) ( 5.5, 11, -90) (11.0, 14, -90) ( 6.0, 15, -89) ( 9.0, 16, -88) (12.0, 18, -88) (18.0, 19, -86) (24.0, 20, -85) (36.0, 24, -81) (48.0, 27, -78) (54.0, 31, -77) ERP settings: short slot time.Neighbors in non-erp mode:
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 58BRKEWN-3012
IOS APsTaming the monster
Interface status
Beacon Flags: 0, Interface Flags 20105, Interface Events 0, Mode 9; Beacons are enabled; Probes are enabled
Channel set to zero
Configured Frequency: 2412 MHz Channel 1
Warn on low PoE scenario
Radio on Low Power Mode due to PoE, restricted to 3 antennas
• QBSS set to higher than 80% (0xCC)
QBSS Load: 0x24
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 59BRKEWN-3012
IOS APsTaming the monster
TX blocks - Pending
Transmit queues: Limit 13919 Current 4 In-Progress 0
---- Active ------ In-Progress -----------
Cnt Quo Bas Max Cl Cnt Quo Bas Uplink
0 64 0 0 0 0 5 0 Voice
0 512 0 0 0 0 60 0 Video
0 1024 0 0 0 0 200 0 Best
4 1024 0 0 0 0 200 0 MC0
0 0 0 0 0 0 7 0
• High Radio reset counts
Radio resets - total:14 retries:0 failed:0
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 60BRKEWN-3012
IOS APsTaming the monster
RF jammer detected
Jammer detected: 1 / 0
• Beacon Stop
Beacon stopped count: 0
Counts > 120 0
Counts > 90 0
Counts > 60 0
Counts > 30 2
Counts > 15 0
Counts > 10 0
Counts > 5 1
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 61BRKEWN-3012
IOS APsTaming the monster
• Silent radio hang
• No traffic received/sent when clients are around
Host Rx K Bytes: 2243173 / 0 Host Tx K Bytes: 3382831 / 0
Unicasts Rx: 368819860 / 0 Unicasts Tx: 457956326 / 0
Unicasts to host: 368819860 / 0 Unicasts by host: 415499647 / 0
Broadcasts Rx: 348114707 / 0 Broadcasts Tx: 354596429 / 439
Beacons Rx: 342650083 / 0 Beacons Tx: 353388822 / 439
PCI Resets
Radio PCI resets:0
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 63BRKEWN-3012
AP-COSDifferent architecture (simplified view)
Wireless Control
WCP
hostapd
Platform
Management
watchdog
LED
capwapd
Capwapbrain
Config/system mng
ssh
Device Management
Light
httpd
CLI
Click DataPath
Linux Networking
stack
Cisco
Services/clickfilesystem
Ethernet
Driver
Radio
Driver
Flash
Driver
Linux
Kernel
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 64BRKEWN-3012
AP-COSDifferent organism, somewhat different health metrics
Some structures remain similar
Some have changed, sh tech as easiest way
Main things to monitor CPU Memory Interfaces Radios CAPWAP
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 65BRKEWN-3012
AP-COSMemory - CPU
Memory
***** show memory summary *****
total used free shared buffers
Mem: 1022672 379456 643216 0 0
CPU
***** show processes cpu *****
Mem: 379456K used, 643216K free, 107792K shrd, 0K buff, 107916K cached
CPU: 0% usr 4% sys 0% nic 95% idle 0% io 0% irq 0% sirq
Load average: 0.08 0.10 0.13 2/111 22143
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
4710 2 root SWN 0 0% 1 4% [kclick]
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 66BRKEWN-3012
AP-COSTemperature
sh thermal-control-summary
This CLI produces large dump, do you want to proceed?(y/n)y
AP thermal condition
--------------------
fair
Temperature readings
--------------------
A38x CPU temp = 57 C
I2C-0-4f temp = 35 C
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 67BRKEWN-3012
AP-COSRadio Crashes
Radio Core files
***** show flash cores *****
Directory of /storage/cores/
total 1524
-rw-r--r-- 1 root root 1430212 Feb 22 17:06 AP1801-3802e-up-dd_core-radio0FW-8.2.145.44.2017-02-22-17-06-27.tgz
-rw-r--r-- 1 root root 123448 Feb 20 10:58 AP1801-3802e.all.10.0.cleanair.tgz
---------------------------------------------------------------------------
Filesystem Size Used Available Use% Mounted on
flash 57.5M 1.7M 52.8M 3% /storage
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 68BRKEWN-3012
AP-COS5 last set of logs
AP syslog data
***** show flash syslogs *****
Directory of /storage/syslogs/
total 296
-rw-r--r-- 1 root root 7969 Feb 8 17:36 176
-rw-r--r-- 1 root root 20479 Feb 7 19:15 176.0
-rw-r--r-- 1 root root 4838 Feb 8 17:36 176.last_write
-rw-r--r-- 1 root root 20480 Feb 7 17:55 176.start
-rw-r--r-- 1 root root 25 Feb 8 17:37 176.watchdog_status
-rw-r--r-- 1 root root 9976 Feb 9 04:36 177
-rw-r--r-- 1 root root 20407 Feb 8 17:42 177.0
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 69BRKEWN-3012
AP-COSClient Information
Different components
barbados-beta-floor1#sh dot11 clients 0
AP Mode - FlexConnect
Client MAC Slot ID WLAN ID AID WLAN Name RSSI Maxrate WGB
00:1E:E5:DF:A3:C4 1 3 1 psk-w2 -40 M15 No
barbados-beta-floor1#sh cont dot11Radio 1 client
mac radio vap aid state encr Maxrate is_wgb_wired wgb_mac_addr
00:1E:E5:DF:A3:C4 1 2 5 FWD AES_CCM128 M15 false 00:00:00:00:00:00
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 70BRKEWN-3012
AP-COSClient Information – 8.3MR2
AP58AC.78DC.AAA0#sh client summary
Radio Driver client Summary:
==============================
wifi0
[*02/08/2017 13:36:58.7460]
[*02/08/2017 13:36:58.7460] Total STA List Count 2
[*02/08/2017 13:36:58.7461] | NO| MAC|STATE|
[*02/08/2017 13:36:58.7461] -----------------------------
[*02/08/2017 13:36:58.7461] | 1|74:23:44:3b: 6: 7| 8|
[*02/08/2017 13:36:58.7461] | 2|3c:97: e:d5:e8:21| 8|
wifi1
[*02/08/2017 13:36:58.7479]
[*02/08/2017 13:36:58.7479] Total STA List Count 0
[*02/08/2017 13:36:58.7479] | NO| MAC|STATE|
[*02/08/2017 13:36:58.7479] -----------------------------
WCP client Summary:
=====================
mac radio vap aid state encr Maxrate is_wgb_wiredwgb_mac_addr
74:23:44:3B:06:07 0 0 2 FWD OPEN M7 false 00:00:00:00:00:00
3C:97:0E:D5:E8:21 0 0 1 FWD OPEN M7 false 00:00:00:00:00:00
NSS client Summary:
=====================
Current Count: 4
| MAC | OPAQUE |PRI POL|VLAN|BR|TN|QCF|BSS|RADID|MYMAC|
|58:AC:78:DC:AA:A0|00000000| 3| 0| 1| 1| 0| 2| 3| 1|
|58:AC:78:DC:AA:A1|00000000| 3| 0| 1| 1| 0| 2| 3| 1|
|3C:97:0E:D5:E8:21|00000000| 1| 0| 1| 1| 0| 0| 0| 0|
|74:23:44:3B:06:07|00000000| 1| 0| 1| 1| 0| 0| 0| 0|
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 71BRKEWN-3012
AP-COSClient Information Details – 8.3MR2
b2802# show controllers dot11Radio 1 client 3C:A9:F4:64:F7:6C
...
...
Legacy Rate Statistics:
(Mbps : Rx, Tx, Tx-Retries)
6 Mbps : 16, 0, 0
HT/VHT Rate Statistics:
(Rate/SS/Width : Rx, Tx, Tx-Retries)
0/1/20 : 10, 0, 0
1/1/20 : 1, 0, 0
2/2/20 : 0, 9, 54
3/2/20 : 0, 2, 1
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 72BRKEWN-3012
AP-COSRadio Status
Available in show tech
Associated client list:
====================
Total Client Count:0
Radio Total_clients Legacy_clients HT/VHT20_clients HT/VHT40_clients VHT80_clients
0 0 0 0 0 0
1 0 0 0 0 0
QBSS Load: cca_load: 0x3, rx_load: 0x0, tx_load: 0x0
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 73BRKEWN-3012
AP-COSRadio Status
Statistics
Tx Watchdog stats:
Tx SW Watchdog 0 / 0
Tx HW Watchdog 0 / 0
Beacons missed: 0-30s 31-60s 61-90s 90s+
0 0 0 0
intf TxData TxUC TxMBC TxBytes TxMBBytes TxFail TxDcrd RxData RxUC RxMBC RxBytes RxErr UCTxRtMCTxRt stats_ago
apr0v2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.700000
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 74BRKEWN-3012
AP-COSWho is around
RRM Neighbors
3802I_sw2_shield_gig15_C53#show rrm neighbor-list
========================================================
RRM Neighbors Slot 0
===================================
MAC Addr RSSI Srv.Chan
RRM Neighbors Slot 1
===================================
MAC Addr RSSI Srv.Chan
58:AC:78:DF:8C:1F -29 120
00:D7:8F:52:4F:6F -26 52
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 75BRKEWN-3012
AP-COSRadio Reset History
=================================================================================
AP58AC.78DC.AE00#show history interface dot11Radio all reset
=================================================================================
Timestamp Slot Client count Reset reason
Fri Jan 27 02:51:10 2017 0 0 84(radio failure (beacons stuck))
Fri Jan 27 02:51:18 2017 0 0 84(radio failure (beacons stuck))
Fri Jan 27 02:51:27 2017 0 0 2(Radio firmware crashed)
Fri Jan 27 02:52:32 2017 1 0 84(radio failure (beacons stuck))
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 76BRKEWN-3012
AP-COSRadio Reset History
APA0EC.F96C.D1D0#sh controllers dot11Radio 1 radio-reset
Last radio reset RC(54): Total 10 Start 5
Normal:
RC Count State When
10 1 START Wed Mar 8 16:53:21 2017 UTC
43 1 STOP Wed Mar 8 16:53:20 2017 UTC
50 1 START Wed Mar 8 16:53:22 2017 UTC
51 1 STOP Wed Mar 8 16:53:22 2017 UTC
52 1 START Wed Mar 8 16:53:22 2017 UTC
53 1 STOP Wed Mar 8 16:53:22 2017 UTC
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 77BRKEWN-3012
AP-COSRadar Detection History
ap3800i-r1sw1-1-0-2#sh history interface dot11Radio 1 radar detected
Timestamp Slot Client count Channel
Mon May 14 13:26:04 2018 1 0 64
Mon May 14 13:27:04 2018 1 0 100
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
AP-COSNetwork Sub System3802I_sw2_shield_gig15_C53#show controllers nss stats
NSS Statistics
Wired0 Port Statistics:
ID : 0 TYPE : 0
RX PKTS : 3428064 TX PKTS : 656862
RX OCTETS : 418299830 TX OCTETS : 151041084
RX ERR : 0 TX ERR : 0
Wired1 Port Statistics:
ID : 1 TYPE : 0
RX PKTS : 0 TX PKTS : 0
RX OCTETS : 0 TX OCTETS : 0
RX ERR : 0 TX ERR : 0
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 79BRKEWN-3012
AP-COSWLAN statsAP00FE.C82D.EF30#sh interfaces dot11Radio 0 wlan 6 statistics
VAP Level Stats: apr0v5 (under radio wifi0)
Tx Data Packets = 120
Tx Data Bytes = 54501
Tx Data Payload Bytes = 49941
Rx Data Packets = 166
Rx Data Bytes = 27204
Rx Data Payload Bytes = 24880
Tx Unicast Data Packets = 120
Rx Unicast Data Packets = 166
Tx Multi/Broadcast Data Packets = 0
Rx Multi/Broadcast Data Packets = 0
Packets Dropped = 0
Packets Errored = 0
Rx errors = 0
Rx Dropped = 0
Tx failures = 0
Tx Dropped = 0
Host Discard = 0
Rx MIC Errors = 0
Last Tx rate for unicast Packets = 11000
Last Tx rate for unicast Packets(mcs) = 22
Last Tx rate for multicast Packets = 48000
Last Tx rate for multicast Packets(mcs) = 96
Total beacons sent to fw in SWBA intr = 3066
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Key “Take Aways”
• Health prevention is good for you and for your network
• There are ways to detect problems before they happen
• Key metrics can be good indicators of how things are doing
81BRKEWN-3012
Complete your online session evaluation
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Give us your feedback to be entered into a Daily Survey Drawing.
Complete your session surveys through the Cisco Live mobile app or on www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Online.
83BRKEWN-3012
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Demos in the Cisco campus
Walk-in self-paced
labs
Meet the engineer
1:1 meetings
Related sessions
Continue your education
BRKEWN-3012 84
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS
Wireless Cisco education offerings
87
Course Description Cisco Certification
• Designing Cisco Wireless Enterprise
Networks• Deploying Cisco Wireless Enterprise
Networks• Troubleshooting Cisco Wireless Enterprise
Networks• Securing Cisco Wireless Enterprise Networks
Professional level instructor led trainings to prepare candidates to
conduct site surveys, imp lement, configure and support APs andcontrollers in converged Enterprise networks. Focused on 802.11 and
related technolog ies to des ign, deploy, troubleshoot as well as secureWireless infrastructure. Course also provide details around Cisco
mobility services Engine, Prime Infrastructure and wireless security.
CCNP® Wireless
Implementing Cisco Unified Wireless Network
Essential
Prepares candidates to design, install, configure, monitor and conduct
basic troubleshooting tasks of a Cisco WLAN in Enterprise installations.
CCNA® Wireless
Deploying Basic Cisco Wireless LANs (WDBWL)
Understanding of the Cisco Unified Wireless Networking for enterprise
deployment scenarios. In this course, you will learn the basics of how toinstall, configure, operate, and maintain a wireless network, both as an
add-on to an ex isting wireless LAN (WLAN) and as a new Cisco UnifiedWireless Networking solution.
1.2
Deploying Advanced Cisco Wireless LANs
(WDAWL)
The WDAWL advanced course is designed with the goal of providing
learners with the knowledge and skills to successfully p lan, install,configure, troubleshoot, monitor, and maintain advanced Cisco wireless
LAN solutions such as QoS, “salt and pepper” mobility, high densitydeployments, and outdoor mesh deployments in an enterprise cus tomer
environment.
1.2
Deploying Cisco Connected Mobile Experiences
(WCMX)
WCMX will prepare professionals to use the Cisco Unified Wireless
Network to configure, administer, manage, troubleshoot, and optimizeutilization of mobile content while gaining meaningful client analytics.
2.0
For more details, please visit: http://learningnetwork.cisco.comQuestions? Visit the Learning@Cisco Booth
Presentation ID