Performance Monitoring on AireOS and APs - Cisco Live

#CLUS

Javier Contreras Albesa. Sr Technical LeadBRKEWN-3012

Performance Monitoring on AireOS and APs

Agenda

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS

• Chinese Medicine - Prevention

• KPI - Health

• Translating KPI to AireOSIOSKPI

• AP-COS KPI

• Q & A

3Presentation ID


Cisco Webex Teams

4

Questions? Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session

Find this session in the Cisco Live Mobile App

Click “Join the Discussion”

Install Webex Teams or go directly to the team space

Enter messages/questions in the team space

How

Webex Teams will be moderated by the speaker until June 18, 2018.

cs.co/ciscolivebot#BRKEWN-3012

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4

1

2

3

4

Chinese Medicine

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public#CLUS 6BRKEWN-3012

Chinese Medicine – Prevention

• We want to stay Healthy

• Focus on prevention and good habits

• Not on “fixing” after the crash/illness

• What to monitor to ensure the Wireless network remains Healthy

Performance Monitoring as Preventive Medicine


Key Performance Indicators

• Key Performance Indicators

• Can be used to monitor success of an organization

• Define a set of values against which to measure

• Defining KPIs for network infrastructure subsystems

• KPI as health indicators: is the network device healthy?

Looking for Symptoms


Doctor am I sick?

• Not good: going to doctor after 5 days with 42C fever

• Preventive Detection: monitoring health overall

• Good detection:

• Memory utilization increasing every day

• Task tending to high CPU

• Radio channel tending to high utilization

• Not optimal detection:

• Uptime

• Channel meltdown

Looking for trouble before it happens


What are Good Health Indicators?

• Resources are Finite

• CPU time

• Memory

• Timers

• ARP entries

• Client entries

• Queues

• Longer Uptimes/Capwap

• You can always exhaust resources

• Utilization remains in expected range under intended load

Where to look


What are Good Health Indicators?

• Negative Events

• Radio Resets (not all!)

• Crashes

• Drops

• Hangs

• Not servicing clients

Where to look


How can I obtain this?

• PI will not provide it all

• It is a more “network wide status” tool

• This is a per device check

• For the moment, scripting tools can help

Getting the data

Translating KPI to AireOS


0 1

Service

Port

AP Client

13BRKEWN-3012

WLC Architecture What we need to Look At

Network

Multicore CPU

(Control)

6 7

PCIe Switch

Multicore CPU

(Data)

I/O Subsystem

Control Plane


0 1

Service

Port

AP Client

14BRKEWN-3012


Network

Multicore CPU

(Control)

6 7

PCIe Switch

Multicore CPU

(Data)

I/O Subsystem

Data Plane


0 1

Service

Port

AP Client

15BRKEWN-3012


Network

Multicore CPU

(Control)

6 7

PCIe Switch

Multicore CPU

(Data)

I/O Subsystem

AP


0 1

Service

Port

AP Client

16BRKEWN-3012


Network

Multicore CPU

(Control)

6 7

PCIe Switch

Multicore CPU

(Data)

I/O Subsystem

RF


AireOS KPIsIt is a complex world

APF

(802.11)

Dot1x

(Keys/Auth)

AAA

Subsystems

PEM

MSCB OS

IP Stack

Service

Port

Driver

Data Plane

RRM

SPAM

(CAPWAPP)DTLS


AireOS Basic Health

• Where to Put the Thermometer?

• Control Plane

• Data Plane

Does it have Fever?


AireOS KPIs

• Queue Utilization

• Per Process CPU

• CPU per core

• DHCP stats

• Timers

• ARP Switch/Kernel

• Memory Utilization

• Mbufs

• Client Count

• PMK Count

Let’s measure – Control Plane


AireOS KPIs

• State monitoring

• DP-CP Drops

• Fragmentation

• Packet Pools

Let’s measure - Data Plane

One CPU to bind them…


AireOS KPIsCPU Monitoring

(Cisco Controller) >show cpu

Current CPU(s) load: 4%

Individual CPU load: 0%/0%, 0%/0%, 0%/1%, 100%/0%, 0%/1%, 0%/1%, 0%/1%, 0%/1%, 0%/1%, 0%/1%, 0%/1%, 0%/1%, 0%/0%, 0%/0%, 0%/1%, 0%/0%, 0%/0%, 0%/0%, 0%/0%, 0%/1%, 0%/0%, 0%/0%, 0%/0%, 0%/0%


AireOS KPIsTask-CPU Utilization

(WLC-5508-7) >show process cpu

Name Priority CPU Use (usr/sys)% hwm CPU Reaper

System Reset Task (240/ 7) 0 ( 0/ 0)% 0 9

reaperWatcher ( 3/ 96) 0 ( 0/ 0)% 0 7 I

osapiReaper ( 10/ 94) 0 ( 0/ 0)% 0 1 I

TempStatus (240/ 7) 0 ( 0/ 0)% 0 9 I

HAPeerToPeerCommTa ( 90/ 64) 0 ( 0/ 0)% 0 6

pktDebugSocketTask (255/ 1) 0 ( 0/ 0)% 0 6

webauthRedirect ( 95/ 62) 100 ( 95/ 4)% 100 9 T 300

emWeb (240/ 7) 0 ( 0/ 0)% 100 1 T 300

CSCuo86819 WLC crashes in 7.6.120.0 - memory corruption caused by Webauth

Queues In London


AireOS KPIs

• Used for messaging between tasks

• Around 128-150 queues in 8.5/8.3

• Some are critical for the well being of the WLC

(5520-2) >show queue-info

Total message queue count = 128

* Column Breached shows the number of times the queue breached the 95% mark

Q-Id Queue Name Allocated MsgSize InUse MaxUsed Breached* Details

--------------------------------------------------------------------------------------------

1 PRINTF-Q 256 80 0 0 0 N

2 GRE Queue 100 8 0 1 0 N

3 dtlqueue 4096 2 0 3 0 N

4 dtlarpqueue 4096 24 0 8 0 Y

Queue Utilization


AireOS KPIs

• Inuse: how many messages in the queue

• MaxUsed: top message count ever

• Breached: how many times we reached 95% utilization

Queue Utilization


AireOS KPIsQueue Utilization

(WiSM-slot1-1) >show queue-info

Q-Id Queue Name Allocated MsgSize InUse MaxUsed Breached* Details

--------------------------------------------------------------------------------------------

1 PRINTF-Q 256 80 0 0 0 N

2 GRE Queue 100 8 0 5 0 N

3 dtlqueue 4096 2 0 3 0 N

4 dtlarpqueue 4096 16 0 75 0 Y

..

70 nfv9queue 4096 316 0 2 0 Y

71 RogueApInfoChangedDB 4000 4 4000 4000 68 N

72 RogueApMeasChangedDB 4000 4 4000 4000 1 N

73 RogueClientInfoChangedDB 5000 4 5000 5000 1 N

74 RogueClientMeasChangedDB 5000 4 5000 5000 1 N


AireOS KPIsQueue Utilization

86 RRM-DCLNT-5_0-Q 375 0 4 0 N

87 RRM-MGR-5_0-Q 375 0 20 0 N

88 RRM-PERIODIC-TASK-Q 93 0 1 0 N

89 SNTP-Q 3 2 3 20010 N

90 CDP-Q 512 0 13 0 N

91 BCAST-Q 2048 0 2 0 Y

92 BCAST-DATA-Q 512 0 2 0 Y

93 IDS Track Event Msg Queue 1024 0 8 0 N


AireOS KPIsQueue Decoder Ring

Dtlarpqueue: Data transformation layer ARP queue

SNMP-Q: SNMP requests

Debugger/logger-q: Debug and Syslog Messages

SPAM: Messages between AP and WLC

NMSP*: location

Dot1x: all authentication (PSK/dot1x)

APF*: 802.11 client state machine

BCAST*: multicast/broadcast

A queue like no other


AireOS KPIsRadius Queues

Radius messages between WLC and AAA server

8.0MR3 and lower: 255 standing requests (ID)

8.0MR4 and higher: 16 queues per server (ID/Source Port)

(5520-2) >show radius queue

Max Radius Queues Per Server..................... 16

Max Radius Buffers Available..................... 8128

Currently number of Buffers consumed............ 0

Radius Authentication Messages Stats

Total Auth Req sent(allocated).................. 140

Total Auth Resp rcvd(freed)..................... 140

Total Auth Req Pkts Dropped(no buffer).......... 0


AireOS KPIsRadius Queues

(Cisco Controller) >show radius queue

Radius Queue

Auth Alloc : 122401616

Acct Alloc : 31288769

Auth Free : 122401615

Acct Free : 31288769

Auth Alloc Err : 0

Acct Alloc Err : 0


AireOS KPIsDHCP

DHCP NAK Count:................................ 3573

Tx Fails:...................................... 26765

Last Rx Time:.................................. 1 seconds ago

Last Tx Time:.................................. 3 seconds ago

(CH-8510W-DC-1) show dhcp stats

DHCP Proxy Behaviour: enabled

DHCP Server IP Address:.......................... 10.5.0.10

DHCP RX DISCOVER Count:........................ 3117892

DHCP TX DISCOVER Count:........................ 3175288

DHCP ACK Count:................................ 4644105

DHCP REQUEST Count:............................ 7641156

DHCP INFORM Count:............................. 200006

DHCP DECLINE Count:............................ 0

DHCP RELEASE Count:............................ 987

DHCP REPLY Count:.............................. 0

DHCP OFFER Count:.............................. 1283401

I remember you..


AireOS KPIs

Memory Monitoring Memory Utilization

Increases with load, remains steady on same load level

Most tasks use OSAPI Buffer based: time, owner, trailers

MBUFs


AireOS KPIsMemory the easy way

show memory history (8.0.132.0 and higher)(5520-2) >show memory history

-------------------------- System Memory History -------------------------

System Name ..................................... 5520-2

Software Version ................................ 8.5.124.55

Current Time .................................... Fry May 11 17:59:44 2017

System UP Time .................................. 6 days 4 hrs 12 mins 48 secs

------------------------- System Memory at BOOT ------------------------

Memory Snapshot Time ............................ Fry May 5 14:17:18 2017

Total System Memory ............................. (32386132 KB) 31627 MB

Total System Free Memory......................... (29429840 KB) 28740 MB (90 %)



Hundreds of data points------------------------- System Memory Snapshot 1 Count: 289 -------

Memory Snapshot Time ............................ Sun Jan 29 14:54:31 2017

Total System Memory ............................. (32386132 KB) 31627 MB

Total System Free Memory......................... (29408732 KB) 28719 MB (90 %)

Total Memory in Buffers.......................... (9960 KB)

Total Memory in Cache............................ (703068 KB) 686 MB

Total Active Memory.............................. (2200724 KB) 2149 MB

Total InActive Memory............................ (597932 KB) 583 MB



MBUF Utilization

------------------------- MBUF Information ------------------------------

Maximum number of Mbufs.......................... 24576

Number of Mbufs Free............................. 24560

Number of Mbufs In Use........................... 16



Buffers in use and Top users------------------------- Pmalloc Pools Information --------------------

Index Pool-Size Chunks-In-Pool Chunks-In-Use Memory(Size/Used/Free)KB

0 16 200000 40881 21875 /19388 /2486

1 64 160000 6794 25000 /15424 /9575

2 128 93600 31317 20475 /12689 /7785

3 256 20000 6406 6875 /3476 /3398

..

-------------- TOP-20 Pools Consuming Tasks (Time: 465689 Secs) ------------

Index Name PID 16-byte 64-byte 128-byte 256-byte 384-byte 512-byte 1024-byte 2048-byte 4096-byte Raw-Pool Total-Pools

001 fp_main_task 2182 3011 4289 155 280 55 9 124058 60 24586 1767 158270

002 emWeb 2664 37373 657 31015 5987 30 4 21 2 7 9 75105

A little bit of everything…


AireOS KPIsHappy Uptime

(5520-2) >show ap uptime

Number of APs.................................... 2

Global AP User Name.............................. admin

Global AP Dot1x User Name........................ Not Configured

AP Name Ethernet MAC AP Up Time Association Up Time

------------------ ----------------- ----------------------- -----------------------

ap1800-3850-3-11 80:e8:6f:d0:12:08 5 days, 19 h 57 m 32 s 5 days, 19 h 55 m 54 s

barbados-beta-floor1 58:ac:78:dc:d3:a0 5 days, 19 h 58 m 24 s 1 days, 04 h 53 m 52 s


AireOS KPIsCoredump Available

(5520-2) >show coredump summary

Core Dump upload is disabled

Core Dump file on flash:

SW Version....................................... 8.2.102.89

Time Stamp....................................... Tue Apr 26 15:01:08 2016

File Size........................................ 50431374

File Name Suffix................................. _8.2.102.89_5520-2_1461682868.gz


AireOS KPIsTime(rs) are gold

(5500-2) >show system timers summary

Total Timers..................................... 150000

Allocated.................................... 2538

Free......................................... 147462

Peak Usage................................... 2842

Alloc Operations................................. 275207

Free Operations.................................. 272669

Alloc Failures................................... 0


AireOS KPIs5520/8540 Chassis Monitoring

(5520-2) >show imm chassis fan

FAN1_TACH1 | 0Fh | ok | 29.1 | 8200 RPM

FAN1_TACH2 | 10h | ok | 29.1 | 9600 RPM

FAN2_TACH1 | 11h | ok | 29.2 | 8200 RPM

FAN2_TACH2 | 12h | ok | 29.2 | 9600 RPM

FAN3_TACH1 | 13h | ok | 29.3 | 8200 RPM

FAN3_TACH2 | 14h | ok | 29.3 | 10000 RPM

FAN4_TACH1 | 15h | ok | 29.4 | 8200 RPM

FAN4_TACH2 | 16h | ok | 29.4 | 10000 RPM

FAN5_TACH1 | 17h | ok | 29.5 | 8200 RPM

FAN5_TACH2 | 18h | ok | 29.5 | 10000 RPM


AireOS KPIsMy Dear Clients

(Cisco Controller) >show client state summary

Client State Summary

====================

State Number of Clients

----- -----------------

START 8

8021X_REQD 25

DHCP_REQD 54

WEBAUTH_REQD 260

RUN 146

----- -----------------

Total 493


AireOS KPIsMy Dear Clients

(Controller_2) >show pmk-cache all

Number of PMK Cache Entries: 156

Type Station Lifetime VLAN Override IP Override Audit-Session-ID

------ -------------- -------- ------------------ --------------- ----------------

RSN 04:54:53:4b:63:5c 86301 0.0.0.0

RSN 04:54:53:4c:22:d9 57521 0.0.0.0

RSN 08:70:45:92:7d:f0 86363 0.0.0.0


AireOS KPIsARPs(5520-2) >show arp stats

ARP statistics:

Current Entries 236

Dup Additions 382947

Bad Additions 0

Bad Deletions 0

(5520-2) >show arp kernel

IP address HW type Flags HW address Mask Device

4.0.7.161 0x1 0x2 00:00:05:00:00:04 * dtl0.4

4.0.7.122 0x1 0x2 00:00:06:00:00:09 * dtl0.4

4.0.7.117 0x1 0x2 00:00:06:00:00:1d * dtl0.4

192.168.5.104 0x1 0x2 58:ac:78:dc:d3:a0 * dtl0.5

Max=client+1000


AireOS KPIsLAG

(5500-4-82) >show lag summary

LAG Disable is in transition. Pls Reboot the switch

This won’t happen unless configured

If you change it, reboot

The Data Plane: happy frames


Data PlaneHappy to forward your frames

Takes care of all frame forwarding for clients

Also: AVC, Encryption, Tunneling

Closely monitored by control plane

Its own memory, CPU, software

Its own stats and performance indicators


Data PlaneHappy to forward your frames

Main command:

(5500-2) >debug fastpath dump fpapool

FP0.01:

FP0.01:(#)Buffer Pool = avail / size (in-use) min-avail (max in-use)

..

FP0.01:Buffer overload back-pressure limits:

FP0.01:Entered _GREENZONE 1 times. _GREENLINE is 27000 buffers available.

FP0.01:Entered YELLOWZONE 0 times. YELLOWLINE is 27000 buffers available.

FP0.01:Entered ORANGEZONE 0 times. ORANGELINE is 26000 buffers available.

FP0.01:Entered ___REDZONE 0 times. ___REDLINE is 10000 buffers available.

FP0.01:Entered _BLACKZONE 0 times. _BLACKLINE is 5000 buffers available.


Data PlaneHappiness gone

FP0.02:Entered _GREENZONE 59575 times. _GREENLINE is 27000 buffers available.

FP0.02:Entered YELLOWZONE 59575 times. YELLOWLINE is 27000 buffers available.

FP0.02:Entered ORANGEZONE 1 times. ORANGELINE is 26000 buffers available.

FP0.02:Entered ___REDZONE 0 times. ___REDLINE is 10000 buffers available.

FP0.02:Entered _BLACKZONE 0 times. _BLACKLINE is 5000 buffers available.

Green Zone: Normal Scenario

Yellow Zone: Traffic Peaks

Orange Zone: Multicast is dropped, CAPWAP is top priority

Red/Black: Heavy packet drop -> recovery through crash


Data Plane – Control PlaneLet’s Talk(5520-2) >debug fastpath dump dpcp-stats

FP0.20:Queue TxCount RateLmtDrop PktLmtDrop

FP0.20:Q0: Internal DP --- 85587 0 0

FP0.20:Q1: Mobility Ctrl --- 19 0 0

FP0.20:Q2: Capwap --- 237449 0 0

FP0.20:Q3: Client Traffic --- 1610 0 0

FP0.20:Q4: ARP, Client services --- 33729 0 0

FP0.20:Q5: IPv6 Client services --- 16634 0 0

FP0.20:Q6: Infra services --- 42932 0 0

FP0.20:Q7: Probe --- 276962 0 0

FP0.20:Q8: IAPP, Rogue --- 85139 1 0

FP0.20:Q9: HA --- 8714 0 0

FP0.20:Q10: WebAuth --- 0 0 0

IOS APs


IOS APsSimpler

IOS as monolithic entity

Main things to monitor CPU Memory Interfaces Radios CAPWAP


IOS APsRadio Interfaces

Show controller d0 | d1

Lot of data, not all human readable

Not intended for “general use”, but treasure of information


IOS APsShow Controller - Monster

interface Dot11Radio0Radio AL 2.4GHz, Base Address b0aa.7733.bf10, BBlock version 0.00, Software version 4.18.2Serial number: FOC19066XZD Unused dynamic SQRAM memory: 0x0000C858 (50 KB)Unused dynamic SDRAM memory: 0x00082674 (521 KB)Spectrum FW version: 1.15.4Number of supported simultaneous BSSID on Dot11Radio0: 16Carrier Set: Spain (ES) (-E)Uniform Spreading Required: NoConfigured Frequency: 2412 MHz Channel 1Serving Frequency: 2412 MHz Channel 1 20MHzAllowed Frequencies: 2412( 1) 2417( 2) 2422( 3) 2427( 4) 2432( 5) 2437( 6) 2442( 7) 2447( 8) 2452( 9) 2457( 10) 2462( 11) 2467( 12) 2472( 13) Listen Frequencies: 2412( 1) 2417( 2) 2422( 3) 2427( 4) 2432( 5) 2437( 6) 2442( 7) 2447( 8) 2452( 9) 2457( 10) 2462( 11) 2467( 12) 2472( 13) 2484( 14) Beacon Flags: 0, Interface Flags 2020105, Interface Events 20, Mode 9; Beacons are enabled; Probes are enabledConfigured TxPower: 16 dBm (Level Index 1)Allowed Power Levels:

16 13 10 7 4 1 dBmAllowed Client Power Levels: 16 13 10 7 4 1 dBmAntenna:Rx[right-a left-b middle-c ]

Tx[right-a left-b middle-c cck all ofdm all]Internal Gain [Allowed 8, In

Use 8] (dBi x 2), In Use 8] (dBi x 2), In Use 8] (dBi x 2), In Use 8] (dBi x 2), Reported 0, Configured 0, In Use 8] (dBi x 2)Active Rates: 9.0 basic-12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2 m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Current Rates: 9.0 basic-12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2 m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Allowed Rates: 1.0 2.0 5.5 6.0 9.0 11.0 12.0 18.0 24.0 36.0 48.0 54.0All Rates: 1.0 2.0 5.5 11.0 6.0 9.0 12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2 m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Default Rates: basic-1.0 basic-2.0 basic-5.5 basic-11.0 6.0 9.0 12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2 m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Best Range Rates: basic-1.0 2.0 5.5 11.0 6.0 9.0 12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2 m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Best Throughput Rates: basic-1.0 basic-2.0 basic-5.5 basic-11.0 basic-6.0 basic-9.0 basic-12.0 basic-18.0 basic-24.0 basic-36.0 basic-48.0 basic-54.0 m0-2 m1-2 m2-2 m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Current Voice Rates: basic-5.5 basic-11.0 6.0 12.0 24.0 [disabled until voice packet-discard enabled]Default Voice Rates: basic-5.5 basic-11.0 6.0 12.0 24.0Managment Rates: 9.0 basic-12.0Multicast Rates: basic-12.0 Multicast Active rate: 12.0RTS Rates: basic-12.0RTS Active rate 12.0RTS Max rate 12.0 Local Packet Rate Probing is disabled : Packet Size 20Data Rate Sensitivity (rate, SNR dB, Contention dBm)( 1.0, 10, -93) ( 2.0, 10, -92) ( 5.5, 11, -90) (11.0, 14, -90) ( 6.0, 15, -89) ( 9.0, 16, -88) (12.0, 18, -88) (18.0, 19, -86) (24.0, 20, -85) (36.0, 24, -81) (48.0, 27, -78) (54.0, 31, -77) ERP settings: short slot time.Neighbors in non-erp mode:


IOS APsTaming the monster

Interface status

Beacon Flags: 0, Interface Flags 20105, Interface Events 0, Mode 9; Beacons are enabled; Probes are enabled

Channel set to zero

Configured Frequency: 2412 MHz Channel 1

Warn on low PoE scenario

Radio on Low Power Mode due to PoE, restricted to 3 antennas

• QBSS set to higher than 80% (0xCC)

QBSS Load: 0x24



TX blocks - Pending

Transmit queues: Limit 13919 Current 4 In-Progress 0

---- Active ------ In-Progress -----------

Cnt Quo Bas Max Cl Cnt Quo Bas Uplink

0 64 0 0 0 0 5 0 Voice

0 512 0 0 0 0 60 0 Video

0 1024 0 0 0 0 200 0 Best

4 1024 0 0 0 0 200 0 MC0

0 0 0 0 0 0 7 0

• High Radio reset counts

Radio resets - total:14 retries:0 failed:0



RF jammer detected

Jammer detected: 1 / 0

• Beacon Stop

Beacon stopped count: 0

Counts > 120 0

Counts > 90 0

Counts > 60 0

Counts > 30 2

Counts > 15 0

Counts > 10 0

Counts > 5 1



• Silent radio hang

• No traffic received/sent when clients are around

Host Rx K Bytes: 2243173 / 0 Host Tx K Bytes: 3382831 / 0

Unicasts Rx: 368819860 / 0 Unicasts Tx: 457956326 / 0

Unicasts to host: 368819860 / 0 Unicasts by host: 415499647 / 0

Broadcasts Rx: 348114707 / 0 Broadcasts Tx: 354596429 / 439

Beacons Rx: 342650083 / 0 Beacons Tx: 353388822 / 439

PCI Resets

Radio PCI resets:0

AP-COS 1800/2800/3800/1560


AP-COSDifferent architecture (simplified view)

Wireless Control

WCP

hostapd

Platform

Management

watchdog

LED

capwapd

Capwapbrain

Config/system mng

ssh

Device Management

Light

httpd

CLI

Click DataPath

Linux Networking

stack

Cisco

Services/clickfilesystem

Ethernet

Driver

Radio

Driver

Flash

Driver

Linux

Kernel


AP-COSDifferent organism, somewhat different health metrics

Some structures remain similar

Some have changed, sh tech as easiest way

Main things to monitor CPU Memory Interfaces Radios CAPWAP


AP-COSMemory - CPU

Memory

***** show memory summary *****

total used free shared buffers

Mem: 1022672 379456 643216 0 0

CPU

***** show processes cpu *****

Mem: 379456K used, 643216K free, 107792K shrd, 0K buff, 107916K cached

CPU: 0% usr 4% sys 0% nic 95% idle 0% io 0% irq 0% sirq

Load average: 0.08 0.10 0.13 2/111 22143

PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND

4710 2 root SWN 0 0% 1 4% [kclick]


AP-COSTemperature

sh thermal-control-summary

This CLI produces large dump, do you want to proceed?(y/n)y

AP thermal condition

--------------------

fair

Temperature readings

--------------------

A38x CPU temp = 57 C

I2C-0-4f temp = 35 C


AP-COSRadio Crashes

Radio Core files

***** show flash cores *****

Directory of /storage/cores/

total 1524

-rw-r--r-- 1 root root 1430212 Feb 22 17:06 AP1801-3802e-up-dd_core-radio0FW-8.2.145.44.2017-02-22-17-06-27.tgz

-rw-r--r-- 1 root root 123448 Feb 20 10:58 AP1801-3802e.all.10.0.cleanair.tgz

---------------------------------------------------------------------------

Filesystem Size Used Available Use% Mounted on

flash 57.5M 1.7M 52.8M 3% /storage


AP-COS5 last set of logs

AP syslog data

***** show flash syslogs *****

Directory of /storage/syslogs/

total 296

-rw-r--r-- 1 root root 7969 Feb 8 17:36 176

-rw-r--r-- 1 root root 20479 Feb 7 19:15 176.0

-rw-r--r-- 1 root root 4838 Feb 8 17:36 176.last_write

-rw-r--r-- 1 root root 20480 Feb 7 17:55 176.start

-rw-r--r-- 1 root root 25 Feb 8 17:37 176.watchdog_status

-rw-r--r-- 1 root root 9976 Feb 9 04:36 177

-rw-r--r-- 1 root root 20407 Feb 8 17:42 177.0


AP-COSClient Information

Different components

barbados-beta-floor1#sh dot11 clients 0

AP Mode - FlexConnect

Client MAC Slot ID WLAN ID AID WLAN Name RSSI Maxrate WGB

00:1E:E5:DF:A3:C4 1 3 1 psk-w2 -40 M15 No

barbados-beta-floor1#sh cont dot11Radio 1 client

mac radio vap aid state encr Maxrate is_wgb_wired wgb_mac_addr

00:1E:E5:DF:A3:C4 1 2 5 FWD AES_CCM128 M15 false 00:00:00:00:00:00


AP-COSClient Information – 8.3MR2

AP58AC.78DC.AAA0#sh client summary

Radio Driver client Summary:

==============================

wifi0

[*02/08/2017 13:36:58.7460]

[*02/08/2017 13:36:58.7460] Total STA List Count 2

[*02/08/2017 13:36:58.7461] | NO| MAC|STATE|

[*02/08/2017 13:36:58.7461] -----------------------------

[*02/08/2017 13:36:58.7461] | 1|74:23:44:3b: 6: 7| 8|

[*02/08/2017 13:36:58.7461] | 2|3c:97: e:d5:e8:21| 8|

wifi1

[*02/08/2017 13:36:58.7479]

[*02/08/2017 13:36:58.7479] Total STA List Count 0

[*02/08/2017 13:36:58.7479] | NO| MAC|STATE|

[*02/08/2017 13:36:58.7479] -----------------------------

WCP client Summary:

=====================

mac radio vap aid state encr Maxrate is_wgb_wiredwgb_mac_addr

74:23:44:3B:06:07 0 0 2 FWD OPEN M7 false 00:00:00:00:00:00

3C:97:0E:D5:E8:21 0 0 1 FWD OPEN M7 false 00:00:00:00:00:00

NSS client Summary:

=====================

Current Count: 4

| MAC | OPAQUE |PRI POL|VLAN|BR|TN|QCF|BSS|RADID|MYMAC|

|58:AC:78:DC:AA:A0|00000000| 3| 0| 1| 1| 0| 2| 3| 1|

|58:AC:78:DC:AA:A1|00000000| 3| 0| 1| 1| 0| 2| 3| 1|

|3C:97:0E:D5:E8:21|00000000| 1| 0| 1| 1| 0| 0| 0| 0|

|74:23:44:3B:06:07|00000000| 1| 0| 1| 1| 0| 0| 0| 0|


AP-COSClient Information Details – 8.3MR2

b2802# show controllers dot11Radio 1 client 3C:A9:F4:64:F7:6C

...

...

Legacy Rate Statistics:

(Mbps : Rx, Tx, Tx-Retries)

6 Mbps : 16, 0, 0

HT/VHT Rate Statistics:

(Rate/SS/Width : Rx, Tx, Tx-Retries)

0/1/20 : 10, 0, 0

1/1/20 : 1, 0, 0

2/2/20 : 0, 9, 54

3/2/20 : 0, 2, 1


AP-COSRadio Status

Available in show tech

Associated client list:

====================

Total Client Count:0

Radio Total_clients Legacy_clients HT/VHT20_clients HT/VHT40_clients VHT80_clients

0 0 0 0 0 0

1 0 0 0 0 0

QBSS Load: cca_load: 0x3, rx_load: 0x0, tx_load: 0x0


AP-COSRadio Status

Statistics

Tx Watchdog stats:

Tx SW Watchdog 0 / 0

Tx HW Watchdog 0 / 0

Beacons missed: 0-30s 31-60s 61-90s 90s+

0 0 0 0

intf TxData TxUC TxMBC TxBytes TxMBBytes TxFail TxDcrd RxData RxUC RxMBC RxBytes RxErr UCTxRtMCTxRt stats_ago

apr0v2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.700000


AP-COSWho is around

RRM Neighbors

3802I_sw2_shield_gig15_C53#show rrm neighbor-list

========================================================

RRM Neighbors Slot 0

===================================

MAC Addr RSSI Srv.Chan

RRM Neighbors Slot 1

===================================

MAC Addr RSSI Srv.Chan

58:AC:78:DF:8C:1F -29 120

00:D7:8F:52:4F:6F -26 52


AP-COSRadio Reset History

=================================================================================

AP58AC.78DC.AE00#show history interface dot11Radio all reset

=================================================================================

Timestamp Slot Client count Reset reason

Fri Jan 27 02:51:10 2017 0 0 84(radio failure (beacons stuck))


Fri Jan 27 02:51:27 2017 0 0 2(Radio firmware crashed)



AP-COSRadio Reset History

APA0EC.F96C.D1D0#sh controllers dot11Radio 1 radio-reset

Last radio reset RC(54): Total 10 Start 5

Normal:

RC Count State When

10 1 START Wed Mar 8 16:53:21 2017 UTC

43 1 STOP Wed Mar 8 16:53:20 2017 UTC

50 1 START Wed Mar 8 16:53:22 2017 UTC

51 1 STOP Wed Mar 8 16:53:22 2017 UTC

52 1 START Wed Mar 8 16:53:22 2017 UTC

53 1 STOP Wed Mar 8 16:53:22 2017 UTC


AP-COSRadar Detection History

ap3800i-r1sw1-1-0-2#sh history interface dot11Radio 1 radar detected

Timestamp Slot Client count Channel

Mon May 14 13:26:04 2018 1 0 64

Mon May 14 13:27:04 2018 1 0 100


AP-COSNetwork Sub System3802I_sw2_shield_gig15_C53#show controllers nss stats

NSS Statistics

Wired0 Port Statistics:

ID : 0 TYPE : 0

RX PKTS : 3428064 TX PKTS : 656862

RX OCTETS : 418299830 TX OCTETS : 151041084

RX ERR : 0 TX ERR : 0

Wired1 Port Statistics:

ID : 1 TYPE : 0

RX PKTS : 0 TX PKTS : 0

RX OCTETS : 0 TX OCTETS : 0

RX ERR : 0 TX ERR : 0


AP-COSWLAN statsAP00FE.C82D.EF30#sh interfaces dot11Radio 0 wlan 6 statistics

VAP Level Stats: apr0v5 (under radio wifi0)

Tx Data Packets = 120

Tx Data Bytes = 54501

Tx Data Payload Bytes = 49941

Rx Data Packets = 166

Rx Data Bytes = 27204

Rx Data Payload Bytes = 24880

Tx Unicast Data Packets = 120

Rx Unicast Data Packets = 166

Tx Multi/Broadcast Data Packets = 0

Rx Multi/Broadcast Data Packets = 0

Packets Dropped = 0

Packets Errored = 0

Rx errors = 0

Rx Dropped = 0

Tx failures = 0

Tx Dropped = 0

Host Discard = 0

Rx MIC Errors = 0

Last Tx rate for unicast Packets = 11000

Last Tx rate for unicast Packets(mcs) = 22

Last Tx rate for multicast Packets = 48000

Last Tx rate for multicast Packets(mcs) = 96

Total beacons sent to fw in SWBA intr = 3066

Key things to remember


Key “Take Aways”

• Health prevention is good for you and for your network

• There are ways to detect problems before they happen

• Key metrics can be good indicators of how things are doing

81BRKEWN-3012

Thank you

#CLUS

Complete your online session evaluation


Give us your feedback to be entered into a Daily Survey Drawing.

Complete your session surveys through the Cisco Live mobile app or on www.CiscoLive.com/us.

Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Online.

83BRKEWN-3012

http://www.ciscolive.com/us

http://www.ciscolive.com/Online


Demos in the Cisco campus

Walk-in self-paced

labs

Meet the engineer

1:1 meetings

Related sessions

Continue your education

BRKEWN-3012 84

Thank you

#CLUS


Wireless Cisco education offerings

87

Course Description Cisco Certification

• Designing Cisco Wireless Enterprise

Networks• Deploying Cisco Wireless Enterprise

Networks• Troubleshooting Cisco Wireless Enterprise

Networks• Securing Cisco Wireless Enterprise Networks

Professional level instructor led trainings to prepare candidates to

conduct site surveys, imp lement, configure and support APs andcontrollers in converged Enterprise networks. Focused on 802.11 and

related technolog ies to des ign, deploy, troubleshoot as well as secureWireless infrastructure. Course also provide details around Cisco

mobility services Engine, Prime Infrastructure and wireless security.

CCNP® Wireless

Implementing Cisco Unified Wireless Network

Essential

Prepares candidates to design, install, configure, monitor and conduct

basic troubleshooting tasks of a Cisco WLAN in Enterprise installations.

CCNA® Wireless

Deploying Basic Cisco Wireless LANs (WDBWL)

Understanding of the Cisco Unified Wireless Networking for enterprise

deployment scenarios. In this course, you will learn the basics of how toinstall, configure, operate, and maintain a wireless network, both as an

add-on to an ex isting wireless LAN (WLAN) and as a new Cisco UnifiedWireless Networking solution.

1.2

Deploying Advanced Cisco Wireless LANs

(WDAWL)

The WDAWL advanced course is designed with the goal of providing

learners with the knowledge and skills to successfully p lan, install,configure, troubleshoot, monitor, and maintain advanced Cisco wireless

LAN solutions such as QoS, “salt and pepper” mobility, high densitydeployments, and outdoor mesh deployments in an enterprise cus tomer

environment.

1.2

Deploying Cisco Connected Mobile Experiences

(WCMX)

WCMX will prepare professionals to use the Cisco Unified Wireless

Network to configure, administer, manage, troubleshoot, and optimizeutilization of mobile content while gaining meaningful client analytics.

2.0

For more details, please visit: http://learningnetwork.cisco.comQuestions? Visit the Learning@Cisco Booth

Presentation ID

http://learningnetwork.cisco.com/

Performance Monitoring on AireOS and APs - Cisco Live

Documents

Transcript of Performance Monitoring on AireOS and APs - Cisco Live