Loss of SCADA Event Report Effort

Vern Kissner [Tacoma Power] – Chair

Critical Infrastructure and Information

Management Subcommittee (CIIMS)

2

Circa 2011 – WECC region has over 50% of reported Loss

of SCADA reports [NERC wide].

3

Team members include:

• Stacen Tyskiewicz (BPA)

• Gray Wright (NVenergy)

• Jonathon Aust (WAPA)

• Brad Calbick (AVA)

• Jeff Dagle (PNNL)

• Brian Meadows (SRP)

• Murat Uludogan (WECC)

• Jason Radetski (TEP)

• Vern Kissner (TPWR)

4

o Initial assumptions / theories for root cause and

contributing factors:

Timeline of EMS Replacement / Upgrade

System Complexity / Supporting Infrastructure

Lack of support staff and or training

Compliance

Reliance on Vendor Support

WECC CC / SCADA Event Reports

5

CIIMS / EMS / DE Team

Reporting

Entity

WECC

Staff Team

NERC

Staff

Questions

6

Started tracking events:

o Per calendar quarter

o Time of day

o Outage duration

o Events per entity

o Vendor type

Caused by Supporting Infrastructure or EMS

specific

Team approach in search of common

themes / root cause.

7

Events per calendar quarter

0

1

2

3

4

5

6

2010 Q4

2011 Q1

2011 Q2

2011 Q3

2011 Q4

2012 Q1

2012 Q2

2012 Q3

2012 Q4

2013 Q1

2013 Q2

2013 Q3

8

Time of Day – 4 hr intervals

0

2

4

6

8

10

12

00:00-03:59 4:00-7:59 8:00-11:59 12:00-15:59 16:00-19:59 20:00-24:00

Why so many events

between 4 and 8 AM???

9

Outage duration 10 min intervals

0 0

3

5

6

5

4

1

4

1 1 1

2

1

0

2

0

1

0 0

1

Outage time spectrum

23 205

10

Event(s) per entity

0

2

4

6

8

10

12

14

16

18

20

3 events 2 events 1 event

11

Cause

Yes EMS related 50%

No infrastucture related 50%

EMS or Other Caused

12

Of Infrastructure Related Events

Comm Network 60%

Power 25%

Fire Alarm 15%

13

Of EMS Specific Related Events

Missized parameter 26%

System Bug 42%

DNS 10%

Admin Error 11%

Compliance Activities

11%

EMS Related

14

WECC BA Vendor Base / Events

BA Base

Events 0%

10%

20%

30%

40%

BA Base

Events

15

• No single root cause.

• Failover issues.

• Increased failures due to network [complexity].

• Increased vendor reliance

• Multiple issue problems – testing > power>

incorrectly sized parameter / human interaction.

Root cause / main common themes

16

• Group involved earlier in process to review and or

ask questions of entity.

• Participation in NERC EA Team.

• Compiling more granular info.

Current team status

17

Team Questions?

Jon Aust – WECC EMSWG Chair

aust@wapa.gov

Stacen Tyskiewicz – WECC DEWG Chair

setyskiewicz@bpa.gov

Vern Kissner – WECC CIIMS Chair

vkissner@cityoftacoma.org