Mobile Virtual Packet Core - Cisco Live

Christian Falckenberg, David Perez Gil

SP Mobility Architects

BRKSPM-2539

Deployment Challenges and Best Practices on Openstack / VMware

Mobile Virtual Packet Core

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSPM-2539

Our own quote

“A virtual / cloud-native mobile core is foundational to building an agile network platform, which enables operators to differentiate themselves. However, it must go hand in hand with a transformation of the way SPs procure, design and operate their networks”

3

Agenda

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSPM-2539 4

• Introduction: The need for an agile network

• Virtualization Options & Cloud-native evolution

• Virtualization Considerations:

• Optimization techniques in a virtualized environment

• Security

• Service Assurance

• Service Provider Transformation Requirements:

• Procurement

• Design / Dimensioning

• Operations

• Conclusion

The Need for an Agile Network


Increased Network / Service Complexity

7

UPF

AMF SMF PCF

UPF

AF

(R)ANUE

N1N2

N4

N9

N3N6

DN

NSSF

NG- RAN, Fixed

Nnssf NausfNudm

Nnef Nnrf

Namf Nsmf Npcf Naf

NEF NRF

APIs (HTTP/JSON)

BSF

NbsfSEPP

Oth

er

PL

MN

Nudr

AUSF UDM UDR NWDAF

Nwdaf

CHF

Nchf

SMSF

Nsmsf

N3IWF

SGSN GGSN


Diverse & Divergent Use Cases

8

• Support evolution of current consumer business models

• Place network resources closer to point consumption

• „Slice“ to tailor network to wide range of use case requirements

• Expand business model:

• return on investment skewed towards the vertical enterprise

Source: ITU


Enterprises are embracing digitisation

• Improve enterprise efficiency and process

• Integration of IoT

• Holistic view of customers, product, manufacturing, partners and employees

All summed up as Industry 4.0:

• Software programmable (SD-everything)

• Identity, context and policy driven

• Multi-hybrid cloud based

• Dynamically extendable over any network

Enterprise Digitisation Journey

External ConnectivityMultiple

Access Internet

3rd Partyproviders

CloudProviders

Enterprise Digitized Platform

Customers

Employees Partners

Own DC

People

Things

External Connectivity

Digital Platform Services

API Catalogue/Services Gateway

SupplyChain Apps

Customer Service Apps

IoTApps

OEM MfgApps

9


The SP as Digital Service ProviderEnterprise Needs

Reach

• Extension of their network into the SP Domain

Control

• Over their devices and identities in the SP Domain

• Policies on security, identity and device

Multi Access

• Wi-Fi and 5G NR are complementary

• 5G NR—determinism

• Wi-Fi for enterprise ubiquity

SP Opportunity

• The SP has the ability to scale to serve multiples of such platforms

Digital Platform

Enterprise Domain SP Domain

ManufacturingSegment

Cisco DNA-C

Supply Chain

Segment

control

control

Partner Segment

control

IoTSegment

control

Ent: Policy & Identity

Device List

Ent. API GW

ViptelaSD-WAN

Supply Chain Slice

Manufacturing Slice

Partners Slice

IoT Slice

ULTRA5G Core

Edge Cloud

Edge Cloud

UPF

Edge Cloud

UPF

UPFEdge Cloud

UPF

AggregationN/W

Wi-Fi

LTE/ 5GNR

control

10


Example: Factory Automation

Customer

premises

Enterprise

Mgmt Portal

LANPolicy

Security

Wifi + CBRS Apps

API

EPC

NTC

SP Edge5-10ms

Service

Chain

Network

Cloud UPFMobile Control Plane

RAN on PremUltra Low latency

Guaranteed QoS

Enterprise Security

IT defines policy

Visibility, Control and Assurance

Service AgilityTraffic stays localBetter reliability

Traffic segmentation Network Slicing

Problem | Security, privacy & reliability for industrial automation

Solution | SP managed private Mobility

Benefits | Deterministic, secured managed network

11

Virtualization Options & Cloud-native Evolution


Virtualization is a Journey

13


Network Virtualization Phases

14

Source: Analysis Mason, 2018


The Road to Virtualization: PNF –> VNF MigrationCisco VPC-DI (since 2014)

• Same software architecture as known already from ASR5000 and ASR5500

• Hardware functions like NPU replaced by software

VM1 – Control Function (Active)

Hypervisor

Controller Tasks:

VPN, Port, Session

High Availability

TasksResource

Manager

VM2 – Control Function (Standby)

Hypervisor

Controller Tasks:

VPN, Port, Session

High Availability

Tasks

Resource

Manager

VM3 – Service Function (Standby)

Hypervisor

SessMgr

(standby)

AAAmgr

(standby)

VM4 – Service Function (Demux)

Hypervisor

Demux

processesNPUsim

VM5 – Service Function (Sessions)

Hypervisor

SessMgr AAAmgr

NPUsim


Hypervisor

SessMgr AAAmgr

NPUsim


Hypervisor

SessMgr AAAmgr

NPUsim


Hypervisor

SessMgr AAAmgr

NPUsim

Fabric interconnectOAM Traffic

“Internal” backplane

15


Compute, Network & Storage Hardware

Cisco UCS C

VIM

Redhat OpenStack

VNFM

Cisco

ESC

VNF

VPC (MME, SGSN, SGW, PGW)

Element Manager

De

plo

y G

UI

Pre-integrated,

tested & validated

by Cisco

Single support

point of contact

The Road to Virtualization: Turnkey SolutionsCisco Ultra-M

16


Cloud Native PrinciplesMicroservices

• Modular, loosely coupled software services

• Individually deployed and lifecycle managed

Containers

• Virtualization and management of Microservices

• Highly portable to different deployment targets

Continuous Delivery

• Automated integration, validation of containers

DevOps

• Automate and manage rapid deployments

• Isolate production changes and deploy once validated

DevOps Continuous Delivery

Microservices Containers

Cloud-Native

17


Cloud Native Benefits

Stateless Application Services

High Performance

Lightweight and Fast

Multi-Cloud Portability

Lifecycle Automation Automated instantiation and placement, upgrade, scale and recovery.

Mobile core disaggregation to the edge across public, private, and hybrid clouds.

Extremely fast startup times improves recovery and scaling event handling.

Bypass the hypervisor overhead when deployed on bare metal. VPP based forwarding plane and vswitch.

State services are separated from the application processing to simplify manageability.

18


Mobile Core Cloud Native Architecture

19

OpenStack NFVI

Hardware

Infrastructureas a Service

Common MobilityPlatform

Common Services

Protocol Load Balancing

Control Plane App Services

SMF AMF

PCF NRF

NSSF NEF

User Plane Services

UPF

TCP Opt

Security

Service Based APIs

Bare Metal OS

SBA

Alerting

Metrics Visualization

Helm Package Management

Common Datastore

API Gateway

Logging

Istio


State Separation Architecture

• Common data layer API exposed to NFs

• Session state can be shared as needed

• Pluggable DB Engines – MongoDB and REDIS

• Ephemeral and long-lived persistence

• Right amount of state in the right place

• Central vs. Local deployment

• Separate lifecycle Management

• HA/Geo-HA built-in

Common Data Layer(grpc/http2)

PCF AMF SMF

MongoDB REDIS

Pluggable DB Engines

What ? Why ?

Other DB

20


Cloud Native 4G/5G Node Design Example

21

Foundation (K8s Resources)

Monitoring/Telemetry

REST EP

Logging (Kibana)

Converged Helm Charts

Common Services

5G containers / store

4G containers / store

Inte

gra

ted

/co

mm

on O

AM

and

Life

cycle

(launch, u

pg

rad

e, c

onfig

ure

, mo

nito

r)

Diameter EP

Application Layer – Workers

Datastore

Composite VNF

Common Infrastructure, OAM and lifecycle management

Common generic services like monitoring, telemetry

App-specific containers with own distinct service configs

Common or distinct shared data layer depending on VNF reqs

Dimensioning of composite VNF is an aggregate of both app dimensioning’s


“Cloud native” means “lots of open source”

22

Virtualization Considerations


Virtualization Considerations

25


VLNP dilemmaVery Large Number of Permutations

Variable Description Variants Count

N(EXGW) Number of VNF Gateway Functions PGW, SGW, ePDG, MME 4

N(MANO) Number of NFV-O MANO Environments HP NFVD, ECM, NetCracker, RHEL Tacker HEAT,NSO/ESC

5

N(VIM) Number of Virtual Infrastructure Managers OpenStack Mercury, Ubuntu OpenStack, vSphere 3

N(HWCPU) Number of hardware compute combinations UCS-B, USC-C, HP C7000, HP DL360, Dell 630 5

N(HWNIC) Number of hardware network combinations Intel Niantic/Fortville/RRC (10 GbE/40 GbE/100 GbE),Cisco VIC

4

N(DPDK) Number of DPDK modes DPDK SR-IOV, DPDK PCI-PT, DPDK vSwitch + VHOST-user

4

N(EPA) Number of hardware EPA attributes Huge Pages, vCPU Pinning, IOMMU, DDIO 4

28

Examples above result in 19,200 different combinations

Impossible to test and measure performance for all


I/O mechanisms

• PCI-Passthrough is not recommended because it requires a physical interface per virtual port

• SR-IOV achieves best performance because it accesses virtual ports on physical NIC

• Virtual switch provides the most flexibility but is slower

Virtual Switch PCI-PassthroughSR-IOV

(Single Root I/O Virtualization)

Passes ThroughOpen vSwitch

Single PortPhysical NICs

Single PortPhysical NIC

(SR-IOV Capable)

29


Data Plane Development Kit (DPDK)

• DPDK can be used inside the VNF application (now the default for Cisco VPC) and in the virtual switch (e.g. OVS+DPDK)

• Key concept is the Poll Mode Driver (PMD): continuously scan the NIC for new packets instead of waiting for interrupt

SR-IOV with DPDK enabled VNFsDPDK-accelerated OVS with DPDK enabled VNFs

30


Compute Optimizations

• Configure NUMA for fast local memory access for CPUs

• Enable huge pages for fast memory searches

• Pin the vCPUs to physical CPU cores/threads

NUMAHuge Pages

CPU Pinning

31


Non-Uniform Memory Access• Each CPU in the node has fast access to local memory

• Slower access to remote node’s memory is used less often

• Most of the data used by any CPU will be stored and accessed locally

Local

Memory

system interconnect

Core 1 Core 2

Core 3 Core 4

CPU 0

Core 1 Core 2

Core 3 Core 4

CPU 1

NUMA Node 0 NUMA Node 1

Local

Memory

0 1 2 3I/O device I/O device

32


Huge Pages

• 4KB is default page size, but can support 2MB or 1GB

• Larger pages reduce the number of pages, shrinking page table size

• Smaller table size means faster searches

• Larger pages increase cache hits and reduce cache misses

• Can be locked to prevent swapping

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

1GB 1GB 1GB

Huge Page Table

4KB Pages 1GB Pages

33


What is a Virtual CPU?

• 2 Sockets (physical CPUs)

• 8 Cores per Socket

• 2 Threads per Core (hyper-threading)

T1

T2

T1

T2

T1

T2

T1

T2

Core 0 Core 1 Core 2 Core 3

T1

T2

T1

T2

T1

T2

T1

T2


Socket 1 Socket 2

2 x 8 x 2 = 32 vCPU

Server with dual Intel E5-

2690 8-core processors

w/Hyper-threading

T1

T2

T1

T2

T1

T2

T1

T2


T1

T2

T1

T2

T1

T2

T1

T2


34


CPU Pinning

Socket 1 Socket 2

HypervisorVM

2 vCPUs

VM

4 vCPUs

T1

T2

T1

T2

T1

T2

T1

T2


T1

T2

T1

T2

T1

T2

T1

T2


T1

T2

T1

T2

T1

T2

T1

T2


T1

T2

T1

T2

T1

T2

T1

T2


35


Optimization Techniques: SummaryVMware OSP Comments

VMXNet3 N N/A Para-virtual network interface card (vNIC)

PCI Passthrough

Y NImproves performance by providing a guest VM exclusive access to a NIC. This option bypasses the hypervisor layer for I/O, which disables advanced features such as vMotion, DRS, Fault Tolerance, HA, Snapshots, etc.

SRIOV N Y SR-IOV makes a single PCI hardware device appears as multiple virtual PCI devices. It provides near line-rate performance to the VNF, at the expense of software abstraction flexibility (requires SR-IOV capable cards)

DPDK Y Y Supported inside the VNF application. Allows vNICs to bypass the Linux kernel in the guest operating system, which gives the application direct access to the vNIC.

NUMA Y Y

Provides fast local memory access for CPUs. When running workloads on NUMA hosts it is important that the CPUs executing the processes are on the same node as the memory used. This ensures that all memory accesses are local to the NUMA node and thus not consuming the limited cross-node memory bandwidth, for example via Intel QuickPath Interconnect (QPI) links, which adds latency to memory accesses.

CPU pinning Y Y Allows processes/threads to have an affinity configured with one or multiple cores. By configuring a CPU affinity, the scheduler is now restricted to only scheduling the thread to execute on one of the nominated cores.

36


5G Security Landscape

• Bigger threat surface due to increased network complexity (multiple slices / access technologies, network distribution)

• DDOS threat magnified due to higher data rates / massive IoT Use Cases

• Scope of security compliance goes beyond conventional IT security metrics

• Virtualized workloads bring a new set of threats:

• VM hopping

• Compromised micro-services

• VM / container image vulnerabilities

• Visibility, segmentation, abnormal flow detection provide foundational security layer

37


Security: Network

• External interfaces similar to non-virtualized deployment

• Intra-VNF interfaces exposed as traffic needs to traverse DC infrastructure:

• Eavesdropping / sniffing / data tampering, in particular on X1, X2 and X3 interfaces.

• CF / SF impersonation

• Rogue image injection when images are exchanged between CF and SF.

Standby CFActive CF

LI

User

SSH

(crypt)

MSGR

(clear)

MSGR

(clear)

MSGR

(clear)

SF

...

SF

... ...

MSGR (clear)

MSGR (c

lear)M

SG

R (clear)

MS

GR

(cle

ar)

DInet exposure

DInet exposure

SCTSSH/CLI SCTRL SCT

SMGR SMGR SMGR SMGR

• CF: Control Function

• SF: Service Function

38


The Service Assurance Challenge

39


Manuel Castells

“The 21st century will not be a dark age. Neither will it deliver to most people the bounties promised by the most extraordinary technological revolution in history. Rather, it may well be characterized by informed bewilderment”

40


Service Assurance

41

• Provide monitoring / root-cause analysis in a dynamically changing network

• Enable orchestration / automation feedback loops

• Designed as loosely coupled modules with open APIs for easy integration

• Enable autonomous operations through machine learning / AI

Source: Analysis Mason, 2018


Service Assurance in a Cloud-Native Environment

42


Probes and Tracing

SMF (PGW-C)

3rd Party Collector

CLI / NetConf Yang API

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

E2E Traces using Dynamic Tracing Capability

Data StoreTrace output

Visualization

43


BuildDevelop

FRPlanning & Design

AutomationDevelop

AutomationExecution

Report &Analysis

Deploy

Operate

RapidRegression

Test

∞

ERCritical Fixes

Automated Release Lifecycle Management

44


Continuous Automation and Integration Testing

• Consulting effort on developing Test cases Leverages library of Cisco-developed keywords including keywords for NSO, traffic generation, and 3rd

party devices

• OVA images that contain all the components necessary for automation execution including job runners

• Documented APIs for easy integration into other toolsets and customer CI/CD models

• All execution runs stored internally as artifacts that can be accessed at any time

• Tests cases are searchable and re-usable between automation runs

Simple Development Language with ROBOT

Integrated Framework with Documented API

Configure, Manage, and Store Results in a Test

Manager UI

45


NFV Management & Orchestration

NFV-O(NFV Orch.)

Virtual Infra Manager

VNF 1 VNF 3VNF 2

NSO (NFV Orch.)

OpenStack / vCenter (VIM)

On-V

iVNF-M (VNF

Managers)

VNF-M (VNF

Manager)F Managers)

ESC

(VNFM)

OSS / BSS

NFVI

Hardware Resources

Compute NetworkStorage

MME Other VNFPGW

Os-Ma

Ve-Vnfm

Nf-ViVirtualization Layer

KVM / ESXi Virtual NetworkVirtual Storage

Vn-Nf

Se-Ma

Nfvo-Vnfm

• Modular architecture that conforms to ETSI NFV framework

• Model driven design for declarative NFV orchestration

• Supports Cisco and 3rd party VNF Managers

• Supports Cisco and 3rd party VNFs

• Offer rapid VNF on-boarding

• Supports Day 1 and Day N VNF Configuration

• Provides Performance and KPI Analytics for Mobile Packet core and Infra

• Integrates Fault analytics for a Mobile Packet core and Infra fault detection & correlation VNF and PNF

NSSF

SMF

NRF BSF

SEPP AMFUPF

OWM

EMS 1 EMS 3EMS 2

Service, VNF & Infrastructure Description

Cisco Matrix

PM and collectionand KPI Analytics

Cisco CrossworkSituation Manager

Analytic a Fault & Root Cause Analyzes

Cisco NSO

VNF & PNF DAY1 –DAY n Config

Operations: Management & Orchestration Framework

46

Vnfm-Vi


STEP 01Manual MOP execution

STEP 02Mass upgrade impacting all sessions

STEP 03Verify to see if everything went ok

STEP 04If it fails, begin rollback of software

Manual S

oft

ware

Upgra

des

COMPLEXOPERATION

HIGHLYRISKY

Auto

mate

d, In

cre

menta

l Upgra

des

Instantiate a new VNF instance with the

upgraded software

Direct a small portion of the subscribers/sessions to the upgraded VNF

2%

Monitor the upgraded VNF & verify key KPIs

Incrementally add more sub. / sessions to the upgraded VNF Canary

Retire original VNF instance once all sub. have been migrated

TOO MANYADMINS

Operations: Automated Upgrades

47


Operations: Canary Upgrade Example SUPI Hash Based Routing

48

SMF

SMF Service

SMF

SMF ServiceSMF Service

SUPI Hash Prefix = 3 SUPI Hash Prefix != 3

All calls routed to same SMF service

Default Behavior Canary Rule Injected

Selective routing of calls is done using ISTIO intelligent service mesh

SP Transformation Requirements


• Skills gap running the platform

• Changing people’s paradigm

• Clouds are very different to traditional telco!

• Getting silo’d groups to work together

• Getting a consistent deployment process and post deployment customizations

SP Transformation Requirements in the Virtualized World

50


• Software is only updated once a year

• Although vendors are releasing software with new features and big fixes now almost monthly, operators are still not prepared to upgrade more than once a year

• Software is tested for weeks and months in a lab

• Any change is done very carefully

• Network freezes prevent any changes for months

• Even smallest configuration changes are done in maintenance windows at night

Service Provider transformationChanging network software and configurations

52

• Implement automation pipelines to automatically receive new software updates and perform initial testing

• Use Canary testing approach to validate new software and any changes in the field

• Improve system resiliency to tolerate failures instead of trying to avoid them

Today Alternative


• Vendor selection cycles are way too long:

• Many of the larger operators went through multiple rounds of vendor selections with durations of 12-18 months; from RFQ preparation to deployment this easily meant two years where technology changed significantly

• 5 year horizon is far too long for TCO calculation and contract commitment

• Many aspects required for price calculation change significantly (Hardware, optimization technologies, software enhancements)

• Vendor landscape and political selection criteria may change significantly

Service Provider transformationVendor integration and procurement

53

• Adding VNFs to an existing Telco cloud environment is much simpler than installing hardware

• Vendor products can be compared and tested (e.g. for performance) under same conditions

• Vendors need to prove that all product promises can actually be fulfilled

• Pricing can be made much more flexible (per month/day/hour instead of buying perpetual licenses)

Today Alternative


• Throughput performance is a key marketing and decision factor

• “Gigabits” per subscriber (nice for demos but who needs that?)

• RFQs demand hardware dimensioning even though

• Forecasts may change

• Hardware platforms will change

• Architectures may change

• Call models have impacts but very few inputs are given or are reliable

Service Provider transformationCapacity planning

54

• Consider all the other scenarios where number of sessions, transaction rates etc. are more important than just throughput

• Consider call model impacts

• Test the performance of different vendors in lab environment instead of relying on a theoretical RFQ dimensioning

• Use real production data to extrapolate capacity needs once deployed

Today Alternative

Conclusion


Conclusion

• Traditional network appliance based infrastructure unsuitable for the future

• A Telco Cloud is required to meet SP goals around network agility, flexibility, efficiency, and scalability

• The move to Telco Cloud is a journey. Virtualization of network functions is a first step towards a cloud-native architecture ready to be provisioned, ported across Clouds

• Continuous Automation across all domains key to delivering on the benefits of the Telco Cloud environment

• The shift to cloud- and software-led networking requires that Service Providers reskill and adjust their processes / organisation

56


Don’t miss the Service Provider Hub !

Demos

Experience 7 Essential Technology and 3 Generate

Revenue with 5G demos and join our guided demo tours

More Sessions

Check out the Service Provider Technology Track by scanning the code

5G Virtual Reality Experience

Enjoy “Running with the 5G Bull”

immersive demo

Digital Transformation Assessment

Take a meeting session to benchmark your

digital readiness against your industry peers

Want to see use cases, solution details and more. Visit www.cisco.com/go/sp

https://cs.co/dna-personal-demo

http://www.cisco.com/go/sp


Cisco Webex Teams

Questions? Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session

Find this session in the Cisco Events Mobile App

Click “Join the Discussion”

Install Webex Teams or go directly to the team space

Enter messages/questions in the team space

How

1

2

3

4

58

cs.co/ciscolivebot#BRKSPM-2539


Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at ciscolive.cisco.com

• Please complete your Online Session Survey after each session

• Complete 4 Session Surveys & the Overall Conference Survey (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Events Mobile App or the Communication Stations

Complete your online session survey

59


Demos in the Cisco Showcase

Walk-in self-paced

labs

Meet the engineer

1:1 meetings

Related sessions

Continue Your Education

60

Thank you

Mobile Virtual Packet Core - Cisco Live

Documents

Transcript of Mobile Virtual Packet Core - Cisco Live