OPENSHIFT CONTAINER PLATFORM - Technical Deep Dive

OPENSHIFT CONTAINER PLATFORM

TECHNICAL DEEP-DIVE

OPENSHIFT TECHNICAL OVERVIEW2

Self-Service

Multi-language

Automation

Collaboration

Multi-tenant

Standards-based

Web-scale

Open Source

Enterprise Grade

Secure


ANYCONTAINER

Amazon Web Services Microsoft Azure Google CloudOpenStackDatacenterLaptop

ANYINFRASTRUCTURE

APPLICATION LIFECYCLE MANAGEMENT

ENTERPRISE CONTAINER HOST

CONTAINER ORCHESTRATION AND MANAGEMENT(KUBERNETES)




Automated Operations*

Kubernetes

Red Hat Enterprise Linux or Red Hat CoreOS

Application Services

CaaS PaaSBest IT Ops Experience Best Developer Experience

*coming soon

Cluster Services

Developer Services

Middleware, Service Mesh, Functions, ISV Metrics, Chargeback, Registry, Logging Dev Tools, Automated Builds, CI/CD, IDE


OPENSHIFT ARCHITECTURE

EXISTING AUTOMATION

TOOLSETS

SCM(GIT)

CI/CD

SERVICE LAYER

ROUTING LAYER

PERSISTENTSTORAGE

REGISTRY

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

C

C

C C

C

C

C CC C

RED HATENTERPRISE LINUX

MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

PHYSICAL VIRTUAL PRIVATE PUBLIC HYBRID

LINUX CONTAINERS


WHAT ARE CONTAINERS?It Depends Who You Ask

● Application processes on a shared kernel

● Simpler, lighter, and denser than VMs

● Portable across different environments

● Package apps with all dependencies

● Deploy to any environment in seconds

● Easily accessed and shared

INFRASTRUCTURE APPLICATIONS


VIRTUAL MACHINES AND CONTAINERS

VIRTUAL MACHINES CONTAINERS

VM isolates the hardware Container isolates the process

VM

OS Dependencies

Kernel

Hypervisor

Hardware

App App App App

Container Host (Kernel)

Container

App

OS deps

Container

App

OS deps

Container

App

OS deps

Container

App

OS deps

Hypervisor

Hardware


Virtual Machine

Application

OS dependencies

Operating System


VM IsolationComplete OSStatic ComputeStatic MemoryHigh Resource Usage

Container IsolationShared KernelBurstable ComputeBurstable MemoryLow Resource Usage

Container Host

Container

Application

OS dependencies



Container Host

Container

Application

OS dependencies

Dev

IT OpsInfrastructure

Virtual Machine

Application

OS dependencies

Operating System

IT Ops(and Dev, sort of)

Infrastructure

Clear ownership boundary between Dev and IT Ops drives DevOps adoption

and fosters agility

Optimized for stability

Optimized for agility


Virtual machines are NOT portable across hypervisor and do NOT provide portable packaging for applications

APPLICATION PORTABILITY WITH VM

VM Type X

Application

OS dependencies

Operating System

BARE METAL PRIVATE CLOUD PUBLIC CLOUDVIRTUALIZATIO

NLAPTOP

Application

OS dependencies

Operating System

VM Type Y

Application

OS dependencies

Operating System

VM Type Z

Application

OS dependencies

Operating System

Guest VM

Application

OS dependencies

Operating System


APPLICATION PORTABILITY WITH CONTAINERS

LAPTOP

Container

Application

OS dependencies

Guest VM

RHEL

BARE METAL

Container

Application

OS dependencies

RHEL

VIRTUALIZATION

Container

Application

OS dependencies

Virtual Machine

RHEL

PRIVATE CLOUD

Container

Application

OS dependencies

Virtual Machine

RHEL

PUBLIC CLOUD

Container

Application

OS dependencies

Virtual Machine

RHEL

RHEL Containers + RHEL Host = Guaranteed PortabilityAcross Any Infrastructure


LINUX AND CONTAINER INFRASTRUCTURE

CONTAINERS ARE LINUX

Red Hat Enterprise Linux is a leader in paid

Linux

70%CY2016 paidLinux share

CONTAINER CONTAINER CONTAINER

LINUX CONTAINER HOST (KERNEL)

LINUX O/SDEPENDENCY

LINUX O/SDEPENDENCY

LINUX O/SDEPENDENCY

APP APP APP

Linux OS host spans every container

1 2Linux is in

every single container


Base Image

Image Layer 1

Image Layer 2

Image Layer 3

Base RHEL

OS Update Layer

Java Runtime Layer

Application Layer

Container Image Layers Example Container Image

RAPID SECURITY PATCHING USINGCONTAINER IMAGE LAYERING

OPENSHIFT TECHNICAL OVERVIEW

A lightweight, OCI-compliant container runtime

15

Minimal and Secure Architecture

Optimized for Kubernetes

Runs any OCI-compliant image

(including docker)

Optional runtime in OCP 3.10, default OCP 3.11+

OPENSHIFT CONCEPTS OVERVIEW


A container is the smallest compute unit

CONTAINER


containers are created from container images

CONTAINERCONTAINERIMAGE

BINARY RUNTIME


IMAGE REGISTRY

container images are stored in an image registry

CONTAINER

CONTAINERIMAGE

CONTAINERIMAGE

CONTAINERIMAGE

CONTAINERIMAGE

CONTAINERIMAGE

CONTAINERIMAGE


an image repository contains all versions of an image in the image registry

IMAGE REGISTRY

frontend:latestfrontend:2.0frontend:1.1frontend:1.0

CONTAINERIMAGE

mongo:latestmongo:3.7mongo:3.6mongo:3.4

CONTAINERIMAGE

myregistry/frontend myregistry/mongo


PODPOD

containers are wrapped in pods which are units of deployment and management

CONTAINER CONTAINERCONTAINER

IP: 10.1.0.11 IP: 10.1.0.55


pods configuration is defined in a deployment

image namereplicaslabelscpumemorystorage

POD

CONTAINER

POD

CONTAINER

POD

CONTAINER

DEPLOYMENT


services provide internal load-balancing and service discovery across pods

POD

CONTAINER

POD

CONTAINER

POD

CONTAINER

BACKEND SERVICE

POD

CONTAINER

role: backend

role: backendrole: backendrole: backendrole: frontend

10.110.1.11 10.120.2.22 10.130.3.3310.140.4.44

172.30.170.110


apps can talk to each other via services

InvokeBackend API

POD

CONTAINER

POD

CONTAINER

POD

CONTAINER

BACKEND SERVICE

POD

CONTAINER

role: backend

role: backendrole: backendrole: backendrole: frontend

10.110.1.11 10.120.2.22 10.130.3.3310.140.4.44

172.30.170.110


POD

routes add services to the external load-balancer and provide readable urls for the app

CONTAINER

POD

CONTAINER

POD

CONTAINER

BACKEND SERVICE

ROUTEapp-prod.mycompany.com

> curl http://app-prod.mycompany.com


projects isolate apps across environments, teams, groups and departments

POD

C

POD

C

POD

C

PAYMENT DEV

POD

C

POD

C

POD

C

PAYMENT PROD

POD

C

POD

C

POD

C

CATALOG

POD

C

POD

C

POD

C

INVENTORY

OPENSHIFT ARCHITECTURE


YOUR CHOICE OF INFRASTRUCTURE



NODES RHEL INSTANCES WHERE APPS RUN

29

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE



RHEL

NODE

c

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

C

C

C C

C

C

C CC C

APPS RUN IN CONTAINERS

30

Container Image

Container

Pod


PODS ARE THE UNIT OF ORCHESTRATION

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

C

C

C C

C

C

C CC C


RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

32

MASTERS ARE THE CONTROL PLANE


MASTER



RHEL

NODE

RHEL

NODE

RHEL

NODE

33

API AND AUTHENTICATION

RHEL

NODE

RHEL

NODE

RHEL

NODE


MASTER

API/AUTHENTICATION



RHEL

NODE

RHEL

NODE

RHEL

NODE

34

DESIRED AND CURRENT STATE

RHEL

NODE

RHEL

NODE

RHEL

NODE



MASTER

API/AUTHENTICATION

DATA STORE



INTEGRATED CONTAINER REGISTRY

RHEL

NODE

RHEL

NODE

RHEL

RHEL

NODE

RHEL

NODE

RHEL

RHEL

NODE



MASTER

API/AUTHENTICATION

DATA STORE

NODE

REGISTRY

RHEL


ORCHESTRATION AND SCHEDULING

RHEL

NODE

RHEL

NODE

RHEL

RHEL

NODE

RHEL

NODE

RHEL

RHEL

NODE



MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

NODE

REGISTRY

RHEL


PLACEMENT BY POLICY

RHEL

NODE

RHEL

NODE

RHEL

NODE



MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULERREGISTRY

RHEL

NODE

RHEL

NODE

C

C

RHEL

NODE

c

C

C


RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

C

C

RHEL

NODE

c

C

C

38

AUTOSCALING PODS



MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULERREGISTRY

HEALTH/SCALING


SERVICE DISCOVERY

SERVICE LAYER

REGISTRY

RHEL

NODE

C

C

RHEL

NODE

C C

RHEL

NODE

c

C

C

RHEL

NODE

C C

RHEL

NODE

C

RHEL

NODE

CRED HATENTERPRISE LINUX

MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING



PERSISTENT DATA IN CONTAINERS

SERVICE LAYER

PERSISTENTSTORAGE

REGISTRY

RHEL

NODE

C

C

RHEL

NODE

C C

RHEL

NODE

c

C

C

RHEL

NODE

C C

RHEL

NODE

C

RHEL

NODE


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING



ROUTING AND LOAD-BALANCING

SERVICE LAYER

ROUTING LAYER

PERSISTENTSTORAGE

REGISTRY

RHEL

NODE

C

C

RHEL

NODE

C C

RHEL

NODE

c

C

C

RHEL

NODE

C C

RHEL

NODE

C

RHEL

NODE


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING



ACCESS VIA WEB, CLI, IDE AND API

EXISTING AUTOMATION

TOOLSETS

SCM(GIT)

CI/CD

SERVICE LAYER

ROUTING LAYER

PERSISTENTSTORAGE

REGISTRY

RHEL

NODE

C

C

RHEL

NODE

C C

RHEL

NODE

c

C

C

RHEL

NODE

C C

RHEL

NODE

C

RHEL

NODE


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING


OPENSHIFT INSTALLATION ARCHITECTURES


PROOF-OF-CONCEPT ARCHITECTURE

ApplicationTraffic

Dev and OpsUser

INFRAMASTER

NODE NODE

An infrastructure node is a node that is dedicated to infrastructure pods such as router, image registry, metrics, and logs


APP HIGH-AVAILABILITY ARCHITECTURE

ENTERPRISE LOAD-BALANCER

ApplicationTraffic

Dev and OpsUser

INFRAMASTER INFRA

NODE NODE NODE NODE


FULL HIGH-AVAILABILITY ARCHITECTURE

ENTERPRISE LOAD-BALANCER

ApplicationTrafficDev and Ops

User

NODE

MASTER MASTER INFRAMASTER INFRA

NODE NODE NODE NODENODE

INFRA

NODE

TECHNICAL DEEP DIVE

MONITORING APPLICATION HEALTH


AUTO-HEALING FAILED PODS

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

C

C

RHEL

NODE

C

C


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

C


AUTO-HEALING FAILED CONTAINERS

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

C

C

RHEL

NODE

C

C


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

C



RHEL

NODE

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

C

C

RHEL

NODE

C

C


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

C



RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

C

C

RHEL

NODE

C

C

c


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

C

c

NETWORKING


BUILT-IN SERVICE DISCOVERYINTERNAL LOAD-BALANCING

SERVICE

app=payroll role=frontend

POD

app=payroll

role=frontend

POD

app=payroll

role=frontend

Name: payroll-frontendIP: 172.10.1.23Port: 8080

POD

app=payroll

role=backendversion=1.0 version=1.0


BUILT-IN SERVICE DISCOVERYINTERNAL LOAD-BALANCING

SERVICE

app=payroll role=frontend

POD

app=payroll

role=frontend

POD

app=payroll

role=frontend

POD

app=payroll

role=frontend

Name: payroll-frontendIP: 172.10.1.23Port: 8080

POD

app=payroll

role=backendversion=2.0 version=1.0 version=1.0


SERVICE

POD POD

ROUTER

POD

EXTERNAL TRAFFIC

INTERNAL TRAFFIC

ROUTE EXPOSES SERVICES EXTERNALLY


● Pluggable routing architecture○ HAProxy Router○ F5 Router

● Multiple-routers with traffic sharding● Router supported protocols

○ HTTP/HTTPS○ WebSockets○ TLS with SNI

● Non-standard ports via cloud load-balancers, external IP, and NodePort

ROUTING AND EXTERNAL LOAD-BALANCING


ROUTE SPLIT TRAFFIC

SERVICE A

App A App A

SERVICE B

App B App B

ROUTE

10% traffic90% traffic

Split Traffic Between Multiple Services For A/B Testing, Blue/Green and Canary Deployments


● NodePort binds a service to a unique port on all the nodes

● Traffic received on any node redirects to a node with the running service

● Ports in 30K-60K range which usually differs from the service

● Firewall rules must allow traffic to all nodes on the specific port

60

EXTERNAL TRAFFIC TO A SERVICE ON A RANDOM PORT WITH NODEPORT

NODE192.10.0.12

NODE192.10.0.11

NODE192.10.0.10

SERVICE

INT IP: 172.1.0.20:90

POD

10.1.0.1:90

POD

10.1.0.2:90

POD

10.1.0.3:90

connect192.10.0.10:31421192.10.0.11:31421192.10.0.12:31421

CLIENT


NODE192.10.0.12

NODE192.10.0.11

NODE192.10.0.10

61

EXTERNAL TRAFFIC TO A SERVICE ON ANY PORT WITH INGRESS

SERVICE

EXT IP: 200.1.0.10:90INT IP: 172.1.0.20:90

POD

10.1.0.1:90

POD

10.1.0.2:90

POD

10.1.0.3:90

connect200.1.0.10:90

CLIENT● Access a service with an external IP on any TCP/UDP port, such as○ Databases○ Message Brokers

● Automatic IP allocation from a predefined pool using Ingress IP Self-Service

● IP failover pods provide high availability for the IP pool


CONTROL OUTGOING TRAFFIC SOURCE IP WITH EGRESS ROUTER

NODEIP1

EGRESSROUTER

PODIP1

EGRESS SERVICEINTERNAL-IP:8080

EXTERNAL SERVICE

Whitelist: IP1

POD

POD

POD


● Built-in internal DNS to reach services by name

● Split DNS is supported via SkyDNS○ Master answers DNS queries for internal services○ Other name servers serve the rest of the queries

● Software Defined Networking (SDN) for a unified cluster network to enable pod-to-pod communication

● OpenShift follows the Kubernetes Container Networking Interface (CNI) plug-in model

OPENSHIFT NETWORKING


OPENSHIFT NETWORK PLUGINS

OpenShift SDN

(OVS)

OPENSHIFT

KUBERNETES CNI

Flannel** NuageTigera

Calico & CNX

JuniperContrail

CiscoContiv &

Contiv-ACIBig Switch

Fully Supported Validated

VMwareNSX-T

In-Progress

DEFAULT

kuryr-kubernetes

OpenShift SDN

(OVN*)

* Coming as default in OCP 4.1** Flannel is minimally verified and is supported only and exactly as deployed in the OpenShift on OpenStack reference architecture

OpenDaylight(CNI & Kuryr)

RH-OSPNeutronPlugin


NODE172.16.1.10

65

OPENSHIFT NETWORKING

POD10.1.2.2

POD10.1.2.4

NODE172.16.1.20

POD10.1.4.2

POD10.1.4.4

IP Network

VxLAN Overlay Network


FLAT NETWORK (Default)

● All pods can communicate with each other across projects

MULTI-TENANT NETWORK

● Project-level network isolation● Multicast support● Egress network policies

NETWORK POLICY

● Granular policy-based isolation

66

OPENSHIFT SDN

NODE

POD POD

PODPOD

NODE

POD POD

PODPOD

PROJECT A PROJECT B

DEFAULT NAMESPACE

✓

PROJECT C

Multi-Tenant Network


PROJECT A

67

OPENSHIFT SDN - NETWORK POLICY

POD

POD

POD

POD

PROJECT B

POD

POD

POD

POD

Example Policies● Allow all traffic inside the project● Allow traffic from green to gray● Allow traffic to purple on 8080

✓

✓

8080

5432

✓

apiVersion: extensions/v1beta1kind: NetworkPolicymetadata:

name: allow-to-purple-on-8080spec:

podSelector:matchLabels:

color: purpleingress:- ports:

- protocol: tcpport: 8080

✓


Container to Container on the Same Host

68

OPENSHIFT SDN - OVS PACKET FLOW

NODE

POD 1veth0

10.1.15.2/24

br010.1.15.1/24

192.168.0.100

eth0

POD 2veth1

10.1.15.3/24

vxlan0


NODE 2

NODE 1

69


POD 1veth0

10.1.15.2/24br0

10.1.15.1/24vxlan0

POD 2veth0

10.1.20.2/24br0

10.1.20.1/24vxlan0

192.168.0.100

eth0

192.168.0.200

eth0

Container to Container on the Different Hosts


Container Connects to External Host

Container to Container on Different Hosts

70


NODE 1

POD 1veth0

10.1.15.2/24br0

10.1.15.1/24tun0

192.168.0.100

ExternalHost

eth0


OPENSHIFT SDN WITH FLANNEL FOR OPENSTACK

NODE 1

POD 1veth0

10.1.15.2/24docker0

10.1.15.1/24Routing Table

flanneld

NODE 2

POD 2veth0

10.1.20.2/24docker0

10.1.20.1/24Routing Table

flanneld

etcd

192.168.0.100

eth0

192.168.0.200

eth0

Flannel is minimally verified and is supported only and exactly as deployed in the OpenShift on OpenStack reference architecture https://access.redhat.com/articles/2743631

https://access.redhat.com/articles/2743631

LOGGING & METRICS


● EFK stack to aggregate logs for hosts and applications

○ Elasticsearch: a search and analytics engine to store logs

○ Fluentd: gathers logs and sends to Elasticsearch.

○ Kibana: A web UI for Elasticsearch.

● Access control

○ Cluster administrators can view all logs

○ Users can only view logs for their projects

● Ability to send logs elsewhere

○ External elasticsearch, Splunk, etc

CENTRAL LOG MANAGEMENT WITH EFK


CENTRAL LOG MANAGEMENT WITH EFK

APPLICATION LOGS

OPERATION LOGS

ELASTICELASTIC

74

RHEL

NODE

POD POD

PODPODFLUENTD

RHEL

NODE

POD POD

PODPOD

FLUENTD

ELASTICSEARCH

RHEL

NODE

POD POD

PODPOD

FLUENTD

USER

ELASTICELASTIC

KIBANA

ELASTICELASTICELASTICSEARCH

ELASTICELASTIC

KIBANA

ADMIN


CONTAINER METRICS


CONTAINER METRICS

76

RHEL

NODE

POD POD

PODPODFLUENTD

CONTAINER METRICS

RHEL

NODE

POD POD

PODPOD

FLUENTD

HEAPSTER

RHEL

NODE

POD POD

PODPOD

CADVISOR

HAWKULAR OPENSHIFTWEB CONSOLE

ELASTICELASTICCASSANDRA

RED HATCLOUDFORMS

CUSTOMDASHBOARDS

API

USER

SECURITY


AUTOMATED & INTEGRATED SECURITY

Container Content

Container Registry

CI/CD Pipeline

Deployment Policies

Security Ecosystem

CONTROLApplication Security

DEFENDInfrastructure

EXTEND

Container Host Multi-tenancyContainer Platform

Network Isolation Storage

Audit & Logging API Management


NODE

MASTER● Secure mechanism for holding sensitive data e.g.

○ Passwords and credentials○ SSH Keys○ Certificates

● Secrets are made available as○ Environment variables○ Volume mounts○ Interaction with external systems

● Encrypted in transit and at rest

● Never rest on the nodes

79

SECRET MANAGEMENT

Container

Distributed Store

Container


CheckExpiry

RedeployCerts

CERTIFICATE MANAGEMENT

● Certificates are used to provide secure connections to

○ master and nodes○ router and registry○ etcd

● Ansible playbooks to automate redeployment

● Redeploy all at once or specific components

● Certificate expiry report generator

MASTER

NODES

ROUTER

REGISTRY

ETCD

✓✓✓✓✓

AnsiblePlaybook


CERTIFICATE CHECKS

● master and nodes

● router and registry service certificates from etcd secrets

● master, node, router, registry, and kubeconfig files for cluster-admin users

● etcd certificates

CERTIFICATE EXPIRY REPORT

PERSISTENT STORAGE


● Persistent Volume (PV) is tied to a piece of network storage● Provisioned by an administrator (static or dynamically)● Allows admins to describe storage and users to request storage● Assigned to pods based on the requested size, access mode, labels and type

PERSISTENT STORAGE

NFS

GlusterFS

OpenStack Cinder

Ceph RBD

AWS EBS

GCE Persistent Disk

iSCSI

Fiber Channel

Azure Disk

Azure File

FlexVolume

VMWare vSphere VMDK

Container Storage Interface

(CSI)*** Shipped and supported by NetApp via TSANet** Tech Preview

NetApp Trident*


PROJECT

POOL OF PERSISTENT VOLUMES

84

PERSISTENT STORAGE

NFSPV

iSCSIPV

NFSPV

Admin

User

register PV

create claim

NFSPV

GlusterFSPV

Pod

claim

Pod

claim

Pod

claim

CephRBDPV


DYNAMIC VOLUME PROVISIONING

Admin

User

define StorageClass

create claim: Fastest

SlowAzure-Disk

FastAWS-SSD

FastestNetApp-Flash

NetAppProvisioner

AWSProvisioner

Pod

claim

PV

OpenShiftPV Controller

provision

AzureProvisioner

bound


● Containerized Red Hat Gluster Storage

● Native integration with OpenShift

● Unified Orchestration using Kubernetes for

applications and storage

● Greater control & ease of use for developers

● Lower TCO through convergence

● Single vendor Support DISTRIBUTED, SECURE, SCALE-OUT STORAGE CLUSTER

APPLICATIONCONTAINER

APPLICATION CONTAINER

APPLICATION CONTAINER

STORAGECONTAINER

STORAGECONTAINER

STORAGECONTAINER

OPENSHIFT CONTAINER STORAGE


NODENODENODE

OPENSHIFT CONTAINER STORAGE

87

NODE

POD POD POD POD POD POD POD

POD POD POD

RHGS RHGS RHGS

POD POD POD

MASTER

SERVICE BROKER


WHY A SERVICE BROKER?

SERVICE CONSUMER

SERVICE PROVIDER

☑ Open ticket☑ Wait for allocation

☑ Receive credentials☑ Add to app☑ Deploy app

Manual, Time-consuming and Inconsistent


A multi-vendor project to standardize how services are consumed on cloud-native platforms across service providers


WHAT IS A SERVICE BROKER?

SERVICE CONSUMER

SERVICE PROVIDER

SERVICE CATALOG

SERVICE BROKER

Automated, Standard and Consistent


OPENSHIFT SERVICE CATALOG

OPENSHIFT SERVICE CATALOG

OpenShift AutomationBroker

OpenShiftTemplateBroker

AWSServiceBroker

OtherServiceBrokers

ANSIBLE

OPENSHIFT

AWS

OTHER COMPATIBLE SERVICES

Ansible Playbook Bundles

OpenShiftTemplates

AWSServices

OtherServices


SERVICE BROKER CONCEPTS

SERVICE CONSUMER

SERVICE PROVIDER

SERVICE CATALOG

SERVICE BROKER

SERVICE: an offering that can be used by an app e.g. database

PLAN: a specific flavor of a service e.g. Gold Tier

SERVICE INSTANCE: an instance of the offering

PROVISION: creating a service instance

BIND: associate a service instance and its credentials to an app


● Deploy service broker on or off OpenShift

● Register the broker referring to the deployed broker

● Register the broker services by creating ServiceClass resources(the service broker might automatically perform this step)

94

HOW TO ADD A SERVICE BROKER

apiVersion: servicecatalog.k8s.io/v1alpha1kind: Brokermetadata:name: asb-broker

spec:url: https://asb-1338-ansible-service-broker.10.2.2.15.nip.io


● Exposes Templates and Instant Apps in the Service Catalog

● Pulled from openshift namespace by default

● Multiple namespaces can be configured for template discovery

TEMPLATE SERVICE BROKER


TEMPLATE SERVER BROKER PROVISIONING

Template Service Broker

Node.jsContainer

openshiftnamespace

nodejs-template

OpenShift Service Catalog

Service Broker creates a the objects from the template


TEMPLATE SERVICE BROKERBINDING

Template Service Broker

Node.jsContainer

openshiftnamespace

nodejs-template


create binding

Service Broker creates a binding and secret for any credentials (config map, secret, etc) created by the template


OPENSHIFT ANSIBLE BROKER

● Use Ansible on OpenShift

○ Deploy containerized applications○ Provision external services (e.g. Oracle database)○ Provision cloud services (e.g. AWS RDS)○ Orchestrate multi-service solutions○ Conditional logic for control on deployments (e.g. database is initialized)

● Leverage existing Ansible playbooks

● Anything you can do with Ansible, you can do with OAB


● Lightweight application definition

● Packaged as a container image

● Embedded Ansible runtime

● Metadata for parameters

● Named playbooks for actions

● Leverage existing Ansible playbooks

● Registry is queried to discover APBs

ANSIBLE PLAYBOOK BUNDLES (APB)

Ansible Playbook Bundle (Container Image)

Ansible Runtime

├─ roles├─ playbooks│ ├─ provision.yaml│ ├─ unprovision.yaml│ ├─ bind.yaml│ └─ unbind.yaml└─ apb.yaml


OpenShift Ansible Broker

Red Hat Container Catalog

Docker H

ubO

penShift Registry

mediawiki-apb

postgresql-apb Discover and list APBs from the configured image registries


OPENSHIFT ANSIBLE BROKER PROVISIONING



APBContainer(postgresql)

oc run postgresql-apb provision $vars

Pull APB image and run it with the broker action as a parameter


Docker H

ubO

penShift Registry

mediawiki-apb

postgresql-apb




Ansible Service Broker


oc run postgresql-apb provision $vars ansible-playbook provision.yaml $vars

PostgreSQL

Container

APB container runsprovision.yamlplaybook to create a PostgreSQL container


Docker H

ubO

penShift Registry

mediawiki-apb

postgresql-apb




PostgreSQL

Container


Docker H

ubO

penShift Registry

mediawiki-apb

postgresql-apb





PostgreSQL

Container

APB container runsbind.yamlplaybook to create database user

oc run postgresql-apb bind $vars ansible-playbook bind.yaml $vars


Docker H

ubO

penShift Registry

mediawiki-apb

postgresql-apb

MediaWikiContainer


OPENSHIFT ANSIBLE BROKER BINDING



PostgreSQL

Container


Docker H

ubO

penShift Registry

mediawiki-apb

postgresql-apb

MediaWikiContainer


mount binding secret

Service Catalog creates a secret for the binding, containing the database credentials




PostgreSQL

Container

APB container goes away and Service Broker creates a binding for the PostgreSQL service


Docker H

ubO

penShift Registry

mediawiki-apb

postgresql-apb

create bindingMediaWikiContainer





PostgreSQL

Container

MediaWiki container uses the credentials in the secret to connect to the PostgreSQL database


Docker H

ubO

penShift Registry

mediawiki-apb

postgresql-apb

MediaWikiContainer


mount binding secret



AWS SERVICE BROKER

● Amazon Athena● Amazon DynamoDB● Amazon ElastiCache● Amazon EMR● Amazon Kinesis Data Streams● Amazon KMS● Amazon Lex● Amazon Polly● Amazon RDS for MariaDB● Amazon RDS for MySQL● Amazon RDS for PostgreSQL● Amazon RedShift● Amazon Rekognition● Amazon Route 53● Amazon S3● Amazon SNS● Amazon SQS● Amazon Translate


AWS PROVISIONING

AWSRDS

Compatible D

ocker R

egistries

oc run rds-apb provision $vars ansible-playbook provision.yaml $vars

APB container runsprovision.yaml playbook to interact with CFN and create RDS instance


APBContainer

(rds)


AWS ECR

s3-apb

rds-apb

AWS Cloud

Formation


AZURE SERVICE BROKER

Available on OpenShift on Azure managed-service and Azure Stack● Azure Cosmos DB● Azure KeyVault● Azure Storage● Azure Redis Cache● Azure DocumentDB● Azure Service Bus and Event Hub● Azure SQL Database● Azure SQL Database Failover Group● Azure Database for MySQL ● Azure Database for PostgreSQL

OPERATOR FRAMEWORK(coming soon)


KUBERNETES OPERATOR FRAMEWORK

Operator Framework is an open source toolkit to manage application instances on Kubernetes in an effective, automated and scalable way.

Installation Upgrade Backup Failure recovery

Metrics& insights Tuning

AUTOMATED LIFECYCLE MANAGEMENT


KUBERNETES OPERATOR FRAMEWORK

Operators codify operational knowledge and workflows to automate lifecycle management of containerized applications with Kubernetes

SDK LIFECYCLE MANAGEMENT METERING


WHY OPERATOR FRAMEWORK?

DEVELOPER DEPLOY

STATEFUL APP

A WHILE

LATER

APP SERVICES OPERATIONS

UPDATEPATCHBACKUPREBALANCESCALE

DEPLOY

STATEFUL APP

UPDATEPATCHBACKUPREBALANCESCALE

APPOPERATORDEVELOPER


OPERATOR LIFECYCLE MANAGER


OPERATOR METERING

● Based on Prometheus

● Reports namespace, pods and custom label query

● Easy to process by accounting or custom software


THE INDUSTRY IS ALIGNING BEHIND THE KUBERNETES OPERATOR FRAMEWORK

60+ Certified ISV Operators in Red Hat Early Access Program

REFERENCE ARCHITECTURES


Application Release Strategies with OpenShift

Building Polyglot Microservices on OpenShift

Building JBoss EAP 6 Microservices on OpenShift

Building JBoss EAP 7 Microservices on OpenShift

Business Process Management with JBoss BPMS on OpenShift

Build and Deployment of Java Applications on OpenShift

Building Microservices on OpenShift with Fuse Integration...

JFrog Artifactory on OpenShift Container Platform

Spring Boot Microservices on Red Hat OpenShift

API Management with Red Hat 3scale on OpenShift

App CI/CD on OCP with Jenkins

OpenShift on VMware vCenter

OpenShift on Red Hat OpenStack Platform

OpenShift on Amazon Web Services

OpenShift on Google Cloud Platform

OpenShift on Microsoft Azure

OpenShift on Red Hat Virtualization

OpenShift on HPE Servers with Ansible Tower

OpenShift on VMware vCenter 6 with Gluster

Deploying an OpenShift Distributed Architecture

OpenShift Architecture and Deployment Guide

OpenShift Scaling, Performance, and Capacity Planning

118

REFERENCE ARCHITECTURES











https://access.redhat.com/documentation/en-us/reference_architectures/2017/html/application_cicd_on_openshift_container_platform_with_jenkins/












BUILD AND DEPLOYCONTAINER IMAGES


BUILD AND DEPLOY CONTAINER IMAGES

DEPLOY YOURSOURCE CODE

DEPLOY YOURCONTAINER IMAGE

DEPLOY YOURAPP BINARY


DEPLOY SOURCE CODE WITH SOURCE-TO-IMAGE (S2I)

GitRepositoryBUILD APP

(OpenShift) Developer

code

Source-to-Image(S2I)

Builder Image

ImageRegistry

BUILD IMAGE(OpenShift)

DEPLOY(OpenShift)

deployApplication Container

OpenShift DoesUser/Tool Does


DEPLOY APP BINARY WITH SOURCE-TO-IMAGE (S2I)

ApplicationBinary

(e.g. WAR)BUILD APP(Build Infra) Existing Build

Process

build

Source-to-Image(S2I)

Builder Image

ImageRegistry

BUILD IMAGE(OpenShift)

DEPLOY(OpenShift)

deployApplication Container



DEPLOY(Openshift)

123

DEPLOY DOCKER IMAGEbuild

Application Container

deploy

Application Image

ImageRegistry

BUILD IMAGE (Build Infra) Existing Image

Build Process

PUSH(Build Infra)



BUILD STAGE 3

BUILD STAGE 2

BUILD STAGE 1

124

BUILD IMAGES IN MULTIPLE STAGES


EXAMPLE: USE ANY RUNTIME IMAGE WITH SOURCE-TO-IMAGE BUILDS

DOCKER BUILDWILDFLY S2I BUILD app.war

WildFly S2IBuilder Image

WildFly Runtime Image

Use Source-to-Image to build app binaries and deploy on lean vanilla runtimes

read more on https://blog.openshift.com/chaining-builds/

https://blog.openshift.com/chaining-builds/


EXAMPLE: USE ANY BUILD TOOL WITH OFFICIAL RUNTIME IMAGES

DOCKER BUILDCUSTOM GRADLE BUILD

CustomGradle S2IBuilder Image

Red HatOpenJDKImage

Use your choice of build tool like Gradle and deploy to official images like the JDK image


app.war



EXAMPLE: SMALL LEAN RUNTIMES

DOCKER BUILDCUSTOM GO BUILD

CustomGo S2IBuilder Image

ScratchImage

Build the app binary and deploy on small scratch images


app


CONTINUOUS INTEGRATION (CI)CONTINUOUS DELIVERY (CD)


CI/CD WITH BUILD AND DEPLOYMENTS

129

BUILDS● Webhook triggers: build the app image whenever the code changes● Image trigger: build the app image whenever the base language or app runtime changes● Build hooks: test the app image before pushing it to an image registry

DEPLOYMENTS● Deployment triggers: redeploy app containers whenever configuration changes or the

image changes in the OpenShift integrated registry or upstream registries


CONTINUOUS DELIVERY WITH CONTAINERS

sourcerepository

CI/CDengine

dev container

physical

virtual

private cloud

public cloud


OPENSHIFT LOVES CI/CD

JENKINS-AS-A SERVICEON OPENSHIFT

HYBRID JENKINS INFRA WITH OPENSHIFT

EXISTING CI/CD DEPLOY TO OPENSHIFT


JENKINS-AS-A-SERVICE ON OPENSHIFT

● Certified Jenkins images with pre-configured plugins○ Provided out-of-the-box○ Follows Jenkins 1.x and 2.x LTS versions

● Jenkins S2I Builder for customizing the image○ Install Plugins○ Configure Jenkins○ Configure Build Jobs

● OpenShift plugins to integrate authentication with OpenShift and also CI/CD pipelines

● Dynamically deploys Jenkins slave containers

PluginsJobs

Configuration

Jenkins(S2I)

CustomJenkinsImage

JenkinsImage


● Scale existing Jenkins infrastructure by dynamically provisioning Jenkins slaves on OpenShift

● Use Kubernetes plug-in on existing Jenkin servers

133

HYBRID JENKINS INFRA WITH OPENSHIFT

OPENSHIFT

APP APPrun jobJENKINSSLAVE

Run Job

JENKINSSLAVE

Run Job

build

JENKINSMASTER

deploy


● Existing CI/CD infrastructure outside OpenShift performs operations against OpenShift○ OpenShift Pipeline Jenkins Plugin for Jenkins○ OpenShift CLI for integrating other CI Engines with OpenShift

● Without disrupting existing processes, can be combined with previous alternative

134

EXISTING CI/CD DEPLOY TO OPENSHIFT

OPENSHIFT

APPEXISTING

CI/CD INFRA

Jenkins, Bamboo, TeamCity, etc

APPbuild

deploy

S2IBuildrun job


OPENSHIFT PIPELINES

● OpenShift Pipelines allow defining a CI/CD workflow via a Jenkins pipeline which can be started, monitored, and managed similar to other builds

● Dynamic provisioning of Jenkins slaves

● Auto-provisioning of Jenkins server

● OpenShift Pipeline strategies○ Embedded Jenkinsfile○ Jenkinsfile from a Git repository

apiVersion: v1kind: BuildConfigmetadata:

name: app-pipelinespec:

strategy:type: JenkinsPipelinejenkinsPipelineStrategy:

jenkinsfile: |-node('maven') {

stage('build app') {git url: 'https://git/app.git'sh "mvn package"

}stage('build image') {

sh "oc start-build app --from-file=target/app.jar

}stage('deploy') {

openshiftDeploy deploymentConfig: 'app'}

}

Provision a Jenkins slave for running Maven


OpenShift Pipelines inWeb Console


APPLICATIONIMAGE

137

CONTINUOUS DELIVERY PIPELINE

DEV TEAM GIT SERVER ARTIFACT REPOSITORY

JENKINSIMAGE BUILD

● S2I build from source code● S2I build from app binary● Existing docker container image

build process


CONTINUOUS DELIVERY PIPELINE

OPENSHIFTIMAGE

REGISTRY

OPENSHIFTCLUSTER

138

DEVELOPER GIT SERVER ARTIFACT REPOSITORY

OPENSHIFT CI/CD PIPELINE (JENKINS)

IMAGE BUILD & DEPLOY

OPENSHIFTIMAGEREGISTRY

OPENSHIFTCLUSTER

NON-PROD PRODDEV


OPENSHIFTIMAGE

REGISTRY

OPENSHIFTCLUSTER

139

CONTINUOUS DELIVERY PIPELINEDEVELOPER GIT SERVER ARTIFACT REPOSITORY




OPENSHIFTCLUSTER

PROMOTETO TEST

NON-PROD PRODDEV TEST


OPENSHIFTIMAGE

REGISTRY

OPENSHIFTCLUSTER

140





OPENSHIFTCLUSTER

PROMOTETO TEST

PROMOTETO UAT

NON-PROD PRODDEV TEST UAT


ServiceNowJIRA Service DeskZendeksBMC Remedy

OPENSHIFTIMAGE

REGISTRY

OPENSHIFTCLUSTER

141





OPENSHIFTCLUSTER

GOLIVE?

PROMOTETO TEST

PROMOTETO UAT

RELEASE MANAGER

NON-PROD PROD

☒

DEV TEST UAT


OPENSHIFTIMAGE

REGISTRY

OPENSHIFTCLUSTER

142

CONTINUOUS DELIVERY PIPELINEGIT SERVER ARTIFACT REPOSITORY




OPENSHIFTCLUSTER

GOLIVE?

PROMOTETO TEST

PROMOTETO UAT

PROMOTETO PROD

RELEASE MANAGER


☒

DEVELOPER


BUT…SOME TEAMS ALREADY HAVE

AUTOMATED DELIVERY PIPELINES


WHAT IF THERE ARE EXISTING DELIVERY PROCESSES?

BUILD APP BINARY

RUN TESTS

PROMOTE APPBINARY

BUILD CONTAINER IMAGE

RUN TESTS

PROMOTE CONTAINER

IMAGESOURCEVERSION CONTROL

ENTERPRISEBINARY REPO

ENTERPRISEIMAGE REGISTRY


WHAT IF THERE ARE EXISTING DELIVERY PROCESSES?

BUILD APP BINARY

RUN TESTS

PROMOTE APPBINARY

BUILD CONTAINER IMAGE

RUN TESTS

PROMOTE CONTAINER

IMAGESOURCEVERSION CONTROL

ENTERPRISEBINARY REPO

AWS ECR

ENTERPRISEIMAGE REGISTRY


ENRICHING EXISTING DELIVERY PROCESSES WITH OPENSHIFT

OPENSHIFTCLUSTER

EXISTINGDELIVERYPROCESS

DEPLOY DEPLOY DEPLOY


ENRICHING EXISTING DELIVERY PROCESSES WITH OPENSHIFT

OPENSHIFTIMAGE

REGISTRY

OPENSHIFTCLUSTER


OPENSHIFTCLUSTER


EXISTINGDELIVERYPROCESS

ENTERPRISEIMAGEREGISTRY


HYBRID APPLICATION AUTOMATIONWITH OPENSHIFT AND ANSIBLE


CONTINUOUSDELIVERYPIPELINE

HYBRID APPLICATION AUTOMATIONWITH OPENSHIFT AND ANSIBLE

VIRTUAL MACHINE

VIRTUAL MACHINE

AWS Azure Google CloudOpenStackVMware RHEVHyper V

DEV PROD - REGION A PROD - REGION B

DEVELOPER WORKFLOW


LOCAL DEVELOPMENT WORKFLOW

DevelopLocal

Deploy Verify Git Push PipelineBootstrap


BOOTSTRAP

● Pick your programming language and application runtime of choice● Create the project skeleton from scratch or use a generator such as

○ Maven archetypes○ Quickstarts and Templates○ OpenShift Generator○ Spring Initializr

152


DevelopLocal



DEVELOP

● Pick your framework of choice such as Java EE, Spring, Ruby on Rails, Django, Express, ...● Develop your application code using your editor or IDE of choice● Build and test your application code locally using your build tools● Create or generate OpenShift templates or Kubernetes objects


DevelopLocal



LOCAL DEPLOY

● Deploy your code on a local OpenShift cluster○ Red Hat Container Development Kit (CDK), minishift and oc cluster

● Red Hat CDK provides a standard RHEL-based development environment● Use binary deploy, maven or CLI rsync to push code or app binary directly into

containers


DevelopLocal



VERIFY

● Verify your code is working as expected● Run any type of tests that are required with or without other components (database, etc)● Based on the test results, change code, deploy, verify and repeat


DevelopLocal



GIT PUSH

● Push the code and configuration to the Git repository● If using Fork & Pull Request workflow, create a Pull Request● If using code review workflow, participate in code review discussions


DevelopLocal



PIPELINE

● Pushing code to the Git repository triggers one or multiple deployment pipelines● Design your pipelines based on your development workflow e.g. test the pull request● Failure in the pipeline? Go back to the code and start again

157


DevelopLocal


APPLICATION SERVICES


A PLATFORM THAT GROWS WITH YOUR BUSINESS

Data Virtualization

Real Time Decision

IntelligentProcess

Integration Messaging Data Grid

Java EE Application

WebApplication

SingleSign-On MobileAPI

Management

Micro services


CrunchyData

GitLab

Iron.io

Couchbase

Sonatype

EnterpriseDB

NuoDB

Fujitsuand many more

160

...and virtually any docker

image out there!

TRUE POLYGLOT PLATFORM

PHPPythonJava NodeJS Perl Ruby.NETCore

ApacheHTTPServer

MySQL Redis

nginx TomcatVarnish

JBossEAP

JBossA-MQ

JBossFuse

JBossBRMS

JBossBPMS

JBossData Grid

JBossData Virt

RH Mobile

RH SSO3SCALE

API mgmt

JBossWeb

Server

SpringBoot

Wildfly Swarm

Vert.x

PostgreSQL MongoDB

Phusion Passenger

Third-partyLanguage Runtimes

Third-partyDatabases

Third-partyApp

Runtimes

Third-partyMiddleware

Third-partyMiddleware

LANGUAGES

DATABASES

WEB SERVERS

MIDDLEWARE


Modern, Cloud-Native Application Runtimes and an Opinionated Developer Experience

OPENSHIFT

SUPPORTED RUNTIMES

Eclipse Vert.x WildFly Swarm Node.js

LAUNCH

Spring Boot JBoss EAP

MICROSERVICES INFRASTRUCTURE:

ISTIO SERVICE MESH


WHAT DO YOU NEED FOR MICROSERVICES?

Visibility & Reporting

Resilience & Fault Tolerance

Routing & Traffic Control

Identity & Security

Policy Enforcement


WHAT YOU NEED FOR MICROSERVICES?




Identity & Security

Policy Enforcement

Infrastructure

Microservice

Service Discovery Load Balancing

Circuit Breaker Traffic Control

Monitoring Tracing

Business Logic

Netflix OSSConfig Server Security Policies

Service Registry Traffic Control

Monitoring Tracing

API Magenement Smart Routing


MICROSERVICES EVOLUTION

Platform

Microservice

Netflix OSS

Netflix OSS

Business Logic

Container Platform

Microservice

Business Logic


WHAT YOU NEED FOR MICROSERVICES?




Identity & Security

Policy Enforcement

Istio


Control Plane

167

WHAT IS ISTIO?a service mesh to connect, manage, and secure microservices

Pilot Mixer Auth

Data Plane

Pod

Envoy

App

Pod

Envoy

App

Pod

Envoy

App

Pod

Envoy

App

TECH PREVIEW OCP 3.10


Platform

Microservice

Service Discovery Load Balancing

Circuit Breaker Traffic Control

Monitoring Tracing

Business Logic

Netflix OSSConfig Server Security Policies


Monitoring Tracing


MicroserviceBusiness Logic

OpenShift + IstioConfig Server Load Balancing


Monitoring Tracing


Microservices App

Microservices App

NETFLIX OSS VS ISTIO

A SNEAK PEAK INTOOPENSHIFT 4


IMMUTABLE INFRASTRUCTURE WITH RED HAT COREOS

● Minimal Linux distribution

● Optimized for running containers

● Decreased attack surface

● Over-the-air automated updates

● Immutable foundation for OpenShift

● Bare-metal and cloud host configuration


AUTOMATED OPERATIONS

Infra provisioning

Embedded OS

Full-stack deployment

On-premises and cloud

Unified experience

Secure defaults

Network isolation

Signing and policies

Audit and logs

Multi-cluster aware

Monitoring and alerts

Zero downtime upgrades

Full-stack patch & upgrade

Vulnerability scanning

INSTALL HARDENDEPLOY OPERATE

AUTOMATED OPERATIONS

Fully automated day-1 and day-2 operations for Kubernetes


OPERATOR AND DEVELOPER CONSOLES


OPERATOR CONSOLE


INFRASTRUCTURE MONITORING

THANK YOU

plus.google.com/+RedHat

linkedin.com/company/red-hat

youtube.com/user/RedHatVideos

facebook.com/redhatinc

twitter.com/RedHatNews

OPENSHIFT CONTAINER PLATFORM - Technical Deep Dive

Documents

Transcript of OPENSHIFT CONTAINER PLATFORM - Technical Deep Dive