分布式数据库与存储

冯若航 2021年0801

树发新芽 mdashmdash 开箱即的开源PostgreSQL发版 Pigsty

About Me

Ruohang Feng (Vonng)httpsvonngcom

独开源贡献者Pigsty作者《Designing Data-Intensive Application》《PG Internal》译者 PostgreSQL中国社区开源技术委员会委员PostgreSQL专家全栈开发者曾任职于AlibabaAppleTanTan

rhvonngcom

bull httpdemopigstycc

About Me

Ruohang Feng (Vonng)httpsvonngcom

独开源贡献者Pigsty作者《Designing Data-Intensive Application》《PG Internal》译者 PostgreSQL中国社区开源技术委员会委员PostgreSQL专家全栈开发者曾任职于AlibabaAppleTanTan

rhvonngcom

bull httpdemopigstycc

FederatedGeoSpatial

TimeSeries

DistributiveOLAP

Streaming

FDW

世界上最先进的开源关系型数据库

性能优异功能强大

历史悠久稳定可靠

扩展丰富协议友善

代码清晰定制灵活

一招鲜吃遍天

成熟的应可能会到许许多多的数据组件缓存OLTPOLAP批处理数据仓库流处理消息队列搜索索引NoSQL档数据库地理

数据库空间数据库时序数据库图数据库

传统的架构选型可能会组合使多种组件典型如Redis + MySQL + GreenplumHadoop + KafkaFlink + ElasticSearch

套组合拳基本能应付多数需求

带来的新问题就是 异构系统集成

需编写量重复繁琐搬砖代码把数据从A组件搬运到B组件

专多的全栈数据库

ldquo开源的Oraclerdquo

唯能对Oracle构成直接威胁的开源数据库

去O扛旗者

原持Oracle 九成功能

开支散叶子孙满堂

健康的社区友善的协议丰富的扩展繁荣的生态开放扩展的数据库

老树祖宗级开源项目

现状 与 未来

PigstyPostgres in Graphic STYle

Ruohang Feng

v100

Battery-Included Open Source PostgreSQL Distribution

Outline

What Where Who Why HowMonitoring

Provisioning

High Available

Sandbox

Analysis

Open Source

Fellowship of PigstyLord of PostgreSQL

ᚆقഠဉᓟሾह

᮱ᗟᓕഴᥴොໜ

ፊഴᔮᕹᥴොໜ

PostgreSQLහഝପᤈᇇݎ

What is PigstyᓟܨአጱრPostgreSQLහഝପݎᤈᇇ۱ෛPG໐ғPostgreSQL 13314-beta2զၹᰁಘൊկҁ୩ێಘғଧහഝTimescaleDB 23ቘᑮᳵPostGIS 31ړୗCitus 10ق᮱ᓟܨአහഝ᮱ᗟଃݢᶌጱӾᳵկ Patroni 21 Pgbouncer115 Haproxy 22 Consul 110۱ኞԾᕆጱᬩᤈሾह Grafana 80 Prometheus 228 Nginx 120 Ansible

ᶎӫӱጱრPostgreSQLፊഴᥴොໜق

ӞᲫጱقᚆPostgreSQLဉᓟሾह

e Baery-Included Open Source PostgreSQL Disibu$on

ଃԧ20ӻᔜஞᦡᦇጱፊഴᶎਖ਼ፊഴຽ۸ԅग़قොጱړຉӨಅࢶᤒ᮷ଃӿጱᛯᬳളӥ᱂ӤܫཞݢளਧඳᵑӨᚆᳯᗝԧचԭPrometheus+AlertManagerӨGrafanaጱಸᦄᔮᕹӷᘏᒵපӬԅղᶼᗝଉአහഝପӨಸᦄᥢڞᓟܨአ

1000+ፊഴຽ ൈᬿԧහഝପ Ӿᳵկ ᅩ զᨮᤍጱොොᶎᶎᶮጱقੴᥦکੜጱܔӻහഝପҁᤒᔱڍහᒵ᮷Ӟᥦ෫

ຉPG CSV෭பጱଫአ pglogړአԭᤰचԭlokiӨpromtailጱਫ෭பතᵞොໜଚݢ

ᛔశጱṛݢአහഝପᵞᗭ Pigsty᮱ᗟጱහഝପᵞᗭݢզᛔଉᥠඳᵑӾ௩ᵞᗭ౮क़ᤒሿᒵհݝᥝՖձਫၚ ᵞᗭक़ጱݱᐿ๐ۓ੪ӧտྊกୗᯈᗝկᦜአ᭗ᬦ160+හൈᬿӨਧګහഝପӨचᏐᦡෞᬩᤈጱොොᶎᶎᘒ෫ᵱஞጱਫሿෆᔄKubernetesጱԯኞአḵ

ԧӞᔮڜᗝᒵۂ አԭ᮱ᗟಘᲀPostgreSQLහഝପᵞᗭӨਫ ᓕቘӱۓහഝପӨአአᓌܔොᛔଃጱCMDBಘӨਧګݢጱCLIӨGUI

Pigstyᰒय़ᥢཛྷኞԾሾहጱPostgreSQLᓕቘᘒᦡᦇԽᦡᦇԅݢਠෆᬩᤈԭ1໐1GBᡦӾԭכ೮ݎၥᦶᶼݎኞԾሾहጱӞᛘ

አݢզӨਫᴬኞԾሾहҁᴻᥢ໒क़ਠقӞᛘጱဉᓟሾहӾԟറᔱPostgreSQLᬰᤈݱᐿਫḵճၥᦶݎଚኞԾᕆፊഴᔮᕹӾقݐᶎӬਫጱහഝݍḇ

෫ᵱᘶᗑᦢᳯقෛCentOS 7ӤӞᤈե10ړᰦਠ౮ಅਞᤰᑕಅᕟկ૪ԅኞԾአঅٵҁӧᦇPigstyӥᕅ1GB

ᗝ๐ݎۓሿګᴻአԭṛݢአጱDCSक़හഝପӨचᏐᦡෞአຂᘠݳཛྷୗහഝପݒҁಘᖽӤӥᕚᛔᭇᯈ෫ᵱಋૡᖌಷचᏐᦡෞӨᬟᔮᕹ

አݢզአᗝጱPostgres amp Grafana amp EchartsݎԻහഝଫአӨහഝݢᥤ۸ߝளԾߝڊଚզຽٵጱොୗᏟਧጱሾहӾړՁᄍᐏӨԻ

ᓌܔฃአጱრPostgreSQLᓕഴᥴොໜ

አᛔ૩ጱᒟᦕ(Macbook)ӤݢӞᲫۑᚆਠෆጱဉᓟҁኧvagrantӨvirtualboxḝ᭗ᬦᓌܔጱᯈᗝᛔ૩ጱᡦԯECSVPSӤਠ౮᮱ᗟ

Pigsty将集群部署扩容缩容主从复制故障切换流量代理连接池服务发现访问控制监控系统告警系统志采集

等产级成熟解决案封装为发版次性解决产环境个使PostgreSQL数据库的各类问题真正做到 开箱即

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

开箱即

v7新UI

Software ToolkitProvisioningInfrastructure

PlatformGUICLIAPIMonitoringLoggingAlertingReportingService DiscoveryHAScalability

LBPoolingACLAuditSecuritybackup

KernelPostgreSQL

Timescale

PostGIS Citus

Pipeline

发版

ӻՈአ

կܯ

փᕹմӱ

ᘶᗑմӱ

Who wants PigstyӾੜᘶᗑմӱ ҁ෭ၚӡᰁᕆQPSጯӡᰁᕆහഝପᵞᗭහጯᰁᕆ

ᔄփᕹմӱݱ۸ᵱጱמ

կᵞ౮կܯ

DBA OPS DEV හഝᎸݎහഝړຉᆽঅᘏኞ

ᵱᥝՈғአکPostgreSQLܩဌፘىՈᵱොໜғᵱᥝ౮ᆧਠෆጱᥴොໜ$ yum instsall ampamp systemctl start ๅᶌᨏ᭄౮ғአጱ౮ېग़ጱԪ

ᰁݢᥡғਫහጯӻ436හጯӡ᭄භഠғᵞᗭਫݒᔺභഠԻӨளᖽᵱ᭄ݢአݢአṛᥝඳᵑᛔశᛔڔഘ᭄ದ๕᭗ᬦහഝḝṛလቘଘ؇ᆽრ

ӧమರᚸᓌܔঅአளӤಋฃᖌಷ᭄ۑᚆቘᑮᳵଧ-621ᬦᑕᇔ۸ᥤׯࢶഭᔱᒵ᭄ݢᶌහഝӧӶӧᲙӧԤղᅒؿବᛔԆݢഴӧమӤԯమᥝᔄԯጱᓕቘḵ

ӤಋளғӞᲫᯈॺἶقᓟܨአᳪདғ໐ጱବᯈᡦԞݢզ᪒ӞᛘғݎၥᦶሾहӨኞԾሾहכ೮Ӟᛘᥤ۸ݢሾहғ3WKRQ64ݳຉጱෆړᚆғ๕හഝቘӨۑق

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

成熟的应可能会到许许多多的数据组件缓存OLTPOLAP批处理数据仓库流处理消息队列搜索索引NoSQL档数据库地理

数据库空间数据库时序数据库图数据库

传统的架构选型可能会组合使多种组件典型如Redis + MySQL + GreenplumHadoop + KafkaFlink + ElasticSearch

套组合拳基本能应付多数需求

带来的新问题就是 异构系统集成

需编写量重复繁琐搬砖代码把数据从A组件搬运到B组件

专多的全栈数据库

ldquo开源的Oraclerdquo

唯能对Oracle构成直接威胁的开源数据库

去O扛旗者

原持Oracle 九成功能

开支散叶子孙满堂

健康的社区友善的协议丰富的扩展繁荣的生态开放扩展的数据库

老树祖宗级开源项目

现状 与 未来

PigstyPostgres in Graphic STYle

Ruohang Feng

v100

Battery-Included Open Source PostgreSQL Distribution

Outline

What Where Who Why HowMonitoring

Provisioning

High Available

Sandbox

Analysis

Open Source

Fellowship of PigstyLord of PostgreSQL

ᚆقഠဉᓟሾह

᮱ᗟᓕഴᥴොໜ

ፊഴᔮᕹᥴොໜ

PostgreSQLහഝପᤈᇇݎ

What is PigstyᓟܨአጱრPostgreSQLහഝପݎᤈᇇ۱ෛPG໐ғPostgreSQL 13314-beta2զၹᰁಘൊկҁ୩ێಘғଧහഝTimescaleDB 23ቘᑮᳵPostGIS 31ړୗCitus 10ق᮱ᓟܨአහഝ᮱ᗟଃݢᶌጱӾᳵկ Patroni 21 Pgbouncer115 Haproxy 22 Consul 110۱ኞԾᕆጱᬩᤈሾह Grafana 80 Prometheus 228 Nginx 120 Ansible

ᶎӫӱጱრPostgreSQLፊഴᥴොໜق

ӞᲫጱقᚆPostgreSQLဉᓟሾह

e Baery-Included Open Source PostgreSQL Disibu$on

ଃԧ20ӻᔜஞᦡᦇጱፊഴᶎਖ਼ፊഴຽ۸ԅग़قොጱړຉӨಅࢶᤒ᮷ଃӿጱᛯᬳളӥ᱂ӤܫཞݢளਧඳᵑӨᚆᳯᗝԧचԭPrometheus+AlertManagerӨGrafanaጱಸᦄᔮᕹӷᘏᒵපӬԅղᶼᗝଉአහഝପӨಸᦄᥢڞᓟܨአ

1000+ፊഴຽ ൈᬿԧහഝପ Ӿᳵկ ᅩ զᨮᤍጱොොᶎᶎᶮጱقੴᥦکੜጱܔӻහഝପҁᤒᔱڍහᒵ᮷Ӟᥦ෫

ຉPG CSV෭பጱଫአ pglogړአԭᤰचԭlokiӨpromtailጱਫ෭பතᵞොໜଚݢ

ᛔశጱṛݢአහഝପᵞᗭ Pigsty᮱ᗟጱහഝପᵞᗭݢզᛔଉᥠඳᵑӾ௩ᵞᗭ౮क़ᤒሿᒵհݝᥝՖձਫၚ ᵞᗭक़ጱݱᐿ๐ۓ੪ӧտྊกୗᯈᗝկᦜአ᭗ᬦ160+හൈᬿӨਧګහഝପӨचᏐᦡෞᬩᤈጱොොᶎᶎᘒ෫ᵱஞጱਫሿෆᔄKubernetesጱԯኞአḵ

ԧӞᔮڜᗝᒵۂ አԭ᮱ᗟಘᲀPostgreSQLහഝପᵞᗭӨਫ ᓕቘӱۓහഝପӨአአᓌܔොᛔଃጱCMDBಘӨਧګݢጱCLIӨGUI

Pigstyᰒय़ᥢཛྷኞԾሾहጱPostgreSQLᓕቘᘒᦡᦇԽᦡᦇԅݢਠෆᬩᤈԭ1໐1GBᡦӾԭכ೮ݎၥᦶᶼݎኞԾሾहጱӞᛘ

አݢզӨਫᴬኞԾሾहҁᴻᥢ໒क़ਠقӞᛘጱဉᓟሾहӾԟറᔱPostgreSQLᬰᤈݱᐿਫḵճၥᦶݎଚኞԾᕆፊഴᔮᕹӾقݐᶎӬਫጱහഝݍḇ

෫ᵱᘶᗑᦢᳯقෛCentOS 7ӤӞᤈե10ړᰦਠ౮ಅਞᤰᑕಅᕟկ૪ԅኞԾአঅٵҁӧᦇPigstyӥᕅ1GB

ᗝ๐ݎۓሿګᴻአԭṛݢአጱDCSक़හഝପӨचᏐᦡෞአຂᘠݳཛྷୗහഝପݒҁಘᖽӤӥᕚᛔᭇᯈ෫ᵱಋૡᖌಷचᏐᦡෞӨᬟᔮᕹ

አݢզአᗝጱPostgres amp Grafana amp EchartsݎԻහഝଫአӨහഝݢᥤ۸ߝளԾߝڊଚզຽٵጱොୗᏟਧጱሾहӾړՁᄍᐏӨԻ

ᓌܔฃአጱრPostgreSQLᓕഴᥴොໜ

አᛔ૩ጱᒟᦕ(Macbook)ӤݢӞᲫۑᚆਠෆጱဉᓟҁኧvagrantӨvirtualboxḝ᭗ᬦᓌܔጱᯈᗝᛔ૩ጱᡦԯECSVPSӤਠ౮᮱ᗟ

Pigsty将集群部署扩容缩容主从复制故障切换流量代理连接池服务发现访问控制监控系统告警系统志采集

等产级成熟解决案封装为发版次性解决产环境个使PostgreSQL数据库的各类问题真正做到 开箱即

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

开箱即

v7新UI

Software ToolkitProvisioningInfrastructure

PlatformGUICLIAPIMonitoringLoggingAlertingReportingService DiscoveryHAScalability

LBPoolingACLAuditSecuritybackup

KernelPostgreSQL

Timescale

PostGIS Citus

Pipeline

发版

ӻՈአ

կܯ

փᕹմӱ

ᘶᗑմӱ

Who wants PigstyӾੜᘶᗑմӱ ҁ෭ၚӡᰁᕆQPSጯӡᰁᕆහഝପᵞᗭහጯᰁᕆ

ᔄփᕹմӱݱ۸ᵱጱמ

կᵞ౮կܯ

DBA OPS DEV හഝᎸݎහഝړຉᆽঅᘏኞ

ᵱᥝՈғአکPostgreSQLܩဌፘىՈᵱොໜғᵱᥝ౮ᆧਠෆጱᥴොໜ$ yum instsall ampamp systemctl start ๅᶌᨏ᭄౮ғአጱ౮ېग़ጱԪ

ᰁݢᥡғਫහጯӻ436හጯӡ᭄භഠғᵞᗭਫݒᔺභഠԻӨளᖽᵱ᭄ݢአݢአṛᥝඳᵑᛔశᛔڔഘ᭄ದ๕᭗ᬦහഝḝṛလቘଘ؇ᆽრ

ӧమರᚸᓌܔঅአளӤಋฃᖌಷ᭄ۑᚆቘᑮᳵଧ-621ᬦᑕᇔ۸ᥤׯࢶഭᔱᒵ᭄ݢᶌහഝӧӶӧᲙӧԤղᅒؿବᛔԆݢഴӧమӤԯమᥝᔄԯጱᓕቘḵ

ӤಋளғӞᲫᯈॺἶقᓟܨአᳪདғ໐ጱବᯈᡦԞݢզ᪒ӞᛘғݎၥᦶሾहӨኞԾሾहכ೮Ӟᛘᥤ۸ݢሾहғ3WKRQ64ݳຉጱෆړᚆғ๕හഝቘӨۑق

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

ldquo开源的Oraclerdquo

唯能对Oracle构成直接威胁的开源数据库

去O扛旗者

原持Oracle 九成功能

开支散叶子孙满堂

健康的社区友善的协议丰富的扩展繁荣的生态开放扩展的数据库

老树祖宗级开源项目

现状 与 未来

PigstyPostgres in Graphic STYle

Ruohang Feng

v100

Battery-Included Open Source PostgreSQL Distribution

Outline

What Where Who Why HowMonitoring

Provisioning

High Available

Sandbox

Analysis

Open Source

Fellowship of PigstyLord of PostgreSQL

ᚆقഠဉᓟሾह

᮱ᗟᓕഴᥴොໜ

ፊഴᔮᕹᥴොໜ

PostgreSQLහഝପᤈᇇݎ

What is PigstyᓟܨአጱრPostgreSQLහഝପݎᤈᇇ۱ෛPG໐ғPostgreSQL 13314-beta2զၹᰁಘൊկҁ୩ێಘғଧහഝTimescaleDB 23ቘᑮᳵPostGIS 31ړୗCitus 10ق᮱ᓟܨአහഝ᮱ᗟଃݢᶌጱӾᳵկ Patroni 21 Pgbouncer115 Haproxy 22 Consul 110۱ኞԾᕆጱᬩᤈሾह Grafana 80 Prometheus 228 Nginx 120 Ansible

ᶎӫӱጱრPostgreSQLፊഴᥴොໜق

ӞᲫጱقᚆPostgreSQLဉᓟሾह

e Baery-Included Open Source PostgreSQL Disibu$on

ଃԧ20ӻᔜஞᦡᦇጱፊഴᶎਖ਼ፊഴຽ۸ԅग़قොጱړຉӨಅࢶᤒ᮷ଃӿጱᛯᬳളӥ᱂ӤܫཞݢளਧඳᵑӨᚆᳯᗝԧचԭPrometheus+AlertManagerӨGrafanaጱಸᦄᔮᕹӷᘏᒵපӬԅղᶼᗝଉአහഝପӨಸᦄᥢڞᓟܨአ

1000+ፊഴຽ ൈᬿԧහഝପ Ӿᳵկ ᅩ զᨮᤍጱොොᶎᶎᶮጱقੴᥦکੜጱܔӻහഝପҁᤒᔱڍහᒵ᮷Ӟᥦ෫

ຉPG CSV෭பጱଫአ pglogړአԭᤰचԭlokiӨpromtailጱਫ෭பතᵞොໜଚݢ

ᛔశጱṛݢአහഝପᵞᗭ Pigsty᮱ᗟጱහഝପᵞᗭݢզᛔଉᥠඳᵑӾ௩ᵞᗭ౮क़ᤒሿᒵհݝᥝՖձਫၚ ᵞᗭक़ጱݱᐿ๐ۓ੪ӧտྊกୗᯈᗝկᦜአ᭗ᬦ160+හൈᬿӨਧګහഝପӨचᏐᦡෞᬩᤈጱොොᶎᶎᘒ෫ᵱஞጱਫሿෆᔄKubernetesጱԯኞአḵ

ԧӞᔮڜᗝᒵۂ አԭ᮱ᗟಘᲀPostgreSQLහഝପᵞᗭӨਫ ᓕቘӱۓහഝପӨአአᓌܔොᛔଃጱCMDBಘӨਧګݢጱCLIӨGUI

Pigstyᰒय़ᥢཛྷኞԾሾहጱPostgreSQLᓕቘᘒᦡᦇԽᦡᦇԅݢਠෆᬩᤈԭ1໐1GBᡦӾԭכ೮ݎၥᦶᶼݎኞԾሾहጱӞᛘ

አݢզӨਫᴬኞԾሾहҁᴻᥢ໒क़ਠقӞᛘጱဉᓟሾहӾԟറᔱPostgreSQLᬰᤈݱᐿਫḵճၥᦶݎଚኞԾᕆፊഴᔮᕹӾقݐᶎӬਫጱහഝݍḇ

෫ᵱᘶᗑᦢᳯقෛCentOS 7ӤӞᤈե10ړᰦਠ౮ಅਞᤰᑕಅᕟկ૪ԅኞԾአঅٵҁӧᦇPigstyӥᕅ1GB

ᗝ๐ݎۓሿګᴻአԭṛݢአጱDCSक़හഝପӨचᏐᦡෞአຂᘠݳཛྷୗහഝପݒҁಘᖽӤӥᕚᛔᭇᯈ෫ᵱಋૡᖌಷचᏐᦡෞӨᬟᔮᕹ

አݢզአᗝጱPostgres amp Grafana amp EchartsݎԻහഝଫአӨහഝݢᥤ۸ߝளԾߝڊଚզຽٵጱොୗᏟਧጱሾहӾړՁᄍᐏӨԻ

ᓌܔฃአጱრPostgreSQLᓕഴᥴොໜ

አᛔ૩ጱᒟᦕ(Macbook)ӤݢӞᲫۑᚆਠෆጱဉᓟҁኧvagrantӨvirtualboxḝ᭗ᬦᓌܔጱᯈᗝᛔ૩ጱᡦԯECSVPSӤਠ౮᮱ᗟ

Pigsty将集群部署扩容缩容主从复制故障切换流量代理连接池服务发现访问控制监控系统告警系统志采集

等产级成熟解决案封装为发版次性解决产环境个使PostgreSQL数据库的各类问题真正做到 开箱即

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

开箱即

v7新UI

Software ToolkitProvisioningInfrastructure

PlatformGUICLIAPIMonitoringLoggingAlertingReportingService DiscoveryHAScalability

LBPoolingACLAuditSecuritybackup

KernelPostgreSQL

Timescale

PostGIS Citus

Pipeline

发版

ӻՈአ

կܯ

փᕹմӱ

ᘶᗑմӱ

Who wants PigstyӾੜᘶᗑմӱ ҁ෭ၚӡᰁᕆQPSጯӡᰁᕆහഝପᵞᗭහጯᰁᕆ

ᔄփᕹմӱݱ۸ᵱጱמ

կᵞ౮կܯ

DBA OPS DEV හഝᎸݎහഝړຉᆽঅᘏኞ

ᵱᥝՈғአکPostgreSQLܩဌፘىՈᵱොໜғᵱᥝ౮ᆧਠෆጱᥴොໜ$ yum instsall ampamp systemctl start ๅᶌᨏ᭄౮ғአጱ౮ېग़ጱԪ

ᰁݢᥡғਫහጯӻ436හጯӡ᭄භഠғᵞᗭਫݒᔺභഠԻӨளᖽᵱ᭄ݢአݢአṛᥝඳᵑᛔశᛔڔഘ᭄ದ๕᭗ᬦහഝḝṛလቘଘ؇ᆽრ

ӧమರᚸᓌܔঅአளӤಋฃᖌಷ᭄ۑᚆቘᑮᳵଧ-621ᬦᑕᇔ۸ᥤׯࢶഭᔱᒵ᭄ݢᶌහഝӧӶӧᲙӧԤղᅒؿବᛔԆݢഴӧమӤԯమᥝᔄԯጱᓕቘḵ

ӤಋளғӞᲫᯈॺἶقᓟܨአᳪདғ໐ጱବᯈᡦԞݢզ᪒ӞᛘғݎၥᦶሾहӨኞԾሾहכ೮Ӟᛘᥤ۸ݢሾहғ3WKRQ64ݳຉጱෆړᚆғ๕හഝቘӨۑق

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

开支散叶子孙满堂

健康的社区友善的协议丰富的扩展繁荣的生态开放扩展的数据库

老树祖宗级开源项目

现状 与 未来

PigstyPostgres in Graphic STYle

Ruohang Feng

v100

Battery-Included Open Source PostgreSQL Distribution

Outline

What Where Who Why HowMonitoring

Provisioning

High Available

Sandbox

Analysis

Open Source

Fellowship of PigstyLord of PostgreSQL

ᚆقഠဉᓟሾह

᮱ᗟᓕഴᥴොໜ

ፊഴᔮᕹᥴොໜ

PostgreSQLහഝପᤈᇇݎ

What is PigstyᓟܨአጱრPostgreSQLහഝପݎᤈᇇ۱ෛPG໐ғPostgreSQL 13314-beta2զၹᰁಘൊկҁ୩ێಘғଧහഝTimescaleDB 23ቘᑮᳵPostGIS 31ړୗCitus 10ق᮱ᓟܨአහഝ᮱ᗟଃݢᶌጱӾᳵկ Patroni 21 Pgbouncer115 Haproxy 22 Consul 110۱ኞԾᕆጱᬩᤈሾह Grafana 80 Prometheus 228 Nginx 120 Ansible

ᶎӫӱጱრPostgreSQLፊഴᥴොໜق

ӞᲫጱقᚆPostgreSQLဉᓟሾह

e Baery-Included Open Source PostgreSQL Disibu$on

ଃԧ20ӻᔜஞᦡᦇጱፊഴᶎਖ਼ፊഴຽ۸ԅग़قොጱړຉӨಅࢶᤒ᮷ଃӿጱᛯᬳളӥ᱂ӤܫཞݢளਧඳᵑӨᚆᳯᗝԧचԭPrometheus+AlertManagerӨGrafanaጱಸᦄᔮᕹӷᘏᒵපӬԅղᶼᗝଉአහഝପӨಸᦄᥢڞᓟܨአ

1000+ፊഴຽ ൈᬿԧහഝପ Ӿᳵկ ᅩ զᨮᤍጱොොᶎᶎᶮጱقੴᥦکੜጱܔӻහഝପҁᤒᔱڍහᒵ᮷Ӟᥦ෫

ຉPG CSV෭பጱଫአ pglogړአԭᤰचԭlokiӨpromtailጱਫ෭பතᵞොໜଚݢ

ᛔశጱṛݢአහഝପᵞᗭ Pigsty᮱ᗟጱහഝପᵞᗭݢզᛔଉᥠඳᵑӾ௩ᵞᗭ౮क़ᤒሿᒵհݝᥝՖձਫၚ ᵞᗭक़ጱݱᐿ๐ۓ੪ӧտྊกୗᯈᗝկᦜአ᭗ᬦ160+හൈᬿӨਧګහഝପӨचᏐᦡෞᬩᤈጱොොᶎᶎᘒ෫ᵱஞጱਫሿෆᔄKubernetesጱԯኞአḵ

ԧӞᔮڜᗝᒵۂ አԭ᮱ᗟಘᲀPostgreSQLහഝପᵞᗭӨਫ ᓕቘӱۓහഝପӨአአᓌܔොᛔଃጱCMDBಘӨਧګݢጱCLIӨGUI

Pigstyᰒय़ᥢཛྷኞԾሾहጱPostgreSQLᓕቘᘒᦡᦇԽᦡᦇԅݢਠෆᬩᤈԭ1໐1GBᡦӾԭכ೮ݎၥᦶᶼݎኞԾሾहጱӞᛘ

አݢզӨਫᴬኞԾሾहҁᴻᥢ໒क़ਠقӞᛘጱဉᓟሾहӾԟറᔱPostgreSQLᬰᤈݱᐿਫḵճၥᦶݎଚኞԾᕆፊഴᔮᕹӾقݐᶎӬਫጱහഝݍḇ

෫ᵱᘶᗑᦢᳯقෛCentOS 7ӤӞᤈե10ړᰦਠ౮ಅਞᤰᑕಅᕟկ૪ԅኞԾአঅٵҁӧᦇPigstyӥᕅ1GB

ᗝ๐ݎۓሿګᴻአԭṛݢአጱDCSक़හഝପӨचᏐᦡෞአຂᘠݳཛྷୗහഝପݒҁಘᖽӤӥᕚᛔᭇᯈ෫ᵱಋૡᖌಷचᏐᦡෞӨᬟᔮᕹ

አݢզአᗝጱPostgres amp Grafana amp EchartsݎԻහഝଫአӨහഝݢᥤ۸ߝளԾߝڊଚզຽٵጱොୗᏟਧጱሾहӾړՁᄍᐏӨԻ

ᓌܔฃአጱრPostgreSQLᓕഴᥴොໜ

አᛔ૩ጱᒟᦕ(Macbook)ӤݢӞᲫۑᚆਠෆጱဉᓟҁኧvagrantӨvirtualboxḝ᭗ᬦᓌܔጱᯈᗝᛔ૩ጱᡦԯECSVPSӤਠ౮᮱ᗟ

Pigsty将集群部署扩容缩容主从复制故障切换流量代理连接池服务发现访问控制监控系统告警系统志采集

等产级成熟解决案封装为发版次性解决产环境个使PostgreSQL数据库的各类问题真正做到 开箱即

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

开箱即

v7新UI

Software ToolkitProvisioningInfrastructure

PlatformGUICLIAPIMonitoringLoggingAlertingReportingService DiscoveryHAScalability

LBPoolingACLAuditSecuritybackup

KernelPostgreSQL

Timescale

PostGIS Citus

Pipeline

发版

ӻՈአ

կܯ

փᕹմӱ

ᘶᗑմӱ

Who wants PigstyӾੜᘶᗑմӱ ҁ෭ၚӡᰁᕆQPSጯӡᰁᕆහഝପᵞᗭහጯᰁᕆ

ᔄփᕹմӱݱ۸ᵱጱמ

կᵞ౮կܯ

DBA OPS DEV හഝᎸݎහഝړຉᆽঅᘏኞ

ᵱᥝՈғአکPostgreSQLܩဌፘىՈᵱොໜғᵱᥝ౮ᆧਠෆጱᥴොໜ$ yum instsall ampamp systemctl start ๅᶌᨏ᭄౮ғአጱ౮ېग़ጱԪ

ᰁݢᥡғਫහጯӻ436හጯӡ᭄භഠғᵞᗭਫݒᔺභഠԻӨளᖽᵱ᭄ݢአݢአṛᥝඳᵑᛔశᛔڔഘ᭄ದ๕᭗ᬦහഝḝṛလቘଘ؇ᆽრ

ӧమರᚸᓌܔঅአளӤಋฃᖌಷ᭄ۑᚆቘᑮᳵଧ-621ᬦᑕᇔ۸ᥤׯࢶഭᔱᒵ᭄ݢᶌහഝӧӶӧᲙӧԤղᅒؿବᛔԆݢഴӧమӤԯమᥝᔄԯጱᓕቘḵ

ӤಋளғӞᲫᯈॺἶقᓟܨአᳪདғ໐ጱବᯈᡦԞݢզ᪒ӞᛘғݎၥᦶሾहӨኞԾሾहכ೮Ӟᛘᥤ۸ݢሾहғ3WKRQ64ݳຉጱෆړᚆғ๕හഝቘӨۑق

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

老树祖宗级开源项目

现状 与 未来

PigstyPostgres in Graphic STYle

Ruohang Feng

v100

Battery-Included Open Source PostgreSQL Distribution

Outline

What Where Who Why HowMonitoring

Provisioning

High Available

Sandbox

Analysis

Open Source

Fellowship of PigstyLord of PostgreSQL

ᚆقഠဉᓟሾह

᮱ᗟᓕഴᥴොໜ

ፊഴᔮᕹᥴොໜ

PostgreSQLහഝପᤈᇇݎ

What is PigstyᓟܨአጱრPostgreSQLහഝପݎᤈᇇ۱ෛPG໐ғPostgreSQL 13314-beta2զၹᰁಘൊկҁ୩ێಘғଧහഝTimescaleDB 23ቘᑮᳵPostGIS 31ړୗCitus 10ق᮱ᓟܨአහഝ᮱ᗟଃݢᶌጱӾᳵկ Patroni 21 Pgbouncer115 Haproxy 22 Consul 110۱ኞԾᕆጱᬩᤈሾह Grafana 80 Prometheus 228 Nginx 120 Ansible

ᶎӫӱጱრPostgreSQLፊഴᥴොໜق

ӞᲫጱقᚆPostgreSQLဉᓟሾह

e Baery-Included Open Source PostgreSQL Disibu$on

ଃԧ20ӻᔜஞᦡᦇጱፊഴᶎਖ਼ፊഴຽ۸ԅग़قොጱړຉӨಅࢶᤒ᮷ଃӿጱᛯᬳളӥ᱂ӤܫཞݢளਧඳᵑӨᚆᳯᗝԧचԭPrometheus+AlertManagerӨGrafanaጱಸᦄᔮᕹӷᘏᒵපӬԅղᶼᗝଉአහഝପӨಸᦄᥢڞᓟܨአ

1000+ፊഴຽ ൈᬿԧහഝପ Ӿᳵկ ᅩ զᨮᤍጱොොᶎᶎᶮጱقੴᥦکੜጱܔӻහഝପҁᤒᔱڍහᒵ᮷Ӟᥦ෫

ຉPG CSV෭பጱଫአ pglogړአԭᤰचԭlokiӨpromtailጱਫ෭பතᵞොໜଚݢ

ᛔశጱṛݢአහഝପᵞᗭ Pigsty᮱ᗟጱහഝପᵞᗭݢզᛔଉᥠඳᵑӾ௩ᵞᗭ౮क़ᤒሿᒵհݝᥝՖձਫၚ ᵞᗭक़ጱݱᐿ๐ۓ੪ӧտྊกୗᯈᗝկᦜአ᭗ᬦ160+හൈᬿӨਧګහഝପӨचᏐᦡෞᬩᤈጱොොᶎᶎᘒ෫ᵱஞጱਫሿෆᔄKubernetesጱԯኞአḵ

ԧӞᔮڜᗝᒵۂ አԭ᮱ᗟಘᲀPostgreSQLහഝପᵞᗭӨਫ ᓕቘӱۓහഝପӨአአᓌܔොᛔଃጱCMDBಘӨਧګݢጱCLIӨGUI

Pigstyᰒय़ᥢཛྷኞԾሾहጱPostgreSQLᓕቘᘒᦡᦇԽᦡᦇԅݢਠෆᬩᤈԭ1໐1GBᡦӾԭכ೮ݎၥᦶᶼݎኞԾሾहጱӞᛘ

አݢզӨਫᴬኞԾሾहҁᴻᥢ໒क़ਠقӞᛘጱဉᓟሾहӾԟറᔱPostgreSQLᬰᤈݱᐿਫḵճၥᦶݎଚኞԾᕆፊഴᔮᕹӾقݐᶎӬਫጱහഝݍḇ

෫ᵱᘶᗑᦢᳯقෛCentOS 7ӤӞᤈե10ړᰦਠ౮ಅਞᤰᑕಅᕟկ૪ԅኞԾአঅٵҁӧᦇPigstyӥᕅ1GB

ᗝ๐ݎۓሿګᴻአԭṛݢአጱDCSक़හഝପӨचᏐᦡෞአຂᘠݳཛྷୗහഝପݒҁಘᖽӤӥᕚᛔᭇᯈ෫ᵱಋૡᖌಷचᏐᦡෞӨᬟᔮᕹ

አݢզአᗝጱPostgres amp Grafana amp EchartsݎԻහഝଫአӨහഝݢᥤ۸ߝளԾߝڊଚզຽٵጱොୗᏟਧጱሾहӾړՁᄍᐏӨԻ

ᓌܔฃአጱრPostgreSQLᓕഴᥴොໜ

አᛔ૩ጱᒟᦕ(Macbook)ӤݢӞᲫۑᚆਠෆጱဉᓟҁኧvagrantӨvirtualboxḝ᭗ᬦᓌܔጱᯈᗝᛔ૩ጱᡦԯECSVPSӤਠ౮᮱ᗟ

Pigsty将集群部署扩容缩容主从复制故障切换流量代理连接池服务发现访问控制监控系统告警系统志采集

等产级成熟解决案封装为发版次性解决产环境个使PostgreSQL数据库的各类问题真正做到 开箱即

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

开箱即

v7新UI

Software ToolkitProvisioningInfrastructure

PlatformGUICLIAPIMonitoringLoggingAlertingReportingService DiscoveryHAScalability

LBPoolingACLAuditSecuritybackup

KernelPostgreSQL

Timescale

PostGIS Citus

Pipeline

发版

ӻՈአ

կܯ

փᕹմӱ

ᘶᗑմӱ

Who wants PigstyӾੜᘶᗑմӱ ҁ෭ၚӡᰁᕆQPSጯӡᰁᕆහഝପᵞᗭහጯᰁᕆ

ᔄփᕹմӱݱ۸ᵱጱמ

կᵞ౮կܯ

DBA OPS DEV හഝᎸݎහഝړຉᆽঅᘏኞ

ᵱᥝՈғአکPostgreSQLܩဌፘىՈᵱොໜғᵱᥝ౮ᆧਠෆጱᥴොໜ$ yum instsall ampamp systemctl start ๅᶌᨏ᭄౮ғአጱ౮ېग़ጱԪ

ᰁݢᥡғਫහጯӻ436හጯӡ᭄භഠғᵞᗭਫݒᔺභഠԻӨளᖽᵱ᭄ݢአݢአṛᥝඳᵑᛔశᛔڔഘ᭄ದ๕᭗ᬦහഝḝṛလቘଘ؇ᆽრ

ӧమರᚸᓌܔঅአளӤಋฃᖌಷ᭄ۑᚆቘᑮᳵଧ-621ᬦᑕᇔ۸ᥤׯࢶഭᔱᒵ᭄ݢᶌහഝӧӶӧᲙӧԤղᅒؿବᛔԆݢഴӧమӤԯమᥝᔄԯጱᓕቘḵ

ӤಋளғӞᲫᯈॺἶقᓟܨአᳪདғ໐ጱବᯈᡦԞݢզ᪒ӞᛘғݎၥᦶሾहӨኞԾሾहכ೮Ӟᛘᥤ۸ݢሾहғ3WKRQ64ݳຉጱෆړᚆғ๕හഝቘӨۑق

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

现状 与 未来

PigstyPostgres in Graphic STYle

Ruohang Feng

v100

Battery-Included Open Source PostgreSQL Distribution

Outline

What Where Who Why HowMonitoring

Provisioning

High Available

Sandbox

Analysis

Open Source

Fellowship of PigstyLord of PostgreSQL

ᚆقഠဉᓟሾह

᮱ᗟᓕഴᥴොໜ

ፊഴᔮᕹᥴොໜ

PostgreSQLහഝପᤈᇇݎ

What is PigstyᓟܨአጱრPostgreSQLහഝପݎᤈᇇ۱ෛPG໐ғPostgreSQL 13314-beta2զၹᰁಘൊկҁ୩ێಘғଧහഝTimescaleDB 23ቘᑮᳵPostGIS 31ړୗCitus 10ق᮱ᓟܨአහഝ᮱ᗟଃݢᶌጱӾᳵկ Patroni 21 Pgbouncer115 Haproxy 22 Consul 110۱ኞԾᕆጱᬩᤈሾह Grafana 80 Prometheus 228 Nginx 120 Ansible

ᶎӫӱጱრPostgreSQLፊഴᥴොໜق

ӞᲫጱقᚆPostgreSQLဉᓟሾह

e Baery-Included Open Source PostgreSQL Disibu$on

ଃԧ20ӻᔜஞᦡᦇጱፊഴᶎਖ਼ፊഴຽ۸ԅग़قොጱړຉӨಅࢶᤒ᮷ଃӿጱᛯᬳളӥ᱂ӤܫཞݢளਧඳᵑӨᚆᳯᗝԧचԭPrometheus+AlertManagerӨGrafanaጱಸᦄᔮᕹӷᘏᒵපӬԅղᶼᗝଉአහഝପӨಸᦄᥢڞᓟܨአ

1000+ፊഴຽ ൈᬿԧහഝପ Ӿᳵկ ᅩ զᨮᤍጱොොᶎᶎᶮጱقੴᥦکੜጱܔӻහഝପҁᤒᔱڍහᒵ᮷Ӟᥦ෫

ຉPG CSV෭பጱଫአ pglogړአԭᤰचԭlokiӨpromtailጱਫ෭பතᵞොໜଚݢ

ᛔశጱṛݢአහഝପᵞᗭ Pigsty᮱ᗟጱහഝପᵞᗭݢզᛔଉᥠඳᵑӾ௩ᵞᗭ౮क़ᤒሿᒵհݝᥝՖձਫၚ ᵞᗭक़ጱݱᐿ๐ۓ੪ӧտྊกୗᯈᗝկᦜአ᭗ᬦ160+හൈᬿӨਧګහഝପӨचᏐᦡෞᬩᤈጱොොᶎᶎᘒ෫ᵱஞጱਫሿෆᔄKubernetesጱԯኞአḵ

ԧӞᔮڜᗝᒵۂ አԭ᮱ᗟಘᲀPostgreSQLහഝପᵞᗭӨਫ ᓕቘӱۓහഝପӨአአᓌܔොᛔଃጱCMDBಘӨਧګݢጱCLIӨGUI

Pigstyᰒय़ᥢཛྷኞԾሾहጱPostgreSQLᓕቘᘒᦡᦇԽᦡᦇԅݢਠෆᬩᤈԭ1໐1GBᡦӾԭכ೮ݎၥᦶᶼݎኞԾሾहጱӞᛘ

አݢզӨਫᴬኞԾሾहҁᴻᥢ໒क़ਠقӞᛘጱဉᓟሾहӾԟറᔱPostgreSQLᬰᤈݱᐿਫḵճၥᦶݎଚኞԾᕆፊഴᔮᕹӾقݐᶎӬਫጱහഝݍḇ

෫ᵱᘶᗑᦢᳯقෛCentOS 7ӤӞᤈե10ړᰦਠ౮ಅਞᤰᑕಅᕟկ૪ԅኞԾአঅٵҁӧᦇPigstyӥᕅ1GB

ᗝ๐ݎۓሿګᴻአԭṛݢአጱDCSक़හഝପӨचᏐᦡෞአຂᘠݳཛྷୗහഝପݒҁಘᖽӤӥᕚᛔᭇᯈ෫ᵱಋૡᖌಷचᏐᦡෞӨᬟᔮᕹ

አݢզአᗝጱPostgres amp Grafana amp EchartsݎԻහഝଫአӨහഝݢᥤ۸ߝளԾߝڊଚզຽٵጱොୗᏟਧጱሾहӾړՁᄍᐏӨԻ

ᓌܔฃአጱრPostgreSQLᓕഴᥴොໜ

አᛔ૩ጱᒟᦕ(Macbook)ӤݢӞᲫۑᚆਠෆጱဉᓟҁኧvagrantӨvirtualboxḝ᭗ᬦᓌܔጱᯈᗝᛔ૩ጱᡦԯECSVPSӤਠ౮᮱ᗟ

Pigsty将集群部署扩容缩容主从复制故障切换流量代理连接池服务发现访问控制监控系统告警系统志采集

等产级成熟解决案封装为发版次性解决产环境个使PostgreSQL数据库的各类问题真正做到 开箱即

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

开箱即

v7新UI

Software ToolkitProvisioningInfrastructure

PlatformGUICLIAPIMonitoringLoggingAlertingReportingService DiscoveryHAScalability

LBPoolingACLAuditSecuritybackup

KernelPostgreSQL

Timescale

PostGIS Citus

Pipeline

发版

ӻՈአ

կܯ

փᕹմӱ

ᘶᗑմӱ

Who wants PigstyӾੜᘶᗑմӱ ҁ෭ၚӡᰁᕆQPSጯӡᰁᕆහഝପᵞᗭහጯᰁᕆ

ᔄփᕹմӱݱ۸ᵱጱמ

կᵞ౮կܯ

DBA OPS DEV හഝᎸݎහഝړຉᆽঅᘏኞ

ᵱᥝՈғአکPostgreSQLܩဌፘىՈᵱොໜғᵱᥝ౮ᆧਠෆጱᥴොໜ$ yum instsall ampamp systemctl start ๅᶌᨏ᭄౮ғአጱ౮ېग़ጱԪ

ᰁݢᥡғਫහጯӻ436හጯӡ᭄භഠғᵞᗭਫݒᔺභഠԻӨளᖽᵱ᭄ݢአݢአṛᥝඳᵑᛔశᛔڔഘ᭄ದ๕᭗ᬦහഝḝṛလቘଘ؇ᆽრ

ӧమರᚸᓌܔঅአளӤಋฃᖌಷ᭄ۑᚆቘᑮᳵଧ-621ᬦᑕᇔ۸ᥤׯࢶഭᔱᒵ᭄ݢᶌහഝӧӶӧᲙӧԤղᅒؿବᛔԆݢഴӧమӤԯమᥝᔄԯጱᓕቘḵ

ӤಋளғӞᲫᯈॺἶقᓟܨአᳪདғ໐ጱବᯈᡦԞݢզ᪒ӞᛘғݎၥᦶሾहӨኞԾሾहכ೮Ӟᛘᥤ۸ݢሾहғ3WKRQ64ݳຉጱෆړᚆғ๕හഝቘӨۑق

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

PigstyPostgres in Graphic STYle

Ruohang Feng

v100

Battery-Included Open Source PostgreSQL Distribution

Outline

What Where Who Why HowMonitoring

Provisioning

High Available

Sandbox

Analysis

Open Source

Fellowship of PigstyLord of PostgreSQL

ᚆقഠဉᓟሾह

᮱ᗟᓕഴᥴොໜ

ፊഴᔮᕹᥴොໜ

PostgreSQLහഝପᤈᇇݎ

What is PigstyᓟܨአጱრPostgreSQLහഝପݎᤈᇇ۱ෛPG໐ғPostgreSQL 13314-beta2զၹᰁಘൊկҁ୩ێಘғଧහഝTimescaleDB 23ቘᑮᳵPostGIS 31ړୗCitus 10ق᮱ᓟܨአහഝ᮱ᗟଃݢᶌጱӾᳵկ Patroni 21 Pgbouncer115 Haproxy 22 Consul 110۱ኞԾᕆጱᬩᤈሾह Grafana 80 Prometheus 228 Nginx 120 Ansible

ᶎӫӱጱრPostgreSQLፊഴᥴොໜق

ӞᲫጱقᚆPostgreSQLဉᓟሾह

e Baery-Included Open Source PostgreSQL Disibu$on

ଃԧ20ӻᔜஞᦡᦇጱፊഴᶎਖ਼ፊഴຽ۸ԅग़قොጱړຉӨಅࢶᤒ᮷ଃӿጱᛯᬳളӥ᱂ӤܫཞݢளਧඳᵑӨᚆᳯᗝԧचԭPrometheus+AlertManagerӨGrafanaጱಸᦄᔮᕹӷᘏᒵපӬԅղᶼᗝଉአහഝପӨಸᦄᥢڞᓟܨአ

1000+ፊഴຽ ൈᬿԧහഝପ Ӿᳵկ ᅩ զᨮᤍጱොොᶎᶎᶮጱقੴᥦکੜጱܔӻහഝପҁᤒᔱڍහᒵ᮷Ӟᥦ෫

ຉPG CSV෭பጱଫአ pglogړአԭᤰचԭlokiӨpromtailጱਫ෭பතᵞොໜଚݢ

ᛔశጱṛݢአහഝପᵞᗭ Pigsty᮱ᗟጱහഝପᵞᗭݢզᛔଉᥠඳᵑӾ௩ᵞᗭ౮क़ᤒሿᒵհݝᥝՖձਫၚ ᵞᗭक़ጱݱᐿ๐ۓ੪ӧտྊกୗᯈᗝկᦜአ᭗ᬦ160+හൈᬿӨਧګහഝପӨचᏐᦡෞᬩᤈጱොොᶎᶎᘒ෫ᵱஞጱਫሿෆᔄKubernetesጱԯኞአḵ

ԧӞᔮڜᗝᒵۂ አԭ᮱ᗟಘᲀPostgreSQLහഝପᵞᗭӨਫ ᓕቘӱۓහഝପӨአአᓌܔොᛔଃጱCMDBಘӨਧګݢጱCLIӨGUI

Pigstyᰒय़ᥢཛྷኞԾሾहጱPostgreSQLᓕቘᘒᦡᦇԽᦡᦇԅݢਠෆᬩᤈԭ1໐1GBᡦӾԭכ೮ݎၥᦶᶼݎኞԾሾहጱӞᛘ

አݢզӨਫᴬኞԾሾहҁᴻᥢ໒क़ਠقӞᛘጱဉᓟሾहӾԟറᔱPostgreSQLᬰᤈݱᐿਫḵճၥᦶݎଚኞԾᕆፊഴᔮᕹӾقݐᶎӬਫጱහഝݍḇ

෫ᵱᘶᗑᦢᳯقෛCentOS 7ӤӞᤈե10ړᰦਠ౮ಅਞᤰᑕಅᕟկ૪ԅኞԾአঅٵҁӧᦇPigstyӥᕅ1GB

ᗝ๐ݎۓሿګᴻአԭṛݢአጱDCSक़හഝପӨचᏐᦡෞአຂᘠݳཛྷୗහഝପݒҁಘᖽӤӥᕚᛔᭇᯈ෫ᵱಋૡᖌಷचᏐᦡෞӨᬟᔮᕹ

አݢզአᗝጱPostgres amp Grafana amp EchartsݎԻහഝଫአӨහഝݢᥤ۸ߝளԾߝڊଚզຽٵጱොୗᏟਧጱሾहӾړՁᄍᐏӨԻ

ᓌܔฃአጱრPostgreSQLᓕഴᥴොໜ

አᛔ૩ጱᒟᦕ(Macbook)ӤݢӞᲫۑᚆਠෆጱဉᓟҁኧvagrantӨvirtualboxḝ᭗ᬦᓌܔጱᯈᗝᛔ૩ጱᡦԯECSVPSӤਠ౮᮱ᗟ

Pigsty将集群部署扩容缩容主从复制故障切换流量代理连接池服务发现访问控制监控系统告警系统志采集

等产级成熟解决案封装为发版次性解决产环境个使PostgreSQL数据库的各类问题真正做到 开箱即

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

开箱即

v7新UI

Software ToolkitProvisioningInfrastructure

PlatformGUICLIAPIMonitoringLoggingAlertingReportingService DiscoveryHAScalability

LBPoolingACLAuditSecuritybackup

KernelPostgreSQL

Timescale

PostGIS Citus

Pipeline

发版

ӻՈአ

կܯ

փᕹմӱ

ᘶᗑմӱ

Who wants PigstyӾੜᘶᗑմӱ ҁ෭ၚӡᰁᕆQPSጯӡᰁᕆහഝପᵞᗭහጯᰁᕆ

ᔄփᕹմӱݱ۸ᵱጱמ

կᵞ౮կܯ

DBA OPS DEV හഝᎸݎහഝړຉᆽঅᘏኞ

ᵱᥝՈғአکPostgreSQLܩဌፘىՈᵱොໜғᵱᥝ౮ᆧਠෆጱᥴොໜ$ yum instsall ampamp systemctl start ๅᶌᨏ᭄౮ғአጱ౮ېग़ጱԪ

ᰁݢᥡғਫහጯӻ436හጯӡ᭄භഠғᵞᗭਫݒᔺභഠԻӨளᖽᵱ᭄ݢአݢአṛᥝඳᵑᛔశᛔڔഘ᭄ದ๕᭗ᬦහഝḝṛလቘଘ؇ᆽრ

ӧమರᚸᓌܔঅአளӤಋฃᖌಷ᭄ۑᚆቘᑮᳵଧ-621ᬦᑕᇔ۸ᥤׯࢶഭᔱᒵ᭄ݢᶌහഝӧӶӧᲙӧԤղᅒؿବᛔԆݢഴӧమӤԯమᥝᔄԯጱᓕቘḵ

ӤಋளғӞᲫᯈॺἶقᓟܨአᳪདғ໐ጱବᯈᡦԞݢզ᪒ӞᛘғݎၥᦶሾहӨኞԾሾहכ೮Ӟᛘᥤ۸ݢሾहғ3WKRQ64ݳຉጱෆړᚆғ๕හഝቘӨۑق

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

Outline

What Where Who Why HowMonitoring

Provisioning

High Available

Sandbox

Analysis

Open Source

Fellowship of PigstyLord of PostgreSQL

ᚆقഠဉᓟሾह

᮱ᗟᓕഴᥴොໜ

ፊഴᔮᕹᥴොໜ

PostgreSQLහഝପᤈᇇݎ

What is PigstyᓟܨአጱრPostgreSQLහഝପݎᤈᇇ۱ෛPG໐ғPostgreSQL 13314-beta2զၹᰁಘൊկҁ୩ێಘғଧහഝTimescaleDB 23ቘᑮᳵPostGIS 31ړୗCitus 10ق᮱ᓟܨአහഝ᮱ᗟଃݢᶌጱӾᳵկ Patroni 21 Pgbouncer115 Haproxy 22 Consul 110۱ኞԾᕆጱᬩᤈሾह Grafana 80 Prometheus 228 Nginx 120 Ansible

ᶎӫӱጱრPostgreSQLፊഴᥴොໜق

ӞᲫጱقᚆPostgreSQLဉᓟሾह

e Baery-Included Open Source PostgreSQL Disibu$on

ଃԧ20ӻᔜஞᦡᦇጱፊഴᶎਖ਼ፊഴຽ۸ԅग़قොጱړຉӨಅࢶᤒ᮷ଃӿጱᛯᬳളӥ᱂ӤܫཞݢளਧඳᵑӨᚆᳯᗝԧचԭPrometheus+AlertManagerӨGrafanaጱಸᦄᔮᕹӷᘏᒵපӬԅղᶼᗝଉአහഝପӨಸᦄᥢڞᓟܨአ

1000+ፊഴຽ ൈᬿԧහഝପ Ӿᳵկ ᅩ զᨮᤍጱොොᶎᶎᶮጱقੴᥦکੜጱܔӻහഝପҁᤒᔱڍහᒵ᮷Ӟᥦ෫

ຉPG CSV෭பጱଫአ pglogړአԭᤰचԭlokiӨpromtailጱਫ෭பතᵞොໜଚݢ

ᛔశጱṛݢአහഝପᵞᗭ Pigsty᮱ᗟጱහഝପᵞᗭݢզᛔଉᥠඳᵑӾ௩ᵞᗭ౮क़ᤒሿᒵհݝᥝՖձਫၚ ᵞᗭक़ጱݱᐿ๐ۓ੪ӧտྊกୗᯈᗝկᦜአ᭗ᬦ160+හൈᬿӨਧګහഝପӨचᏐᦡෞᬩᤈጱොොᶎᶎᘒ෫ᵱஞጱਫሿෆᔄKubernetesጱԯኞአḵ

ԧӞᔮڜᗝᒵۂ አԭ᮱ᗟಘᲀPostgreSQLහഝପᵞᗭӨਫ ᓕቘӱۓහഝପӨአአᓌܔොᛔଃጱCMDBಘӨਧګݢጱCLIӨGUI

Pigstyᰒय़ᥢཛྷኞԾሾहጱPostgreSQLᓕቘᘒᦡᦇԽᦡᦇԅݢਠෆᬩᤈԭ1໐1GBᡦӾԭכ೮ݎၥᦶᶼݎኞԾሾहጱӞᛘ

አݢզӨਫᴬኞԾሾहҁᴻᥢ໒क़ਠقӞᛘጱဉᓟሾहӾԟറᔱPostgreSQLᬰᤈݱᐿਫḵճၥᦶݎଚኞԾᕆፊഴᔮᕹӾقݐᶎӬਫጱහഝݍḇ

෫ᵱᘶᗑᦢᳯقෛCentOS 7ӤӞᤈե10ړᰦਠ౮ಅਞᤰᑕಅᕟկ૪ԅኞԾአঅٵҁӧᦇPigstyӥᕅ1GB

ᗝ๐ݎۓሿګᴻአԭṛݢአጱDCSक़හഝପӨचᏐᦡෞአຂᘠݳཛྷୗහഝପݒҁಘᖽӤӥᕚᛔᭇᯈ෫ᵱಋૡᖌಷचᏐᦡෞӨᬟᔮᕹ

አݢզአᗝጱPostgres amp Grafana amp EchartsݎԻහഝଫአӨහഝݢᥤ۸ߝளԾߝڊଚզຽٵጱොୗᏟਧጱሾहӾړՁᄍᐏӨԻ

ᓌܔฃአጱრPostgreSQLᓕഴᥴොໜ

አᛔ૩ጱᒟᦕ(Macbook)ӤݢӞᲫۑᚆਠෆጱဉᓟҁኧvagrantӨvirtualboxḝ᭗ᬦᓌܔጱᯈᗝᛔ૩ጱᡦԯECSVPSӤਠ౮᮱ᗟ

Pigsty将集群部署扩容缩容主从复制故障切换流量代理连接池服务发现访问控制监控系统告警系统志采集

等产级成熟解决案封装为发版次性解决产环境个使PostgreSQL数据库的各类问题真正做到 开箱即

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

开箱即

v7新UI

Software ToolkitProvisioningInfrastructure

PlatformGUICLIAPIMonitoringLoggingAlertingReportingService DiscoveryHAScalability

LBPoolingACLAuditSecuritybackup

KernelPostgreSQL

Timescale

PostGIS Citus

Pipeline

发版

ӻՈአ

կܯ

փᕹմӱ

ᘶᗑմӱ

Who wants PigstyӾੜᘶᗑմӱ ҁ෭ၚӡᰁᕆQPSጯӡᰁᕆහഝପᵞᗭහጯᰁᕆ

ᔄփᕹմӱݱ۸ᵱጱמ

կᵞ౮կܯ

DBA OPS DEV හഝᎸݎහഝړຉᆽঅᘏኞ

ᵱᥝՈғአکPostgreSQLܩဌፘىՈᵱොໜғᵱᥝ౮ᆧਠෆጱᥴොໜ$ yum instsall ampamp systemctl start ๅᶌᨏ᭄౮ғአጱ౮ېग़ጱԪ

ᰁݢᥡғਫහጯӻ436හጯӡ᭄භഠғᵞᗭਫݒᔺභഠԻӨளᖽᵱ᭄ݢአݢአṛᥝඳᵑᛔశᛔڔഘ᭄ದ๕᭗ᬦහഝḝṛလቘଘ؇ᆽრ

ӧమರᚸᓌܔঅአளӤಋฃᖌಷ᭄ۑᚆቘᑮᳵଧ-621ᬦᑕᇔ۸ᥤׯࢶഭᔱᒵ᭄ݢᶌහഝӧӶӧᲙӧԤղᅒؿବᛔԆݢഴӧమӤԯమᥝᔄԯጱᓕቘḵ

ӤಋளғӞᲫᯈॺἶقᓟܨአᳪདғ໐ጱବᯈᡦԞݢզ᪒ӞᛘғݎၥᦶሾहӨኞԾሾहכ೮Ӟᛘᥤ۸ݢሾहғ3WKRQ64ݳຉጱෆړᚆғ๕හഝቘӨۑق

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

Fellowship of PigstyLord of PostgreSQL

ᚆقഠဉᓟሾह

᮱ᗟᓕഴᥴොໜ

ፊഴᔮᕹᥴොໜ

PostgreSQLහഝପᤈᇇݎ

What is PigstyᓟܨአጱრPostgreSQLහഝପݎᤈᇇ۱ෛPG໐ғPostgreSQL 13314-beta2զၹᰁಘൊկҁ୩ێಘғଧහഝTimescaleDB 23ቘᑮᳵPostGIS 31ړୗCitus 10ق᮱ᓟܨአහഝ᮱ᗟଃݢᶌጱӾᳵկ Patroni 21 Pgbouncer115 Haproxy 22 Consul 110۱ኞԾᕆጱᬩᤈሾह Grafana 80 Prometheus 228 Nginx 120 Ansible

ᶎӫӱጱრPostgreSQLፊഴᥴොໜق

ӞᲫጱقᚆPostgreSQLဉᓟሾह

e Baery-Included Open Source PostgreSQL Disibu$on

ଃԧ20ӻᔜஞᦡᦇጱፊഴᶎਖ਼ፊഴຽ۸ԅग़قොጱړຉӨಅࢶᤒ᮷ଃӿጱᛯᬳളӥ᱂ӤܫཞݢளਧඳᵑӨᚆᳯᗝԧचԭPrometheus+AlertManagerӨGrafanaጱಸᦄᔮᕹӷᘏᒵපӬԅղᶼᗝଉአහഝପӨಸᦄᥢڞᓟܨአ

1000+ፊഴຽ ൈᬿԧහഝପ Ӿᳵկ ᅩ զᨮᤍጱොොᶎᶎᶮጱقੴᥦکੜጱܔӻහഝପҁᤒᔱڍහᒵ᮷Ӟᥦ෫

ຉPG CSV෭பጱଫአ pglogړአԭᤰचԭlokiӨpromtailጱਫ෭பතᵞොໜଚݢ

ᛔశጱṛݢአහഝପᵞᗭ Pigsty᮱ᗟጱහഝପᵞᗭݢզᛔଉᥠඳᵑӾ௩ᵞᗭ౮क़ᤒሿᒵհݝᥝՖձਫၚ ᵞᗭक़ጱݱᐿ๐ۓ੪ӧտྊกୗᯈᗝկᦜአ᭗ᬦ160+හൈᬿӨਧګහഝପӨचᏐᦡෞᬩᤈጱොොᶎᶎᘒ෫ᵱஞጱਫሿෆᔄKubernetesጱԯኞአḵ

ԧӞᔮڜᗝᒵۂ አԭ᮱ᗟಘᲀPostgreSQLහഝପᵞᗭӨਫ ᓕቘӱۓහഝପӨአአᓌܔොᛔଃጱCMDBಘӨਧګݢጱCLIӨGUI

Pigstyᰒय़ᥢཛྷኞԾሾहጱPostgreSQLᓕቘᘒᦡᦇԽᦡᦇԅݢਠෆᬩᤈԭ1໐1GBᡦӾԭכ೮ݎၥᦶᶼݎኞԾሾहጱӞᛘ

አݢզӨਫᴬኞԾሾहҁᴻᥢ໒क़ਠقӞᛘጱဉᓟሾहӾԟറᔱPostgreSQLᬰᤈݱᐿਫḵճၥᦶݎଚኞԾᕆፊഴᔮᕹӾقݐᶎӬਫጱහഝݍḇ

෫ᵱᘶᗑᦢᳯقෛCentOS 7ӤӞᤈե10ړᰦਠ౮ಅਞᤰᑕಅᕟկ૪ԅኞԾአঅٵҁӧᦇPigstyӥᕅ1GB

ᗝ๐ݎۓሿګᴻአԭṛݢአጱDCSक़හഝପӨचᏐᦡෞአຂᘠݳཛྷୗහഝପݒҁಘᖽӤӥᕚᛔᭇᯈ෫ᵱಋૡᖌಷचᏐᦡෞӨᬟᔮᕹ

አݢզአᗝጱPostgres amp Grafana amp EchartsݎԻහഝଫአӨහഝݢᥤ۸ߝளԾߝڊଚզຽٵጱොୗᏟਧጱሾहӾړՁᄍᐏӨԻ

ᓌܔฃአጱრPostgreSQLᓕഴᥴොໜ

አᛔ૩ጱᒟᦕ(Macbook)ӤݢӞᲫۑᚆਠෆጱဉᓟҁኧvagrantӨvirtualboxḝ᭗ᬦᓌܔጱᯈᗝᛔ૩ጱᡦԯECSVPSӤਠ౮᮱ᗟ

Pigsty将集群部署扩容缩容主从复制故障切换流量代理连接池服务发现访问控制监控系统告警系统志采集

等产级成熟解决案封装为发版次性解决产环境个使PostgreSQL数据库的各类问题真正做到 开箱即

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

开箱即

v7新UI

Software ToolkitProvisioningInfrastructure

PlatformGUICLIAPIMonitoringLoggingAlertingReportingService DiscoveryHAScalability

LBPoolingACLAuditSecuritybackup

KernelPostgreSQL

Timescale

PostGIS Citus

Pipeline

发版

ӻՈአ

կܯ

փᕹմӱ

ᘶᗑմӱ

Who wants PigstyӾੜᘶᗑմӱ ҁ෭ၚӡᰁᕆQPSጯӡᰁᕆහഝପᵞᗭහጯᰁᕆ

ᔄփᕹմӱݱ۸ᵱጱמ

կᵞ౮կܯ

DBA OPS DEV හഝᎸݎහഝړຉᆽঅᘏኞ

ᵱᥝՈғአکPostgreSQLܩဌፘىՈᵱොໜғᵱᥝ౮ᆧਠෆጱᥴොໜ$ yum instsall ampamp systemctl start ๅᶌᨏ᭄౮ғአጱ౮ېग़ጱԪ

ᰁݢᥡғਫහጯӻ436හጯӡ᭄භഠғᵞᗭਫݒᔺභഠԻӨளᖽᵱ᭄ݢአݢአṛᥝඳᵑᛔశᛔڔഘ᭄ದ๕᭗ᬦහഝḝṛလቘଘ؇ᆽრ

ӧమರᚸᓌܔঅአளӤಋฃᖌಷ᭄ۑᚆቘᑮᳵଧ-621ᬦᑕᇔ۸ᥤׯࢶഭᔱᒵ᭄ݢᶌහഝӧӶӧᲙӧԤղᅒؿବᛔԆݢഴӧమӤԯమᥝᔄԯጱᓕቘḵ

ӤಋளғӞᲫᯈॺἶقᓟܨአᳪདғ໐ጱବᯈᡦԞݢզ᪒ӞᛘғݎၥᦶሾहӨኞԾሾहכ೮Ӟᛘᥤ۸ݢሾहғ3WKRQ64ݳຉጱෆړᚆғ๕හഝቘӨۑق

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

ᚆقഠဉᓟሾह

᮱ᗟᓕഴᥴොໜ

ፊഴᔮᕹᥴොໜ

PostgreSQLහഝପᤈᇇݎ

What is PigstyᓟܨአጱრPostgreSQLහഝପݎᤈᇇ۱ෛPG໐ғPostgreSQL 13314-beta2զၹᰁಘൊկҁ୩ێಘғଧහഝTimescaleDB 23ቘᑮᳵPostGIS 31ړୗCitus 10ق᮱ᓟܨአහഝ᮱ᗟଃݢᶌጱӾᳵկ Patroni 21 Pgbouncer115 Haproxy 22 Consul 110۱ኞԾᕆጱᬩᤈሾह Grafana 80 Prometheus 228 Nginx 120 Ansible

ᶎӫӱጱრPostgreSQLፊഴᥴොໜق

ӞᲫጱقᚆPostgreSQLဉᓟሾह

e Baery-Included Open Source PostgreSQL Disibu$on

ଃԧ20ӻᔜஞᦡᦇጱፊഴᶎਖ਼ፊഴຽ۸ԅग़قොጱړຉӨಅࢶᤒ᮷ଃӿጱᛯᬳളӥ᱂ӤܫཞݢளਧඳᵑӨᚆᳯᗝԧचԭPrometheus+AlertManagerӨGrafanaጱಸᦄᔮᕹӷᘏᒵපӬԅղᶼᗝଉአහഝପӨಸᦄᥢڞᓟܨአ

1000+ፊഴຽ ൈᬿԧහഝପ Ӿᳵկ ᅩ զᨮᤍጱොොᶎᶎᶮጱقੴᥦکੜጱܔӻහഝପҁᤒᔱڍහᒵ᮷Ӟᥦ෫

ຉPG CSV෭பጱଫአ pglogړአԭᤰचԭlokiӨpromtailጱਫ෭பතᵞොໜଚݢ

ᛔశጱṛݢአහഝପᵞᗭ Pigsty᮱ᗟጱහഝପᵞᗭݢզᛔଉᥠඳᵑӾ௩ᵞᗭ౮क़ᤒሿᒵհݝᥝՖձਫၚ ᵞᗭक़ጱݱᐿ๐ۓ੪ӧտྊกୗᯈᗝկᦜአ᭗ᬦ160+හൈᬿӨਧګහഝପӨचᏐᦡෞᬩᤈጱොොᶎᶎᘒ෫ᵱஞጱਫሿෆᔄKubernetesጱԯኞአḵ

ԧӞᔮڜᗝᒵۂ አԭ᮱ᗟಘᲀPostgreSQLහഝପᵞᗭӨਫ ᓕቘӱۓහഝପӨአአᓌܔොᛔଃጱCMDBಘӨਧګݢጱCLIӨGUI

Pigstyᰒय़ᥢཛྷኞԾሾहጱPostgreSQLᓕቘᘒᦡᦇԽᦡᦇԅݢਠෆᬩᤈԭ1໐1GBᡦӾԭכ೮ݎၥᦶᶼݎኞԾሾहጱӞᛘ

አݢզӨਫᴬኞԾሾहҁᴻᥢ໒क़ਠقӞᛘጱဉᓟሾहӾԟറᔱPostgreSQLᬰᤈݱᐿਫḵճၥᦶݎଚኞԾᕆፊഴᔮᕹӾقݐᶎӬਫጱහഝݍḇ

෫ᵱᘶᗑᦢᳯقෛCentOS 7ӤӞᤈե10ړᰦਠ౮ಅਞᤰᑕಅᕟկ૪ԅኞԾአঅٵҁӧᦇPigstyӥᕅ1GB

ᗝ๐ݎۓሿګᴻአԭṛݢአጱDCSक़හഝପӨचᏐᦡෞአຂᘠݳཛྷୗහഝପݒҁಘᖽӤӥᕚᛔᭇᯈ෫ᵱಋૡᖌಷचᏐᦡෞӨᬟᔮᕹ

አݢզአᗝጱPostgres amp Grafana amp EchartsݎԻහഝଫአӨහഝݢᥤ۸ߝளԾߝڊଚզຽٵጱොୗᏟਧጱሾहӾړՁᄍᐏӨԻ

ᓌܔฃአጱრPostgreSQLᓕഴᥴොໜ

አᛔ૩ጱᒟᦕ(Macbook)ӤݢӞᲫۑᚆਠෆጱဉᓟҁኧvagrantӨvirtualboxḝ᭗ᬦᓌܔጱᯈᗝᛔ૩ጱᡦԯECSVPSӤਠ౮᮱ᗟ

Pigsty将集群部署扩容缩容主从复制故障切换流量代理连接池服务发现访问控制监控系统告警系统志采集

等产级成熟解决案封装为发版次性解决产环境个使PostgreSQL数据库的各类问题真正做到 开箱即

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

开箱即

v7新UI

Software ToolkitProvisioningInfrastructure

PlatformGUICLIAPIMonitoringLoggingAlertingReportingService DiscoveryHAScalability

LBPoolingACLAuditSecuritybackup

KernelPostgreSQL

Timescale

PostGIS Citus

Pipeline

发版

ӻՈአ

կܯ

փᕹմӱ

ᘶᗑմӱ

Who wants PigstyӾੜᘶᗑմӱ ҁ෭ၚӡᰁᕆQPSጯӡᰁᕆහഝପᵞᗭහጯᰁᕆ

ᔄփᕹմӱݱ۸ᵱጱמ

կᵞ౮կܯ

DBA OPS DEV හഝᎸݎහഝړຉᆽঅᘏኞ

ᵱᥝՈғአکPostgreSQLܩဌፘىՈᵱොໜғᵱᥝ౮ᆧਠෆጱᥴොໜ$ yum instsall ampamp systemctl start ๅᶌᨏ᭄౮ғአጱ౮ېग़ጱԪ

ᰁݢᥡғਫහጯӻ436හጯӡ᭄භഠғᵞᗭਫݒᔺභഠԻӨளᖽᵱ᭄ݢአݢአṛᥝඳᵑᛔశᛔڔഘ᭄ದ๕᭗ᬦහഝḝṛလቘଘ؇ᆽრ

ӧమರᚸᓌܔঅአளӤಋฃᖌಷ᭄ۑᚆቘᑮᳵଧ-621ᬦᑕᇔ۸ᥤׯࢶഭᔱᒵ᭄ݢᶌහഝӧӶӧᲙӧԤղᅒؿବᛔԆݢഴӧమӤԯమᥝᔄԯጱᓕቘḵ

ӤಋளғӞᲫᯈॺἶقᓟܨአᳪདғ໐ጱବᯈᡦԞݢզ᪒ӞᛘғݎၥᦶሾहӨኞԾሾहכ೮Ӟᛘᥤ۸ݢሾहғ3WKRQ64ݳຉጱෆړᚆғ๕හഝቘӨۑق

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

Pigsty将集群部署扩容缩容主从复制故障切换流量代理连接池服务发现访问控制监控系统告警系统志采集

等产级成熟解决案封装为发版次性解决产环境个使PostgreSQL数据库的各类问题真正做到 开箱即

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

开箱即

v7新UI

Software ToolkitProvisioningInfrastructure

PlatformGUICLIAPIMonitoringLoggingAlertingReportingService DiscoveryHAScalability

LBPoolingACLAuditSecuritybackup

KernelPostgreSQL

Timescale

PostGIS Citus

Pipeline

发版

ӻՈአ

կܯ

փᕹմӱ

ᘶᗑմӱ

Who wants PigstyӾੜᘶᗑմӱ ҁ෭ၚӡᰁᕆQPSጯӡᰁᕆහഝପᵞᗭහጯᰁᕆ

ᔄփᕹմӱݱ۸ᵱጱמ

կᵞ౮կܯ

DBA OPS DEV හഝᎸݎහഝړຉᆽঅᘏኞ

ᵱᥝՈғአکPostgreSQLܩဌፘىՈᵱොໜғᵱᥝ౮ᆧਠෆጱᥴොໜ$ yum instsall ampamp systemctl start ๅᶌᨏ᭄౮ғአጱ౮ېग़ጱԪ

ᰁݢᥡғਫහጯӻ436හጯӡ᭄භഠғᵞᗭਫݒᔺභഠԻӨளᖽᵱ᭄ݢአݢአṛᥝඳᵑᛔశᛔڔഘ᭄ದ๕᭗ᬦහഝḝṛလቘଘ؇ᆽრ

ӧమರᚸᓌܔঅአளӤಋฃᖌಷ᭄ۑᚆቘᑮᳵଧ-621ᬦᑕᇔ۸ᥤׯࢶഭᔱᒵ᭄ݢᶌහഝӧӶӧᲙӧԤղᅒؿବᛔԆݢഴӧమӤԯమᥝᔄԯጱᓕቘḵ

ӤಋளғӞᲫᯈॺἶقᓟܨአᳪདғ໐ጱବᯈᡦԞݢզ᪒ӞᛘғݎၥᦶሾहӨኞԾሾहכ೮Ӟᛘᥤ۸ݢሾहғ3WKRQ64ݳຉጱෆړᚆғ๕හഝቘӨۑق

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

v7新UI

Software ToolkitProvisioningInfrastructure

PlatformGUICLIAPIMonitoringLoggingAlertingReportingService DiscoveryHAScalability

LBPoolingACLAuditSecuritybackup

KernelPostgreSQL

Timescale

PostGIS Citus

Pipeline

发版

ӻՈአ

կܯ

փᕹմӱ

ᘶᗑմӱ

Who wants PigstyӾੜᘶᗑմӱ ҁ෭ၚӡᰁᕆQPSጯӡᰁᕆහഝପᵞᗭහጯᰁᕆ

ᔄփᕹմӱݱ۸ᵱጱמ

կᵞ౮կܯ

DBA OPS DEV හഝᎸݎහഝړຉᆽঅᘏኞ

ᵱᥝՈғአکPostgreSQLܩဌፘىՈᵱොໜғᵱᥝ౮ᆧਠෆጱᥴොໜ$ yum instsall ampamp systemctl start ๅᶌᨏ᭄౮ғአጱ౮ېग़ጱԪ

ᰁݢᥡғਫහጯӻ436හጯӡ᭄භഠғᵞᗭਫݒᔺභഠԻӨளᖽᵱ᭄ݢአݢአṛᥝඳᵑᛔశᛔڔഘ᭄ದ๕᭗ᬦහഝḝṛလቘଘ؇ᆽრ

ӧమರᚸᓌܔঅአளӤಋฃᖌಷ᭄ۑᚆቘᑮᳵଧ-621ᬦᑕᇔ۸ᥤׯࢶഭᔱᒵ᭄ݢᶌහഝӧӶӧᲙӧԤղᅒؿବᛔԆݢഴӧమӤԯమᥝᔄԯጱᓕቘḵ

ӤಋளғӞᲫᯈॺἶقᓟܨአᳪདғ໐ጱବᯈᡦԞݢզ᪒ӞᛘғݎၥᦶሾहӨኞԾሾहכ೮Ӟᛘᥤ۸ݢሾहғ3WKRQ64ݳຉጱෆړᚆғ๕හഝቘӨۑق

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

Software ToolkitProvisioningInfrastructure

PlatformGUICLIAPIMonitoringLoggingAlertingReportingService DiscoveryHAScalability

LBPoolingACLAuditSecuritybackup

KernelPostgreSQL

Timescale

PostGIS Citus

Pipeline

发版

ӻՈአ

կܯ

փᕹմӱ

ᘶᗑմӱ

Who wants PigstyӾੜᘶᗑմӱ ҁ෭ၚӡᰁᕆQPSጯӡᰁᕆහഝପᵞᗭහጯᰁᕆ

ᔄփᕹմӱݱ۸ᵱጱמ

կᵞ౮կܯ

DBA OPS DEV හഝᎸݎහഝړຉᆽঅᘏኞ

ᵱᥝՈғአکPostgreSQLܩဌፘىՈᵱොໜғᵱᥝ౮ᆧਠෆጱᥴොໜ$ yum instsall ampamp systemctl start ๅᶌᨏ᭄౮ғአጱ౮ېग़ጱԪ

ᰁݢᥡғਫහጯӻ436හጯӡ᭄භഠғᵞᗭਫݒᔺභഠԻӨளᖽᵱ᭄ݢአݢአṛᥝඳᵑᛔశᛔڔഘ᭄ದ๕᭗ᬦහഝḝṛလቘଘ؇ᆽრ

ӧమರᚸᓌܔঅአளӤಋฃᖌಷ᭄ۑᚆቘᑮᳵଧ-621ᬦᑕᇔ۸ᥤׯࢶഭᔱᒵ᭄ݢᶌහഝӧӶӧᲙӧԤղᅒؿବᛔԆݢഴӧమӤԯమᥝᔄԯጱᓕቘḵ

ӤಋளғӞᲫᯈॺἶقᓟܨአᳪདғ໐ጱବᯈᡦԞݢզ᪒ӞᛘғݎၥᦶሾहӨኞԾሾहכ೮Ӟᛘᥤ۸ݢሾहғ3WKRQ64ݳຉጱෆړᚆғ๕හഝቘӨۑق

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

ӻՈአ

կܯ

փᕹմӱ

ᘶᗑմӱ

Who wants PigstyӾੜᘶᗑմӱ ҁ෭ၚӡᰁᕆQPSጯӡᰁᕆහഝପᵞᗭහጯᰁᕆ

ᔄփᕹմӱݱ۸ᵱጱמ

կᵞ౮կܯ

DBA OPS DEV හഝᎸݎහഝړຉᆽঅᘏኞ

ᵱᥝՈғአکPostgreSQLܩဌፘىՈᵱොໜғᵱᥝ౮ᆧਠෆጱᥴොໜ$ yum instsall ampamp systemctl start ๅᶌᨏ᭄౮ғአጱ౮ېग़ጱԪ

ᰁݢᥡғਫහጯӻ436හጯӡ᭄භഠғᵞᗭਫݒᔺභഠԻӨளᖽᵱ᭄ݢአݢአṛᥝඳᵑᛔశᛔڔഘ᭄ದ๕᭗ᬦහഝḝṛလቘଘ؇ᆽრ

ӧమರᚸᓌܔঅአளӤಋฃᖌಷ᭄ۑᚆቘᑮᳵଧ-621ᬦᑕᇔ۸ᥤׯࢶഭᔱᒵ᭄ݢᶌහഝӧӶӧᲙӧԤղᅒؿବᛔԆݢഴӧమӤԯమᥝᔄԯጱᓕቘḵ

ӤಋளғӞᲫᯈॺἶقᓟܨአᳪདғ໐ጱବᯈᡦԞݢզ᪒ӞᛘғݎၥᦶሾहӨኞԾሾहכ೮Ӟᛘᥤ۸ݢሾहғ3WKRQ64ݳຉጱෆړᚆғ๕හഝቘӨۑق

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

ӻՈኪᚏဉᓟሾह

ԯᡦᄍᐏሾह

ၥᦶݎճሾह

ኞԾሾह

Where to run Pigstyᓕቘᅩғ3-5

አṛᥢ໒ᇔቘᡦ᮱ᗟአӻᓕቘᅩഴګፊᓕپጯॺහഝପᵞᗭਫᴬ๐ۓԭᕚӤӱۓӞॺ3LJVW᮱ᗟඪ೮پጯॺහഝପᵞᗭپጯӡ436Ӥӡ෭ၚӧᦾӥ

ᓕቘᅩғ1-3

ᓕቘᅩғ1

ᓕቘᅩғ1

ӨኞԾሾहכ೮Ӟᛘጱ6WDJLQJ8$7ᶼݎሾह൫ୌلአጱݎၥᦶሾहճᕚӤሾह

አԯ๐ۓ᮱ᗟړՁلᄍᐏᑕଧள๐ۓੜஙහഝӱۓ

ᬩᤈԭӻՈᒟᦕampݣୗMac amp PC ᡦӾአԭ൫ୌHYER[ၥᦶਫḵԟ3RVWJUH64ᥤ۸ଫአݢහഝଫአԻୗහഝګ

64໐ 400GB PCI-E SSD x n

8໐ 16B SSD x n

2໐ 4GB x 1ӗ4

1໐ 2GB x 1

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

痛点痒点爽点功能

管理

机器成本

性能

安装

部署

维护

实施

可环境

平台

可靠性

监控系统

SQL变更

成本

伸缩性

扩展性

商锁定

社区活

商持

才储备

学习成本

复杂度

技能通性

技能市场价值

软件成本项命

技能贬值速度

演示

可扩展性

数据安全

险

审计

志

访问控制

权限管理

数据迁移

扩容缩容

户管理

备份案

故障预案

恢复案

软件升级

插件管理

配置管理

分区分

地理空间

时间序列

全检索

告警系统

参数调优

动化

中间件

可观测性

负载均衡

接

并发

存储过程

JSONXML

ACID事务

读写分离

主从延迟

HTAP

CDC

ETL

JOIN

投资认可度

炒作价值

吹嘘价值

谈资审美

弹性伸缩

调度

态易性是否开源

协议费

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

从到有痛点

痒点

爽点

从有到优

从优到易

不可替代性 从易到廉 开源免费

开箱即全能沙箱

发版部署解决案

监控系统可架构

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

Installation Wizard

安装

Someone told me that each equation I included in the book would halve the salesmdash Stephen Hawking 有告诉我安装脚本每多命令户数量就会减半

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

ӥრդᎱ (ᶱ) pigstytgz --mdashӥᥴᛗmdashmdashgt ~pigsty

կ۱ (ݢᶱ) pkgtgz mdashmdashmdashmdashmdashmdashmdashmdashmdashmdashgt tmppkgtgz

$ git clone httpsgithubcomVonngpigsty$ cd ~

$ curl -SL httpsgithubcomVonngpigstyreleasesdownloadv100-beta1pigstytgz | gzip -d | tar -xC ~

አcurlӥ

$ curl -SLO httpsgithubcomVonngpigstyreleasesdownloadv100-betapkgtgz -o tmppkgtgz

ӥݢᐶᕚկਞᤰ۱ےݢਞᤰፓຽᅩ෫ᘶᗑᦢᳯԅᐶᕚկ۱ԞݢզconfigureᬦᑕӾӥ

ᯈᗝ$ configure $ cd ~pigsty

[-n|--non-interactive] ইຎ૪ᕪᏟਧԧӣӻහᶱݢզአ -n ᬦԻୗݻ[-m|--mode ltconfig_modegt] ਧᵱᥝአጱᯈᗝկཛྷғdemo demo4 pg14 tiny oltpᒵ[-d|--download] ইຎtmppkgtgzӧ੪Githubӥᐶᕚਞᤰ۱[-i|--ip ltprimary_ipgt] ग़ӻIPਧӾӞӻԅḒᥝIP

ٵ໐ Linux ຝ x86_64ᇇ CentOS 782003 (RHEL7ᒵපᇇ)

አ root (عᎱsudoአ)

ᥢ໒ 1໐2G (വគአ2໐4GզӤᥢ໒)

$ echo $(uname -s) $(uname -m) $(cat etcredhat-release) Linux x86_64 CentOS Linux release 782003 (Core)

How to get Pigsty

ਞᤰ$ make install

Ansibleۂinfraymlਖ਼metaړᕟӤಗᤈզਠ౮ਞᤰansibleਖ਼configureᬦᑕӾᐶᕚկ۱ݢአጱრӾਞᤰ

ᦢᳯᧆᅩጱ3000ᒒݢܨݗၨᥦ3LJVWፊഴḒᶭhttp101010103000 ഘԅᛔ૩ጱ3ἕᦊአݷӨᎱ adminpigsty

አऒݷਖ਼ଃๅঅጱአḵই httpgpigsty ಗᤈsudo make dns տਖ਼Pigstyፘىऒفݷetchosts10101010 meta pigsty gpigsty

Get pigs wi ree lines of code

Ꮯᦊ෫ݢܨݸতӥᯈᗝਞᤰ

አᐿᯈᗝҘ(᭗ᬦ-mහਧ)demo (ဉᓟἕᦊܔᅩ) demo4 (4ᅩဉᓟ) pg14 tiny (ଉᥢἕᦊ) oltp hellipฎވӥկ۱ (ইຎtmppkgtgzӧڞտᧃᳯ)

Y ሿGithubӤӥpkgtgzN ݸᖅਞᤰতӤრӥ (ইຎᔮᕹᶋCentOS 78᧗አྌᶱ)ԆᥝጱIP (ইຎݎሿग़ᗑܜӨग़ӻݢአIPտᧃᳯ)ᬌف ୮ڹᅩҁᓕቘᅩአጱIP(ᦢᳯᧆᅩአጱIPӧᥝአلᗑIP)

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

A WIZARD

Installation download rarr configure rarr install

httpspigstycczhdocsquick-start

Monitoring

You canrsquot manage what you donrsquot MEASUREmdash Peter FDrucker

监控系统

产验证案

规模集群管理

多层次全覆盖

开源监控 佳实践

专业 全

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statments

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

Domains

Entity Relation Identifier

Database

Instance

Cluster

Overview

Service

pg-meta-1

pg-test-1testpg-test-2testpg-test-3test

pg-test-1pg-test-2pg-test-3

pg-meta-1meta

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

pg-meta-primarypg-meta-replicapg-meta-defaultpg-meta-offline

pg-metapg-testpg-test1pg-test2

Shard pg-test

IdentifierEntity Label

job = ldquopgsqlrdquo

job = ldquopgsqlrdquo cls =~ lsquopg-testd+rsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo ins = lsquopg-test-1rsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo ins = lsquopg-test-1rsquo datname = lsquotestrsquo

Objectqueryid = 1078821737499742052relname = publicpgbench_accountsidxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

job = ldquopgsqlrdquo cls = lsquopg-testrsquo ins = lsquopg-test-1rsquo datname = lsquotestrsquo relname = lsquopublicpgbench_accountsrsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo svc = lsquopg-test-primaryrsquo

pgsqlredismysql

Entities amp Identifiers

v7新UI

PGSQL Overview

PGSQL Alert

Grafana

Prometheus AlertManager

等效实现 互为补充

交互式图表

点击跳转 查看详情 屏蔽告警 下钻具体集群实例

PGSQLCluster

cluster-wide insights

PGSQLActivity

Replication

跟踪复制进度

监控复制延迟

定位复制延迟问题

逻辑复制进度

同步复制状态

Service

Load BalancerProxy InstanceTraffic Controlhelliphellip

HAProxy Admin

点击管理图标

直接从监控板跳转

流量管理板

ServiceRegistry

With

Consul

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

监控系统

产验证案

规模集群管理

多层次全覆盖

开源监控 佳实践

专业 全

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statments

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

Domains

Entity Relation Identifier

Database

Instance

Cluster

Overview

Service

pg-meta-1

pg-test-1testpg-test-2testpg-test-3test

pg-test-1pg-test-2pg-test-3

pg-meta-1meta

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

pg-meta-primarypg-meta-replicapg-meta-defaultpg-meta-offline

pg-metapg-testpg-test1pg-test2

Shard pg-test

IdentifierEntity Label

job = ldquopgsqlrdquo

job = ldquopgsqlrdquo cls =~ lsquopg-testd+rsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo ins = lsquopg-test-1rsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo ins = lsquopg-test-1rsquo datname = lsquotestrsquo

Objectqueryid = 1078821737499742052relname = publicpgbench_accountsidxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

job = ldquopgsqlrdquo cls = lsquopg-testrsquo ins = lsquopg-test-1rsquo datname = lsquotestrsquo relname = lsquopublicpgbench_accountsrsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo svc = lsquopg-test-primaryrsquo

pgsqlredismysql

Entities amp Identifiers

v7新UI

PGSQL Overview

PGSQL Alert

Grafana

Prometheus AlertManager

等效实现 互为补充

交互式图表

点击跳转 查看详情 屏蔽告警 下钻具体集群实例

PGSQLCluster

cluster-wide insights

PGSQLActivity

Replication

跟踪复制进度

监控复制延迟

定位复制延迟问题

逻辑复制进度

同步复制状态

Service

Load BalancerProxy InstanceTraffic Controlhelliphellip

HAProxy Admin

点击管理图标

直接从监控板跳转

流量管理板

ServiceRegistry

With

Consul

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statments

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

Domains

Entity Relation Identifier

Database

Instance

Cluster

Overview

Service

pg-meta-1

pg-test-1testpg-test-2testpg-test-3test

pg-test-1pg-test-2pg-test-3

pg-meta-1meta

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

pg-meta-primarypg-meta-replicapg-meta-defaultpg-meta-offline

pg-metapg-testpg-test1pg-test2

Shard pg-test

IdentifierEntity Label

job = ldquopgsqlrdquo

job = ldquopgsqlrdquo cls =~ lsquopg-testd+rsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo ins = lsquopg-test-1rsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo ins = lsquopg-test-1rsquo datname = lsquotestrsquo

Objectqueryid = 1078821737499742052relname = publicpgbench_accountsidxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

job = ldquopgsqlrdquo cls = lsquopg-testrsquo ins = lsquopg-test-1rsquo datname = lsquotestrsquo relname = lsquopublicpgbench_accountsrsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo svc = lsquopg-test-primaryrsquo

pgsqlredismysql

Entities amp Identifiers

v7新UI

PGSQL Overview

PGSQL Alert

Grafana

Prometheus AlertManager

等效实现 互为补充

交互式图表

点击跳转 查看详情 屏蔽告警 下钻具体集群实例

PGSQLCluster

cluster-wide insights

PGSQLActivity

Replication

跟踪复制进度

监控复制延迟

定位复制延迟问题

逻辑复制进度

同步复制状态

Service

Load BalancerProxy InstanceTraffic Controlhelliphellip

HAProxy Admin

点击管理图标

直接从监控板跳转

流量管理板

ServiceRegistry

With

Consul

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Database

Instance

Cluster

Overview

Service

pg-meta-1

pg-test-1testpg-test-2testpg-test-3test

pg-test-1pg-test-2pg-test-3

pg-meta-1meta

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

pg-meta-primarypg-meta-replicapg-meta-defaultpg-meta-offline

pg-metapg-testpg-test1pg-test2

Shard pg-test

IdentifierEntity Label

job = ldquopgsqlrdquo

job = ldquopgsqlrdquo cls =~ lsquopg-testd+rsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo ins = lsquopg-test-1rsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo ins = lsquopg-test-1rsquo datname = lsquotestrsquo

Objectqueryid = 1078821737499742052relname = publicpgbench_accountsidxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

job = ldquopgsqlrdquo cls = lsquopg-testrsquo ins = lsquopg-test-1rsquo datname = lsquotestrsquo relname = lsquopublicpgbench_accountsrsquo

job = ldquopgsqlrdquo cls = lsquopg-testrsquo svc = lsquopg-test-primaryrsquo

pgsqlredismysql

Entities amp Identifiers

v7新UI

PGSQL Overview

PGSQL Alert

Grafana

Prometheus AlertManager

等效实现 互为补充

交互式图表

点击跳转 查看详情 屏蔽告警 下钻具体集群实例

PGSQLCluster

cluster-wide insights

PGSQLActivity

Replication

跟踪复制进度

监控复制延迟

定位复制延迟问题

逻辑复制进度

同步复制状态

Service

Load BalancerProxy InstanceTraffic Controlhelliphellip

HAProxy Admin

点击管理图标

直接从监控板跳转

流量管理板

ServiceRegistry

With

Consul

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

v7新UI

PGSQL Overview

PGSQL Alert

Grafana

Prometheus AlertManager

等效实现 互为补充

交互式图表

点击跳转 查看详情 屏蔽告警 下钻具体集群实例

PGSQLCluster

cluster-wide insights

PGSQLActivity

Replication

跟踪复制进度

监控复制延迟

定位复制延迟问题

逻辑复制进度

同步复制状态

Service

Load BalancerProxy InstanceTraffic Controlhelliphellip

HAProxy Admin

点击管理图标

直接从监控板跳转

流量管理板

ServiceRegistry

With

Consul

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

PGSQL Overview

PGSQL Alert

Grafana

Prometheus AlertManager

等效实现 互为补充

交互式图表

点击跳转 查看详情 屏蔽告警 下钻具体集群实例

PGSQLCluster

cluster-wide insights

PGSQLActivity

Replication

跟踪复制进度

监控复制延迟

定位复制延迟问题

逻辑复制进度

同步复制状态

Service

Load BalancerProxy InstanceTraffic Controlhelliphellip

HAProxy Admin

点击管理图标

直接从监控板跳转

流量管理板

ServiceRegistry

With

Consul

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

PGSQL Alert

Grafana

Prometheus AlertManager

等效实现 互为补充

交互式图表

点击跳转 查看详情 屏蔽告警 下钻具体集群实例

PGSQLCluster

cluster-wide insights

PGSQLActivity

Replication

跟踪复制进度

监控复制延迟

定位复制延迟问题

逻辑复制进度

同步复制状态

Service

Load BalancerProxy InstanceTraffic Controlhelliphellip

HAProxy Admin

点击管理图标

直接从监控板跳转

流量管理板

ServiceRegistry

With

Consul

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

交互式图表

点击跳转 查看详情 屏蔽告警 下钻具体集群实例

PGSQLCluster

cluster-wide insights

PGSQLActivity

Replication

跟踪复制进度

监控复制延迟

定位复制延迟问题

逻辑复制进度

同步复制状态

Service

Load BalancerProxy InstanceTraffic Controlhelliphellip

HAProxy Admin

点击管理图标

直接从监控板跳转

流量管理板

ServiceRegistry

With

Consul

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

PGSQLCluster

cluster-wide insights

PGSQLActivity

Replication

跟踪复制进度

监控复制延迟

定位复制延迟问题

逻辑复制进度

同步复制状态

Service

Load BalancerProxy InstanceTraffic Controlhelliphellip

HAProxy Admin

点击管理图标

直接从监控板跳转

流量管理板

ServiceRegistry

With

Consul

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

PGSQLActivity

Replication

跟踪复制进度

监控复制延迟

定位复制延迟问题

逻辑复制进度

同步复制状态

Service

Load BalancerProxy InstanceTraffic Controlhelliphellip

HAProxy Admin

点击管理图标

直接从监控板跳转

流量管理板

ServiceRegistry

With

Consul

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Replication

跟踪复制进度

监控复制延迟

定位复制延迟问题

逻辑复制进度

同步复制状态

Service

Load BalancerProxy InstanceTraffic Controlhelliphellip

HAProxy Admin

点击管理图标

直接从监控板跳转

流量管理板

ServiceRegistry

With

Consul

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Service

Load BalancerProxy InstanceTraffic Controlhelliphellip

HAProxy Admin

点击管理图标

直接从监控板跳转

流量管理板

ServiceRegistry

With

Consul

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

HAProxy Admin

点击管理图标

直接从监控板跳转

流量管理板

ServiceRegistry

With

Consul

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

ServiceRegistry

With

Consul

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

PG Instance

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

NodeProxy

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

XactsSession QueriesPersist

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

PGSQL Database

数据库级别指标

快速访问

每张表

每个查询

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

PGSQL Table

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

PGSQL Query

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

PGCAT

动为新数据库注册数据源

通过Grafana直接浏览系统录

带来限新可能性

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

PGCAT Bloat

表与索引膨胀总览

从Catalog获取信息

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Table Query

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

PGLOG Analysis

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

PGLOG Session

bull 单条连接详情

bull Annotation

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

PG Instance Log

bull Loki + Promtail bull Realtime log metricsbull Full-text search

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Summary based on pgbadgerLog Events

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Original Statements Metrics Query Statements Dashboards

Observability

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

PG 14

new metrics are

READY

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

metrics data flowMetrics Source

pg_exporter

metrics pgbouncer_

Aggregated Metrics Namespace pg node haproxy

Pgbouncer connection pooler metrics such as QPS TPS RT session etchellip

pg_exporter

Database

metrics pg_

Postgres database metrics such as activities

replication persist etchellip

Middleware

node_exporter

Operating System

metrics node_

Node Podmetrics for

processors network disks filesystem

network etchellip

haproxy_exporter

Load Balancer

metrics haproxy_

HAProxymetrics for health

status traffic routes backend weights

etchellip

Hierarchy Metrics all cls svc ins datname dev pool

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

metric type for a clusterMetrics Ratio

metrics type count

170

83

281344

48

229

pg_ pgbouncer_ pg node node haproxy

1155 metrics per env

3k~10k+ time-series per instance

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

metrics numbers comparing to othersMetrics Coverage

2016

Pigsty

pgmonitor

pgwatch

AWS RDS

datadog

Aliyun RDS

0 188 375 563 750

8

69

99

123

156

621

metrics related to database are counted (ie node metrics are excluded)

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Overhead

Typical Scrape Duration 10~100ms

Typical Scrape Count 1500 ~ 4500

Typical Scrape Interval 15s

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Provisioning

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

Node

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

我们必须跳出电脑指令序列的窠 叙述定义描述元数据梳理关系不是编写过程

Interface

Clearly we must break away from the sequential and not limit the computers We must state definitions and provide for priorities and descriptions of data We must state relation- ships not procedures mdashGrace Murray Hopper Management and the Computer of the Future (1962)

mdashmdash Grace Murray Hopper未来的计算机及其管理（1962）

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

管控

DBMS包括数据库内核和配套软件

云商卖开源PG卖的不是内核是 数据库管控能

数据库是管理数据的软件管控系统是管理数据库的软件

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

pg-test ਧԎӞӻෛጱ pg-test ᵞᗭ hosts ۱ތӞԆӷӣӻਫ᮱ᗟӣӻᅩӤ 10101011 pg_seq 1 pg_role primary 10101012 pg_seq 2 pg_role replica 10101013 pg_seq 3 pg_role replica vars ᵞᗭݷᑍአጱVIPӱۓአӨහഝପ pg_cluster pg-test vip_address 1010102 pg_users [ name test ] pg_databases [ name test ] Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

pgsqlyml -l pg-test

Provisioning

从配置到实现只要命令

创建出该集群

定义套集群只要代码

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Interface

infrayml ᮱ᗟץදचᏐᦡෞpgsqlyml ᮱ᗟץදහഝପᵞᗭਫpgsql-removeyml ӥᕚහഝପᵞᗭਫpgsql-createuseryml ڠୌץදӱۓᜋampአpgsql-createdbyml ڠୌץදӱۓහഝପ

Idempotent Ansible Playbooks

infrayml --tags=environ ෛᓕቘᅩӤᯈᗝሾह

infrayml --tags=repo -e repo_rebuild=true ୩ګෛڠୌრ

infrayml --tags=repo_upstream فےӤYumRepo

infrayml --tags=prometheus ෛڠୌPrometheus

infrayml --tags=nginx_confignginx_restart ෛኞ౮Nginxᯈᗝկ

चᏐᦡෞڡত۸pgsqlyml --tags=infra ਠ౮चᏐᦡෞጱڡত۸۱ೡᅩڡত۸ӨDCS᮱ᗟ

pgsqlyml --tags=node ਠ౮ᅩጱڡত۸᭗ଉӧտߥᬩᤈӾහഝପਫ

pgsqlyml --tags=dcs ਠ౮DCSғconsuletcdጱڡত۸

pgsqlyml --tags=dcs -e dcs_exists_action ਠ౮consuletcdጱڡত۸୩ګಾᴻ

හഝପڡত۸pgsqlyml --tags=pgsql ਠ౮හഝପ᮱ᗟғහഝପፊഴ๐ۓ

pgsqlyml --tags=postgres ਠ౮හഝପ᮱ᗟ

pgsqlyml --tags=monitor ਠ౮ፊഴጱ᮱ᗟ

pgsqlyml --tags=service ਠ౮ᨮᤍጱ᮱ᗟҁHaproxy amp VIP

pgsqlyml --tags=register ਖ਼๐ۓᛗचᏐᦡෞ

---------------------------------------------------------------- name Infra Init init infra on common database node become yes hosts all gather_facts no tags infra roles

- role node init common database node tags node

- role consul init dcsconsul clients tags [ dcs consul ]

---------------------------------------------------------------- name Pgsql Init init postgres clusterinstance become yes hosts all gather_facts no tags pgsql roles

- role postgres init postgres pgbouncer patroni tags postgres

- role monitor init monitor exporters tags monitor

- role service init service lb vip tags service

- role register register clusterinstance to infra tags register

---------------------------------------------------------------

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

CLI vs GUI

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

160+配置参数全定制数据库与基础设施 （可以不不能没有）

Completeness

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

High Available

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Architecture

A complex system that works is invariably found to have evolved from a simple system that works

The inverse proposition also appears to be true A complex system designed from scratch never works and cannot be made to work

mdashJohn Gall Systemantics (1975)

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTP

Repo

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

visualize

firing

expose

servicediscovery

delegate

metrics

Infrastructure

Versatile Application Runtime

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Managed by vip-managerTied to primary via dcs info

VIP

consul etcd clusterDistributed Configuration System

DCS

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Leader electionConfiguration Management

HA Agent

Monitor

5433 5434 5436 5438

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

vip-manager

Database Monitorservice discoveryleader electionhealth check

traffic

peer nodes

metricsscrape

registerconsensus

supervise

leaderinquiry

pooling

healthcheck

Database Node 以 复杂度实现必须功能

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

VIP

DCS

Load Balancer

Middleware

Database

HA Agent

Monitor

DNS

Access

Alert

TSDB

DCS

Pigsty

Graph

NTPNTP

DNS

Patroni8008

Consul 8500

Postgres5432

primary

Pgbouncer6432 9631PGB Exporter

9100Node Exporter

5433 5434 5436 5438

Haproxy 9101

9630PG Exporter

5433 5434 5436 5438

vip-manager

Prometheus 9090

8500 ConsulServer

80 Yum Repo

123NTPD Server

DNSMASQ53

Alert Manager 9093

Grafana 3000

MetaDB5432

primary

h c y a p g

Nginx80

8600ConsulDNS

Ansible Cronjobs

NTPChronyconsensus

visualize

traffic

peer nodes

firing

scrape

expose

register

consensus

supervise

leaderinquiry admin

pooling

identitymapping

healthcheck

metrics

resolve

install

delegate

time-sync

Infrastructure amp Node

数据库集群与基础设施松耦合通过注册的式关联 基础设施升级amp故障不会影响数据库集群

（除可使的DCS外维护模式下数据库集群亦免受DCS影响）

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Database Cluster

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

基于DCS

Fencing amp Softdog

动故障切换

流量路由动修复

实例提供幂等的服务

集群可动从常故障中恢复 实例成员对外表现等价 只要仍有任意实例存活

集群对外提供的各种服务就不会停

从库故障基本影响 主库故障切换时读写流量影响约15秒

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

standby service will route ip|name5435 to sync replicas pgbouncer (5435-gt6432 standby)- name standby required service name the actual svc name will be prefixed with `pg_cluster` eg pg-meta-standby src_ip required service bind ip address `` for all ip `vip` for cluster `vip_address` src_port 5435 required service exposed port (work as kubernetes service node port mode) dst_port postgres optional destination port postgres|pgbouncer|ltport_numbergt pgbouncer(6432) by default check_method http optional health check method http is the only available method for now check_port patroni optional health check port patroni|pg_exporter|ltport_numbergt patroni(8008) by default check_url read-onlylag=0 optional health check url path by default check_code 200 optional health check expected http code 200 by default selector [] required JMESPath to filter inventory () selector_backup [ pg_role == `primary`] primary used as backup server for standby service (will not work because sync for ) haproxy optional adhoc parameters for haproxy service provider (vip_l4 is another service provider) maxconn 3000 optional max allowed front-end connection balance roundrobin optional haproxy load balance algorithm (roundrobin by default other leastconn) default_server_options inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100

Service

使配置件定义新的服务

采NodePort的式对外提供服务

任何实例都对外暴露完整服务通过不同端区分

从外部户看来集群中每个实例都是相同的

传统数据库带来 分布式数据库 的体验

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

L2 VIP1010103

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

supervise

consensus

pooling

activebackup

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

L2 VIP managed by vip-managerTied to primary via dcs info

VIP

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + L2 VIP + HAProxy DNS + L4 VIP

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

5433 5434 5436 5438

Haproxy

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Idempotent LB that expose services Service are distinguished via ports

Load Balancer

no vip enabled

VIP

pg-test resolve to allhaproxy instances among cluster

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test

5433 5434 5436 5438

L4 VIP1010103

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

DNS + HAProxy DNS + L4 VIP

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Consul DNS SD Pure DNS

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

Consul DNSreplicapg-test

6432 5432

Consul DNSprimarypg-test

6432 5432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

No dns used

VIP

primarypg-test resolve primary instancereplicapg-test resolve replica instances

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

DNSpg-test-replica

DNSpg-test-primary

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

Use L4 VIP as load balancer directly

Load Balancer

Work as Load balancerRoute traffic via health check

VIP (L4)

pg-test resolve to VIPor fixed load balancer

DNS

ApplicationAccess via pg-testDistinguish services via port

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestestpg-test5434test

postgresdbuser_dbapg-test5436testdb

postgresdbuser_statspg-test5438testdb

non-interactive read-write access via pool

non-interactive read-only access via pool

interactive primary direct access (DDL DML Admin)

interactive offline direct access (ETLSAGAPersonal)

postgrestestpg-test5433test

Patroni8008

Consul8500

Postgres5432

replica

supervise

consensus

Pgbouncer6432

pooling

Patroni8008

Consul8500

Postgres5432

primary

Pgbouncer6432

Patroni8008

Consul8500

Postgres5432

replica

Pgbouncer6432

replication

consul etcd clusterDistributed Configuration System

DCS

Leader electionConfiguration Management

HA Agent

Connection poolerStatistic collector

Middleware

Streaming replication

Database

no lb used

Load Balancer

QRYLSXVHG

VIP (L4)

no dns used

DNS

Applicationaccess via IP address

node-2 10101012 pg-test-2

replica node-110101011 pg-test-1

primary node-3 10101013 pg-test-3

replica

Access Layer

traffic replica replicationprimary superviseofflinedefault checkconsensus inquiry

replica

primary

offline

default

postgrestest101010116432101010126432101010136432testtarget_session_attrs=prefer-standby

postgresdbuser_dba101010111010101210101013testtarget_session_attrs=read-write

postgresdbuser_stats101010111010101210101013testtarget_session_attrs=standby

postgrestest101010116432101010126432101010136432testtarget_session_attrs=primary

IP + Smart Client

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Sandbox

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

meta 10101010

pg_cluster pg-meta

pg_role primary

pg_instance pg-meta-1

pg_service pg-meta-primary

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-2 10101012

pg_cluster pg-test

pg_role replica

pg_instance pg-test-2

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-3 10101013

pg_cluster pg-test

pg_role replica

pg_instance pg-test-3

pg_service pg-test-replica

Haproxy5433

5434

Pgbouncer6432

Postgres5432

Patroni8008

Node Exporter 9100

PGB Exporter 9631

PG Exporter 9630

Consul 8500

node-1 10101011

pg_cluster pg-test

pg_role primary

pg_instance pg-test-1

pg_service pg-test-primary

Prometheus 9090

8500DCS Server

Nginx80

80 Yum Repo

123 NTP Server

DNS53 Server

Alert Manager 9093

Grafana 3000

Infra

pg-meta

1010102 1010103

pg-test

bind on primary

resolve to VIP

access via DNS

10101010 10101011

Service

本地运键拉起低配置压

Just like Minikube for Kubernetes

JupyterlabGrafana Echarts 便易助数据分析

确定性的环境便于交付演示案例

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

InstanceCluster

Node Service

contiain expose

deploy

pg-test

Database

pg-test-1test

pg-test-1pg-test-2pg-test-3

pg-test-primarypg-test-replicapg-test-defaultpg-test-offlinepg-test-standby

Table

queryid = 1078821737499742052idxname = pigstycluster_pkeyfuncname = monitorpg_stat_statements

Instance serve

contains

Object

1

n

1

n

1

n

1

n

1

1

contain

1

n

contain

n

1contain

n

1

relname = publicpgbench_accounts

101010111010101210101013

InstanceEnvironment

Config Infrastructure

has has

pgsql

contains

1

n

1

1

1

1Domain Extended

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Analytic 数据分析与可视化

The best vision is insightmdash Malcolm Forbes

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Pigsty for Analysis

专注于数据处理分析与可视化

使Grafana承载rdquo前端rdquo与rdquo后端的主体功能

使PostgreSQL作为主要数据库存储数据并实现业务逻辑

使Echarts与Grafana实现数据可视化与交互功能

采统的格式可以在Pigsty环境中键安装运

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

样例COVID-19 疫情数据盘点击国家跳转对应国家详情数据

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Grafana太丑Echarts来凑100代码解决所有

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

全球地表象站历史数据查询

3万个地表象站

120年来时级

原始象观测数据

样例ISD

点击象站跳图标

跳转单象站详情

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

单象站历史数据摘要查询（年）

点击份图例跳转具体份原始观测记录点击观测站图例跳转周边观测站

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

单个象站原始观测数据（时级）

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

带应Pigsty CMDB

使PG管理PG

More

将静态配置件

替换为动态CMDB查询

从与外部系统与平台集成

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Beta应Redis

使同样的基础设施

部署监控Redis

More

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Open Source

A system cannot be successful if it is too strongly influenced by a single person Once the initial design is complete and fairly robust the real test begins as people with many different viewpoints undertake their own experiments

mdash Donald Knuth

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Pigsty Timeline1

2

3

4

2020

5

6

7

8

9

10

11

12

1

2

3

4

2021

5

6

7

8

9

10

11

12

2019-05-15

Pigsty

v002

v030

v004

v003

v005

v040

v050

v060

v070v080v090

v091

v100

v110

v120

v130

v1402022

2020-06-20

2021-08-01

ஷғpg-test ဉᓟ

2020-04-30

2020-07-10

2020-07-27

2020-08-19

2020-10-22

2020-12-14

2021-01-07

2021-02-19

2021-03-01

2021-03-28

2021-04-20

2021-09-01

2021-10-01

2021-11-01

2021-12-01

ෆຝਧ

Yၥᦶሾहḵᦤ

Yᶼݎሾहḵᦤ

զ$QVLEOH5ROHཛྷୗ

ᐶᕚਞᤰཛྷୗ

හഝପᕳොໜෆݢአ

33DWURQLඪ೮ਥොᒊӤᕚ

ਧګහഝପཛྷඪ೮

ᕳොໜදᬰ

ፊഴᔮᕹᇿᒈ᮱ᗟඪ೮

ਧګ๐ۓӨളفොୗampૡ

ਞᤰᑕս۸ฃአ8ૡಘ32ګampਥොamp1

ෆץᗂਠ౮Ḓӻ$ᇇ

ਠ643Ө3amp$7ಘଫአ

ਠ23Ө3)2ಘଫአ

7LPHVFDOHampLWXV᮱ᗟඪ೮

ἕᦊහഝପᇇᕆᛗ3RVWJUH64

2019

2020-11-20 ۹Ղ2020හഝପଙ3RVWJUH64ፊഴᔮᕹń3LJVW

2021-01-16 ଠ2020 PostgreSQLӾದय़տ ኞԾᕆ3RVWJUH64ፊഴᔮᕹń3LJVW

2020-04-18 PostgreSQLӾᐒፗඎPGṛݢአ៧ਫғPatroni

2020-06-06 PostgreSQLӾᐒፗඎPostgreSQLፊഴғἎᰂຽӨᵞᗭፊഴ

2020-08-23 PostgreSQLӾᐒፗඎPostgreSQLӨහഝݢᥤ۸

2021-01-07 ᴨԯPgSQLӠॠᦒᕞ០PostgreSQLፊഴਫचԭPigstyᥴਫᴬፊഴᳯ

2021-06-08 य़ᬳቘૡय़ لፅദ හഝପᤈӱفᳪܖWhatWhy How amp Pigsty

2021-07-31 რӾશտ-ړୗහഝପහഝପړտ ᓟܨአጱහഝପݎᤈᇇPigsty

202108-31 DTCC-2021-ଧහഝପӫ Prometheusහഝཛྷ

2021-05-23 PostgreSQLӾᐒፗඎრPGقਹӤಋܖmdashmdashPigsty v091

2020-08-20 क़᮱አғ᮱ᴚ

2021-02-24 क़᮱አғᱷᤈၾᩇᰂᣟ᮱ᳪ

2021-06-20 LWKXE6WDUහᏈጯӥᰁᬦጯ

2020-10-22 Ḓӻ3XEOLF5HOHDVHᇇ

2020-05-30 Githubՙପୌᒈ

2019-12-06 pg_exporter ՙପڠୌ

v100-beta1 ݎ2021-07-15

v001

2021-07-31 v100 GA

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

ᓕቘๅग़რහഝପ

ᔮᕹඪ೮ԯኞද

ፊഴᔮᕹଫአໜ

ๅग़ۑᚆଘ۸ݣ

Next Episode of Pigstyኞඪ೮ ᮱ᗟCitusṛݢአଘړᇆᵞᗭᕟ

ඪ೮ARMຝඪ೮ๅग़LinuxݎᤈᇇғSUSEҁ

DebianUbuntu۸ොໜKubernetes ඪ೮Kubernetes Operator

ඪ೮አ etcd դਠෆጱ consul ۑᚆႲےᐶᕚପପ᮱ᗟඪ೮

ᵞӾୗղ௩ොໜਞق୩۸ғӸ໒ཛྷୗ෭ಸಸᦄړຉ

ᵞ౮ཛྷୗݒๅොໜᵞ౮හഝᬢᑏොໜ

ๅग़ๅقᶎๅӫӱጱፊഴᶎๅग़ๅਫአጱӫଫአᵞ౮pgbager෭பړຉಸޞᵞ౮PEVᧃݢᥤ۸ړຉૡ

ᵞ౮ἓፋፊഴ blackbox_exporterᵞ౮ๅग़෭பᤉኞຽғmtailਠጱPostgreSQLᔮᕹፓ୯ၨᥦғPGCAT

ඪ೮᮱ᗟፊഴๅग़ᐿᔄጱრහഝପғRedisҁ

MongoDBMySQLhelliphellip

ᓕቘଘݣ amp CMDB

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

Open Source

让户能好数据库好数据库

提供种ldquo主可控rdquo的选择让中企业个户更有由选择的底

第时间获取反馈了解户的痛点痒点与爽点解决真正的问题

降低槛扩户群促进PostgreSQL态繁荣发展

THANKS

THANKS