IBM TotalStorage Enterprise Storage Server

ibm.com/redbooks

Front cover

IBM TotalStorageEnterprise Storage ServerPPRC Extended Distance

Gustavo CastetsBarry Mellish

Daniel LeplaideurMarcus Thordal

PPRC Extended Distance description and characteristics

Management and operation of PPRC Extended Distance

Uses and implementations of PPRC Extended Distance

International Technical Support Organization

IBM TotalStorage Enterprise Storage ServerPPRC Extended Distance

June 2002

SG24-6568-00

© Copyright International Business Machines Corporation 2002. All rights reserved.Note to U.S Government Users - Documentation related to restricted rights - Use, duplication or disclosure is subject to restrictions setforth in GSA ADP Schedule Contract with IBM Corp.

First Edition (June 2002)

This edition applies to the IBM TotalStorage Enterprise Storage Server — IBM 2105 model F. For other information you can see the Publications section of the IBM Announcement letter for the IBM TotalStorage Enterprise Storage Server and you can visit our Web site at:

http://www.storage.ibm.com

Comments may be addressed to:IBM Corporation, International Technical Support OrganizationDept. QXXE Building 80-E2650 Harry RoadSan Jose, California 95120-6099

When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you.

Take Note! Before using this information and the product it supports, be sure to read the general information in “Notices” on page xiii.

http://www.storage.ibm.com

Contents

Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiTrademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xvThe team that wrote this redbook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xvTerminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiNotice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiComments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii

Chapter 1. Data integrity and availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 What is disaster recovery? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Business objectives for disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Tiers of disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4 IBM TotalStorage solutions for disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Chapter 2. PPRC Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.1 PPRC Extended Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.2 PPRC Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.3 PPRC operation methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.4 Synchronous PPRC (PPRC-SYNC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.4.1 Data consistency at the recovery site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.4.2 Application availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.4.3 Write operations response time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.4.4 Distance factor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.4.5 PPRC-SYNC Positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.5 PPRC volume states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.6 Synchronous PPRC using static volumes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.6.1 How PPRC-SYNC can be used over long distances . . . . . . . . . . . . . . . . . . . . . . 142.6.2 Data consistency at the recovery site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.6.3 Application availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.6.4 Write operations response time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.6.5 Distance factor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.6.6 PPRC using static volumes — positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.6.7 PPRC with static volumes — implementation examples. . . . . . . . . . . . . . . . . . . . 18

2.7 PPRC Extended Distance (PPRC-XD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.7.1 How PPRC Extended Distance works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.7.2 Data consistency at the recovery site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.7.3 Application availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.7.4 Write operations response time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.7.5 Distance factor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.7.6 PPRC Extended Distance positioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.7.7 PPRC-XD implementations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Chapter 3. PPRC Extended Distance characteristics and management . . . . . . . . . . . 253.1 PPRC options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

© Copyright IBM Corp. 2002 iii

3.1.1 Basic characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.1.2 PiT Implementation basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3.2 PPRC-XD states change logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.3 Managing PPRC Extended Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

3.3.1 Establishing XD pairs using TSO commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.3.2 Catch-up (go-to-SYNC) using TSO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.3.3 CESTPAIR command parameter values. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343.3.4 Establishing XD pairs using the Web user interface . . . . . . . . . . . . . . . . . . . . . . . 343.3.5 Catch-up (go-to-SYNC) and suspend using the WUI . . . . . . . . . . . . . . . . . . . . . . 363.3.6 Web user interface options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

3.4 Querying PPRC-XD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373.4.1 CQUERY command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.4.2 rsQueryComplete command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.4.3 rsQuery command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.4.4 The Information Panel of the Web user interface . . . . . . . . . . . . . . . . . . . . . . . . . 40

Chapter 4. PPRC configuration and data consistency . . . . . . . . . . . . . . . . . . . . . . . . . 434.1 PPRC configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4.1.1 Establishing concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.1.2 Defining paths between two LSSs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.1.3 Displaying path information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.2 Building consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494.2.1 Consistency groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504.2.2 Defining consistency groups using TSO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514.2.3 Defining consistency groups using the Web user interface . . . . . . . . . . . . . . . . . 514.2.4 Freezing operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534.2.5 Freeze operation with TSO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544.2.6 Freeze operation with the Web user interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Chapter 5. PPRC Extended Distance implementation and use . . . . . . . . . . . . . . . . . . 575.1 PPRC-XD point-in-time consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585.2 PPRC-XD for application recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

5.2.1 Starting with a catch-up of the volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585.2.2 PPRC-XD PiT within the synchronous range . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625.2.3 Starting with the quiesce of the application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

5.3 PPRC-XD solutions and positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635.3.1 Functional positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635.3.2 Remote data migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655.3.3 Transmission of database logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 665.3.4 Off-site backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 665.3.5 Mixed environment (XD and SYNC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675.3.6 Application performance positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675.3.7 Initial establish for a large number of volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

5.4 PPRC-XD implementation considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685.4.1 More data can be PPRC protected . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 695.4.2 Consistency management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 695.4.3 Automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 695.4.4 Bandwidth management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

5.5 PPRC connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 705.5.1 PPRC supported distances. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 705.5.2 PPRC channel extender support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715.5.3 PPRC Dense Wave Division Multiplexor (DWDM) support. . . . . . . . . . . . . . . . . . 71

5.6 PPRC-XD requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

iv PPRC Extended Distance

Chapter 6. Using the WUI and the CLI to manage PPRC-XD. . . . . . . . . . . . . . . . . . . . . 736.1 ESS Specialist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 746.2 Creating ESS Copy Services tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 756.3 Tasks for a PiT backup with PPRC-XD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

6.3.1 Establish paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 776.3.2 Establish PPRC-XD pair — initial copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 776.3.3 Establish PPRC-SYNC pair — go-to-SYNC and suspend . . . . . . . . . . . . . . . . . . 786.3.4 Quiesce the application write activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 786.3.5 Re-synchronize PPRC pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 796.3.6 Freeze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 796.3.7 Resume application write activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 806.3.8 FlashCopy PPRC target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 806.3.9 Reestablish paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 816.3.10 Reestablish PPRC-XD pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 816.3.11 Additional tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

6.4 Using the CLI for a PiT backup with PPRC-XD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 826.4.1 Establish paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 836.4.2 Establish PPRC-XD pair — initial copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 846.4.3 Establish PPRC-SYNC pair — go-to-SYNC and suspend . . . . . . . . . . . . . . . . . . 856.4.4 Quiesce the application write activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 876.4.5 Re-synchronize PPRC pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 876.4.6 Freeze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 886.4.7 Resume application write activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 906.4.8 FlashCopy PPRC target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 906.4.9 Reestablish paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 916.4.10 Reestablish PPRC-XD pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 916.4.11 Consistency group created . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 926.4.12 Creating the script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

Appendix A. ESS Copy Services creation of tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95A.1 ESS Copy Services tasks: step-by-step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

A.1.1 Establish paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96A.1.2 Establish PPRC XD pair — initial copy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98A.1.3 Establish PPRC SYNC pair — go-to-SYNC and suspend . . . . . . . . . . . . . . . . . 102A.1.4 Quiesce the application write activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104A.1.5 Re-synchronize PPRC pair. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104A.1.6 Freeze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106A.1.7 Resume application write activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108A.1.8 FlashCopy PPRC target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108A.1.9 Reestablish paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111A.1.10 Reestablish PPRC XD pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111A.1.11 Withdraw FlashCopy pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113A.1.12 Suspend PPRC pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114A.1.13 Terminate PPRC pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115A.1.14 Consistency-group-created. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

A.2 Installing the CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118A.2.1 CLI supported systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118A.2.2 Installing procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

Appendix B. Additional examples of PiT and data migration solutions . . . . . . . . . . 123B.1 Data migration for a large data center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

B.1.1 Planning the migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124B.1.2 Preparing the volume pairs and dumping the data . . . . . . . . . . . . . . . . . . . . . . . 125B.1.3 Restoring the data and re-synchronizing the pairs . . . . . . . . . . . . . . . . . . . . . . . 126

Contents v

B.2 PiT backup solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128B.2.1 Considerations when using PPRC-XD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128B.2.2 Distance considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128B.2.3 Other planning considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129B.2.4 Impending disasters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Appendix C. PPRC-XD Managing in the TSO environment . . . . . . . . . . . . . . . . . . . . . 131C.1 Managing with TSO commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

C.1.1 PPRC-XD Initialization and monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132C.1.2 Catch-up transition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135C.1.3 Build consistency and freeze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137C.1.4 Resume application write activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137C.1.5 Save checkpoint and resume PPRC-XD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

C.2 Path failures scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

Appendix D. PPRC-XD operation for z/VM and VSE/ESA users . . . . . . . . . . . . . . . . . 145D.1 ICKDSF PPRC command functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146D.2 PPRCOPY ESTPATH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146D.3 PPRCOPY DELPATH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146D.4 PPRCOPY ESTPAIR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147D.5 PPRCOPY DELPAIR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147D.6 PPRCOPY SUSPEND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147D.7 PPRCOPY RECOVER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148D.8 PPRCOPY QUERY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

Appendix E. PPRC with static volumes implementations . . . . . . . . . . . . . . . . . . . . . . 149PPRC with static primary volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150Remote data migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150Transmission of database log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151Remote backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

Other resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155Referenced Web sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

IBM Redbooks collections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

vi PPRC Extended Distance

Figures

1-1 Disaster recovery components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21-2 Business objectives for disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31-3 Tiers for disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41-4 Disaster recovery solutions portfolio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52-1 ESS Peer-to-Peer Remote Copy Extended Distance (PPRC-XD) . . . . . . . . . . . . . . . 82-2 PPRC-SYNC mode of operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112-3 PPRC-SYNC positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122-4 Volume states in a PPRC environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132-5 PPRC-SYNC initial copy — duplex pending until duplex is reached . . . . . . . . . . . . . 142-6 PPRC-SYNC with static volumes — operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152-7 PPRC-SYNC implementation with static volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . 162-8 “PPRC-SYNC using static volumes” positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172-9 PPRC Extended Distance new duplex-pending XD volume state . . . . . . . . . . . . . . . 192-10 PPRC-XD — basic operation characteristics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202-11 PPRC-XD. Positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233-1 PPRC Extended Distance basic characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263-2 PPRC Extended Distance basic setup example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283-3 PPRC state change logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313-4 CESTPAIR command with new OPTION (XD/SYNC) parameter . . . . . . . . . . . . . . . 323-5 Establishing an XD initial copy pair with CESTPAIR . . . . . . . . . . . . . . . . . . . . . . . . . 333-6 Selecting the volume pair and launching the Task Wizard . . . . . . . . . . . . . . . . . . . . 343-7 Establishing the PPRC-XD pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353-8 Selecting PPRC-XD copy options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353-9 Catch-up (go-to-SYNC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363-10 Suspending as soon as the catch-up is finished . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373-11 Output information for a CQUERY on primary XD volume . . . . . . . . . . . . . . . . . . . . 383-12 rsQueryComplete reporting at shorter intervals (seconds) . . . . . . . . . . . . . . . . . . . . 393-13 rsQuery of an XD primary volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393-14 Getting the Information Panel output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403-15 Information Panel, out-of-sync cylinders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414-1 Establishing paths between LSSs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444-2 Path definition, configuration guidelines, and error handling . . . . . . . . . . . . . . . . . . . 464-3 Paths information before the paths are defined . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474-4 Paths information of valid paths. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484-5 Paths information for links not used. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494-6 Consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504-7 CGROUP parameter when establishing a path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514-8 Establishing a PPRC path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524-9 Establish PPRC path with consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524-10 Freeze and resume operations characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544-11 FREEZE and RUN options of the CGROUP command. . . . . . . . . . . . . . . . . . . . . . . 544-12 Freezing using the Logical Subsystems panel of the Web user interface . . . . . . . . . 555-1 Consistent PiT copy, starting with the catch-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595-2 Building global consistency considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615-3 Consistent PiT copy, waiting for application writes to quiesce. . . . . . . . . . . . . . . . . . 635-4 PPRC Extended Distance functional positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 645-5 Data center migration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655-6 Transmission of database logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

© Copyright IBM Corp. 2002 vii

5-7 PPRC-XD split mirror implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675-8 Flexible continuum of PPRC and PPRC-XD capabilities . . . . . . . . . . . . . . . . . . . . . . 685-9 Implementation considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 695-10 Connectivity, distance and support considerations . . . . . . . . . . . . . . . . . . . . . . . . . . 706-1 Selecting the ESS Copy Services interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 746-2 ESS Copy Services interface - Welcome panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 756-3 Source and target PPRC-XD volume pair icons . . . . . . . . . . . . . . . . . . . . . . . . . . . . 766-4 Establish paths without consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 776-5 Establish PPRC-XD — Copy entire volume — . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 776-6 Establish PPRC-SYNC pair — go-to-SYNC, and suspend . . . . . . . . . . . . . . . . . . . . 786-7 Re-synchronization of the pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 796-8 Freeze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 796-9 Suspend PPRC pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 806-10 FlashCopy of the PPRC target. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 816-11 Paths are reestablished . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 816-12 Reestablish PPRC-XD pair - incremental copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 826-13 rsExecuteTask EST_PATH_12_13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 836-14 rsExecuteTask EST_PPRC_XD_V1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 846-15 rsQuery PPRC source volume. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 846-16 rsQuery target volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 856-17 rsExecuteTask PPRC_SYN_SUSP_V1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 856-18 rsQueryComplete PPRC_SYN_SUSP_V1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 866-19 rsQuery PPRC source volume. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 866-20 rsQuery PPRC target volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 876-21 rsExecuteTask RESYN_PPRC_V1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 886-22 rsQueryComplete RESYN_PPRC_V1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 886-23 rsExecuteTask FREEZE_LSS12_13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 896-24 rsQuery PPRC source volume. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 896-25 rsQuery PPRC target volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 896-26 rsExecuteTask EST_FC_V1_W1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 906-27 rsQuery FlashCopy source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 906-28 rsQuery FlashCopy target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 916-29 rsExecuteTask EST_PATH_12_13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 916-30 rsExecuteTask REEST_PPRC_XD_V1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 926-31 rsExecuteTask CG_CRE_LSS12_13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93A-1 Establish paths. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96A-2 Path options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97A-3 Define task ‘EST_PATHS_V1’. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97A-4 Selecting source and target LSSs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98A-5 Selecting the volume pair. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99A-6 Launching the Task Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100A-7 Establish Extended Distance PPRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100A-8 Copy entire volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101A-9 Define task ‘EST_PPRC_XD_V1’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102A-10 Establish synchronous PPRC copy pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103A-11 Suspend PPRC after establish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103A-12 Define task ‘PPRC_SYN_SUSP_V1’. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104A-13 Re-sync synchronous PPRC copy pair.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105A-14 Re-SYNC, copy out-of-sync cylinders only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105A-15 Define task RESYN_PPRC_V1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106A-16 Freeze PPRC consistency group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107A-17 Define task ‘FREEZE_LSS12_13’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108A-18 Selecting source and target FlashCopy volumes. . . . . . . . . . . . . . . . . . . . . . . . . . . 109

viii PPRC Extended Distance

A-19 Establish FlashCopy pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109A-20 Selecting FlashCopy options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110A-21 Define Task ‘EST_FC_V1_W1’.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110A-22 Establish Extended Distance PPRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111A-23 Copy out-of-sync tracks only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112A-24 Define task ‘REEST_PPRC_XD_V1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112A-25 Withdraw FlashCopy pair. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113A-26 Define task ‘WDRAW_FC_V1_W1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113A-27 Suspend PPRC copy pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114A-28 Specify LSS where task is scheduled. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114A-29 Define task ‘SUSP_PPRC_V1_S. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115A-30 Terminate PPRC pair. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115A-31 Specify LSS where task is scheduled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116A-32 Define task ‘TERM_PAIR_V1_S . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116A-33 PPRC Consistency Created. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117A-34 Define task ‘CG_CRE_LSS12_13’. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118A-35 Executing the CLI setup file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119A-36 Preparing the InstallShield Wizard. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119A-37 IBM 2105 CLI InstallShield Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120A-38 Choosing the destination folder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121A-39 Installing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121A-40 CLI has completed the installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122A-41 Verifying the installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122B-1 Large z/OS center remote data migration (1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125B-2 Large z/OS center remote data migration (2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127B-3 PiT backup with PPRC XD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128C-1 TSO commands for establishing paths and pairs . . . . . . . . . . . . . . . . . . . . . . . . . . 132C-2 CQUERY Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133C-3 CQUERY Secondary Volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134C-4 CQUERY on primary XD volume - Out-of-sync tracks 19471 . . . . . . . . . . . . . . . . . 134C-5 CQUERY on primary XD volume. Out-of-sync tracks 3549. . . . . . . . . . . . . . . . . . . 135C-6 Catch-up using TSO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136C-7 CESTPAIR to command the go-to-SYNC transition (catch-up) . . . . . . . . . . . . . . . . 136C-8 Build global consistency at the recovery site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137C-9 Resuming write updates onto primary volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137C-10 Save secondary copy and resume PPRC-XD copy. . . . . . . . . . . . . . . . . . . . . . . . . 138C-11 z/OS DFSMS/dss FlashCopy invocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138C-12 z/OS DFSMS/dss FlashCopy Execution (1) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139C-13 z/OS DFSMS/dss FlashCopy Execution (2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140C-14 Initial path status, with two paths already interrupted . . . . . . . . . . . . . . . . . . . . . . . 141C-15 The third link is interrupted. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141C-16 CQUERY path status report after third link was interrupted . . . . . . . . . . . . . . . . . . 142C-17 Interrupting the last path — link failure messages . . . . . . . . . . . . . . . . . . . . . . . . . . 143C-18 Final status path report — none of the paths are operative . . . . . . . . . . . . . . . . . . 144E-1 PPRC-SYNC with static volumes for remote data migration . . . . . . . . . . . . . . . . . . 150E-2 PPRC-SYNC with static volumes for transmission of database log files . . . . . . . . . 151E-3 PPRC-SYNC with static volumes for remote backup — establish initial copy . . . . . 152E-4 PPRC-SYNC with static volumes — suspend pairs after initial copy. . . . . . . . . . . . 152E-5 PPRC keeps track of incremental modifications while volumes suspended . . . . . . 153E-6 PPRC-SYNC with static volumes — incremental copy in subsequent backups . . . 154

Figures ix

x PPRC Extended Distance

Tables

3-1 CESTPAIR parameter values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343-2 Web user interface options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

© Copyright IBM Corp. 2002 xi

xii PPRC Extended Distance

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces.

© Copyright IBM Corp. 2002 xiii

TrademarksThe following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:

AIX®AS/400®CICS®CUA®DB2®DFS™DFSMS/MVS®DFSMS/VM®DFSMSdfp™DFSMSdss™DFSMShsm™DFSORT™DYNIX®DYNIX/ptx®Enterprise Storage Server™Enterprise Systems Connection® Architecture®ESCON®FICON™

FlashCopy™IBM®IMS™IMS/ESA®iSeries™Magstar®MVS™NUMA-Q®NUMACenter™OS/390®OS/400®Perform™PR/SM™Predictive Failure Analysis®PR/SM™pSeries™RACF®RAMAC®

Redbooks™Redbooks(logo)™RETAIN®RMF™RS/6000®S/390®Seascape®Sequent®SP™StorWatch™System/390®Tivoli®TotalStorage™VM/ESA®VSE/ESA™xSeries™z/OS™z/VM™zSeries™

The following terms are trademarks of International Business Machines Corporation and Lotus Development Corporation in the United States, other countries, or both:

Lotus® Notes® Word Pro®

The following terms are trademarks of other companies:

ActionMedia, LANDesk, MMX, Pentium and ProShare are trademarks of Intel Corporation in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

C-bus is a trademark of Corollary, Inc. in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

SET, SET Secure Electronic Transaction, and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC.

Other company, product, and service names may be trademarks or service marks of others.

xiv IBM TotalStorage Enterprise Storage Server PPRC Extended Distance

Preface

This IBM Redbook describes the characteristics of the new PPRC Extended Distance remote copy function of the IBM TotalStorage Enterprise Storage Server.

In this redbook you will find information on how to operate and manage this new remote copy function of the ESS. It also explains how to use and implement PPRC Extended Distance for remote data copy, off-site backup, remote data migration, and disaster recovery solutions.

PRC Extended Distance is supported on the IBM TotalStorage Enterprise Storage Server models F and newer (please refer to 5.6, “PPRC-XD requirements” on page 71 for detailed information).

The team that wrote this redbookThis redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, San Jose Center.

Gustavo Castets is a Project Leader at the International Technical Support Organization, San Jose Center. He has co-authored three previous redbooks and teaches IBM classes worldwide on areas of Disk Storage Systems. Before joining the ITSO, Gustavo worked in System Sales as Field Technical Support Specialist. Gustavo has worked for more than 22 years in many IT areas, in Buenos Aires, for IBM Argentina.

Barry Mellish is a Project Leader at the International Technical Support Organization, San Jose Center. He has coauthored nine previous redbooks and has taught many classes on storage subsytems.He joined IBM UK 18 years ago, and before joining the ITSO, he worked as a Senior Storage Specialist on the Disk Expert team in EMEA.

Daniel Leplaideur joined IBM France in 1967 as a mathematician and statistician to develop scientific subsystems and software packages. He has worked as a System Engineer in Networking and Large Systems in the Services, in Branch Offices, and in Field Support Centers. He is now in the European Mainz Advanced Technical Support for ESS and Disaster Recovery. His job mainly deals with defining and implementing solutions for various very complex environments.

Marcus Thordal is an Advisory IT Specialist at IBM Global Services, Denmark, and has been with IBM for more than 4 years. He is MCSE certified, and his areas of expertise include supporting Open Systems storage solution and Microsoft BackOffice products, in particular MS Cluster and MS SQL Server. He holds a BSc. Eng. degree from The Technical University of Denmark, and has co-authored two redbooks on Tivoli Data Protection for MS SQL Server and MS Exchange Server.

© Copyright IBM Corp. 2002 xv

The team that wrote this book: Gustavo, Marcus, Daniel, Barry

Thanks to the following people for their diligence in reviewing this manual:

William Micka, from IBM Tucson, Subsystem ArchitectureHyeong Ge Park, from IBM Japan, Storage SpecialistKaren Roberts, from IBM San Jose, SSG Open Systems ProductsDavid Sacks, from Chicago, Storage Sales SpecialistJohn Sing, from IBM San Jose, ESS Copy Services World Wide MarketingGail Spear, from Tucson, Microcode Development

Special thanks to William Micka and Gail Spear for helping us understand how this new function works, and how it will better benefit their users.

Thanks to John Sing who created the material used in Chapter 1 and Appendix E — and also reviewed our early drafts.

Thanks to Bill Worthington, from IBM Advanced Technical Support in San Jose, who wrote the support document for z/VM and VSE/ESA users —included in Appendix D.

Thanks also to Yvonne Lyon of the IBM ITSO, San Jose, for technical editing and formatting.

Thanks to the following people for their invaluable contributions to this project:

Laura Balstad, from IBM TucsonWerner Bauer, from IBM GermanyLinda Benhase, from IBM TucsonJerry Boyle, from IBM TucsonIsabelle Claverie-Berge, from IBM FrancePierre Couvidat, from IBM FranceHoward Disney, from IBM TucsonTom Guinane, from IBM San JoseKenneth Hallam, fromIBM TucsonBob Kern, from IBM TucsonLee LaFrese, from IBM TucsonDenise Luzar, from IBM TucsonGreg McBride, from IBM TucsonLinda McKinnon, from IBM TucsonCindy Regens, from IBM TucsonJames Teng, from IBM Santa Teresa

xvi PPRC Extended Distance

Amy Therrien, from IBM San JoseJohn Thompson, from IBM TucsonDavid Valverde, from IBM TucsonSony E. Williams, from IBM Tucson

TerminologyThroughout our book, we used these new terms and acronyms:

PPRC-XD PPRC Extended Distance. This term is used to refer to the new PPRC Extended Distance remote copy function of the IBM TotalStorage Enterprise Storage Server.

XD This is the short form of referring to PPRC Extended Distance. Most frequently you will find it used when referring to pairs of volumes established in a PPRC Extended Distance relationship — XD pair.

PPRC-SYNC This term is used to refer to the traditional synchronous PPRC remote copy function of the IBM TotalStorage Enterprise Storage Server.

SYNC This is the short form of referring to the traditional synchronous PPRC. Most frequently, you will find it used when referring to pairs of volumes established in a PPRC synchronous relationship — SYNC pair.

catch-up This term is used to refer to the transition that occurs to the volume pairs, when they go from an XD out-of-sync condition to a full synchronous condition. This transition is explained in detail later in the book.

go-to-sync This term refers to the commanded way of requesting from PPRC-XD to do a catch-up.

duplex pending XD This is the new state that you can find the volume pairs, when the pairs are established in a PPRC Extended Distance relationship.

NoticeThis publication is intended to help IT professionals who are implementing remote backup, data migration, or disaster recovery solutions using the new PPRC Extended Distance remote copy function of the IBM TotalStorage Enterprise Storage Server.

The information in this publication is not intended as the specification of any programming interfaces that are provided by IBM or third parties. See the PUBLICATIONS section of the IBM Programming Announcement for the IBM TotalStorage Enterprise Storage Server —Model F — for more information about what publications are considered to be product documentation.

Preface xvii

Comments welcomeYour comments are important to us!

We want our Redbooks to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways:

� Use the online Contact us review redbook form found at:

ibm.com/redbooks

� Send your comments in an Internet note to:

[email protected]

� Mail your comments to the address on page ii.

xviii PPRC Extended Distance

http://www.redbooks.ibm.com/

http://www.ibm.com/redbooks/


http://www.redbooks.ibm.com/contacts.html

Chapter 1. Data integrity and availability

In this chapter we briefly review some introductory considerations about the strategies for data integrity and availability, and where the new PPRC Extended Distance remote copy solution is positioned.

When planning for disaster recovery solutions, we suggest that you refer to the IBM Redbook IBM TotalStorage Solutions for Disaster Recovery, SG24-6547.

1

© Copyright IBM Corp. 2002 1

1.1 What is disaster recovery?How would a shutdown of your IT system affect your business? What about a site disaster? Is your business-critical processing and data protected from a site disaster? Do you put off system maintenance and upgrades to avoid system downtime? Consider Figure 1-1.

Figure 1-1 Disaster recovery components

In today's highly competitive e-business world, outages can have a devastating impact on a business — they can even mean its demise. IBM suggests that disaster recovery is much more than just mirroring the disk data; rather, as Figure 1-1 illustrates, disaster recovery is a total business continuance solution comprising five major IT components:

� Servers� Storage � Software and automation � Networking � Services for integration

A disaster recovery implementation that only covers the storage component will leave the organization open to significant additional costs and time requirements if the other components are not covered.

What is disaster recovery?

Five components of disaster recovery:1. Servers2. Storage3. Software and automation 4. Networking5. Services

Operations StaffOperations Staff

Applications Staff

Network Staff

Management Control

Physical FaciltiesTelecom Network

Data

Operating SystemApplications

SolarisHP-UXWinNT/2000xSeriespSeriesiSeries

zSeries

zSeries

SolarisHP-UXWinNT/2000xSeriespSeriesiSeries

2 PPRC Extended Distance

1.2 Business objectives for disaster recoveryAs shown in Figure 1-2, to design a cost-effective solution, we start determining the following objectives by application or business line:

� Recovery Time Objective (RTO): What is the business cost-justified elapsed time to recovery?

� Recovery Point Objective (RPO): When the Recovery Time Objective is met, what amount of data is needed to be recreated?

� Network Recovery Objective (NRO): How long does it take to switch the entire network over to the backup data center?

Figure 1-2 Business objectives for disaster recovery

These first two objectives (RTO and RPO) can often be balanced against each other to optimize the cost/benefit ratio. When the third objective (NRO) comes into play, networking issues will come into consideration — for example: there is no need to purchase a 30 minute RTO solution if the network provider requires 2 hours to switch the network.

Planned versus unplanned outagesPlanned outages are just as effective at removing service from the end users as unplanned outages — and they are much more frequent. Yet, typically, disaster recovery solution cost justification is attempted based on the unplanned outage cost alone. So a realistic return of investment analysis should always consider the unplanned outages effects.

Recovery Time Objective (RTO)How long can you afford to be without your systems?

Recovery Point Objective (RPO)When it is recovered, how much data can you afford to recreate?

Network Recovery Objective (NRO)How long will it take to switch over the network?

Determine telecom line costs at various bandwidths.

What is your cost / recovery time curve:If I spend a little more, how much faster is disaster recovery?

If I spend a little less, how much slower is disaster recovery?

Determining the cost vs. RTO recovery curve is the key to selecting Determining the cost vs. RTO recovery curve is the key to selecting proper solution(s)proper solution(s)

Backup Application FailureApplication Processing

RPO

Chapter 1. Data integrity and availability 3

1.3 Tiers of disaster recoveryAnd finally, it is not only one RTO/RPO set that you are looking for. When looking for a solution, you will always want to know how much faster/slower the solution will be if you invest a little more or less. This is the primary business issue, which leads us to the tiers consideration (refer to Figure 1-3).

Figure 1-3 Tiers for disaster recovery

The chart in Figure 1-3 is a standard disaster recovery industry tier chart, showing that for varying types of RTO/RPO combinations, there are a variety of technologies.

In general, there are three main bands of application criticality. Within each band, there are tiers. The tiers vary as follows: tier 0 (no recovery capability), tier 2,3, (tape intensive recovery methods), tier 4 (disk mirroring or disk point-in-time facilities), tier 5 (software/ database specific recovery, tier 6 — near zero or zero data loss disk mirroring, and finally, the newest tier 7 — which is a completely automated self-managed take of servers, storage, software, and automation, networking all integrated with services into the user’s application.

Hardware failure data consistency versus transaction data consistencyThere are two levels of data consistency:

� Hardware power failure: An example might be if the primary machine room were to experience an instantaneous power failure.

� Transaction consistency: Application or database level in which multiple transactions are interlocked, and if any operation is unsuccessful, then the application will back out the corresponding companion operations, thus preserving data consistency.

Therefore, software recovery on top of hardware recovery is essential and cannot be avoided.

Time to recover Tiers based on SHARE definitions

15 Min. 1-4 Hr.. 4 -8 Hr.. 8-12 Hr.. 12-16 Hr.. 24 Hr.. Days

Tier 4 - Batch/Online database shadowing and journaling, repetitive PiT copies, fuzzy copy disk mirroring, TSM-DRM

Tier 3 - Electronic vaulting, TSM

Tier 1 - PTAM*

Tier 2 - PTAM, Hot Site, TSMPoint-in-time backup

Active secondary site

Dedicated remote hot site

Co

st

*PTAM = Pickup Truck Access Method

Tier 7 - Near zero or zero data loss: highly automated takeover on a complex-wide or business-wide basis, using remote disk mirroring

Tier 6 - Near zero or zero data loss remote disk mirroring helping with data integrity and data consistency

Tier 5 - software two site, two phase commit (transaction integrity); or repetitive PiT copies with small data loss

Best D/R practice is blend tiers of solutions in order to maximize application coverage at lowest possible cost . Especially from a cost standpoint, one size doesn't fit all applications.


1.4 IBM TotalStorage solutions for disaster recoveryAs already discussed in 1.1, “What is disaster recovery?” on page 2, the disaster recovery solution is comprised of five major IT components — one of which is the storage component. The value of IBM is that it blends the appropriate mix of solutions for the whole set of requirements.

To the IBM TotalStorage solutions portfolio, a new solution is added — the PPRC Extended Distance remote copy function of the IBM TotalStorage Enterprise Storage Server. Figure 1-4 illustrates where this new disaster recovery TotalStorage solution is positioned.

Figure 1-4 Disaster recovery solutions portfolio

Middle-size solutions FAStT Storage Servers (Fibre Channel disks)

FlashCopy - Tier 4 Peer-to-Peer copy (10Km) - Tier 6

Enterprise solutions for Open Systems and zSeries IBM TotalStorage Enterprise Storage Server

FlashCopy for point-in-time copy - Tier 4PPRC-XD for remote copy and data migration - Tier 4

Non-synchronous long or short distanceIP connection capability (by channel extender vendors)

PPRC-SYNC - Tier 6Synchronous (supported up to 103 Km)IP connection capability (by channel extender vendors)

Enterprise solutions for zSeries IBM TotalStorage Enterprise Storage Server

Asynchronous copy with data integrity - Tier 6

SANsymphony

Servers

Enterprise Storage Serve rs SANsy mphony

Servers

Enterpri se Storage Servers

System DataMover

TELECOMMUNICATIONS

Parallel Sysplex

CF

CF

Parallel Sysplex

CF

CF

System DataMover(s)

System DataMover(s)

System DataMover(s)

System DataMover(s)

Snapshot /FlashCopy

FlashCopy

Chapter 1. Data integrity and availability 5

Chapter 2. PPRC Introduction

Peer-to-Peer Remote Copy (PPRC) is a remote copy hardware based solution of the IBM TotalStorage Enterprise Storage Server that can be used in different ways — thus giving you more flexibility when addressing the implementation of your data integrity and availability solution.

In this chapter, we:

� Review the characteristics of the traditional synchronous PPRC remote copy function.

� Describe the additional capabilities of the synchronous PPRC remote copy function — when using static volumes.

� Introduce the new non-synchronous PPRC Extended Distance remote copy function.

� Position the different ways of using the PPRC remote copy function.

2


2.1 PPRC Extended DistancePeer-to Peer Remote Copy Extended Distance (PPRC-XD) brings new flexibility to the IBM TotalStorage Enterprise Storage Server and PPRC, with the introduction of a non-synchronous long distance copy option ideally suited for data migration, off-site backups, and remote transmission of database logs. It can also be used for application recovery solutions based on periodic PiT (point-in-time) backups —if the application can briefly be quiesced.

PPRC Extended Distance can operate at very long distances — continental distances, well beyond the 103 km supported for PPRC synchronous transmissions — with minimal impact on the application, and the distance is only limited by the network and channel extender technology capabilities (see Figure 2-1).

Figure 2-1 ESS Peer-to-Peer Remote Copy Extended Distance (PPRC-XD)

This redbook will allow you to understand how this new PPRC option works, and how you can manage it. You will also find examples and recommendations, for the implementation of data integrity and availability solutions — based on PPRC Extended Distance.

2.2 PPRC OverviewPeer-to-Peer Remote Copy (PPRC) is an ESS function that allows the shadowing of application system data from one site (usually called the application site) to a second site (called the recovery site). The logical volumes that hold the data in the ESS at the application site are called primary volumes, and the corresponding logical volumes that hold the mirrored data at the recovery site are called secondary volumes. The connection between the primary and the secondary ESSs is done using ESCON links.

Hardware-based solution for all serversPPRC is a hardware-based ESS shadowing solution that can be used on environments with Intel-based servers, UNIX based servers, and IBM zSeries and iSeries servers.

channelextender

non-synchronous PPRC

non-synchronous PPRC

o

ver contin

ental distances

o

ver contin

ental distances

PPRC-XDPPRC-XD

PPRC-XDPPRC-XD


For further information on PPRC, we refer the reader to the following redbooks: Implementing ESS Copy Services on UNIX and Windows NT/2000, SG24-5757; and Implementing ESS Copy Services on S/390, SG24-5680.

2.3 PPRC operation methodsPPRC is an optional remote copy function feature that can be configured when ordering an ESS. Primary and secondary ESSs must have this priced feature installed.

When this function is installed, there are three different ways of using it:

� Synchronous operation (PPRC-SYNC): This synchronously mirrors the updates done to the primary volumes. Can be used in distances of up to 103 km (an RPQ has to be submitted if slighter longer distances need to be implemented).

� Synchronous operation using primary static volumes: This can be used to move or copy data at very long distances using channel extenders.

� Extended Distance operation (PPRC-XD): This operates non-synchronously and can be used over continental distances, with excellent application performance. When implementing this solution over long distances, you need to use channel extenders.

In the following sections we review the synchronous method, then describe the characteristics of the new extended distance method.

Managing PPRC operationsTo manage and control PPRC —in either the PPRC-SYNC or the PPRC-XD ways — the ESS provides an ESS Copy Services Web user interface (WUI) and a ESS Command Line interface (CLI). The CLI is supported only on selected open systems servers. For the list of supported servers, please refer to:

http://www.storage.ibm.com/hardsoft/products/ess/supserver_summary_open.html

For detailed information on using the WUI and the CLI to manage the PPRC operations, the reader should refer to the following publications: IBM TotalStorage Enterprise Storge Server Web Interface User’s Guide, SC26-7346; and IBM TotalStorage Enterprise Storage Server Copy Services Command-Line Interface Reference, SC26-7434.

Additionally, for the IBM zSeries servers, PPRC can also be managed using the PPRCOPY commands of the ICKDSF utility and — exclusively for z/OS operating systems — the much more powerful TSO commands for PPRC.

For detailed information on using the TSO commands to manage the PPRC operations, the reader should refer to the following publication: z/OS DFSMS Advanced Copy Services, SC35-0428.

ICKDSFPPRC Extended Distance cannot be managed by the ICKDSF utility alone, as certain PPRC-XD parameters can only be issued from TSO commands, the ESS Copy Services Web user interface (WUI), or the ESS Copy Services command line interface (CLI).

Note: PPRC Extended Distance is supported on the IBM TotalStorage Enterprise Storage Server models F — or newer. For E models, they must first be upgraded to F models in order to install the PPRC Extended Distance microcode. For additional information, please refer to 5.6, “PPRC-XD requirements” on page 71.

Chapter 2. PPRC Introduction 9

http://www.storage.ibm.com/hardsoft/products/ess/supserver_summary_open.html

Current PPRC users of ICKDSF will find in Appendix D, “PPRC-XD operation for z/VM and VSE/ESA users” on page 145, a useful aid for transitioning to the WUI and the CLI, for PPRC Extended Distance operation.

In the following chapters we describe the new options available in the TSO commands and in the ESS Copy Services Web user interface and Command Line Interface, which support the PPRC-XD operation. We will be also using these new options in the examples that are presented in the following chapters.

2.4 Synchronous PPRC (PPRC-SYNC)In the synchronous type of operation, the updates done to the application site primary volumes are synchronously shadowed onto the secondary volumes at the recovery site. Because this is a synchronous solution, write updates are ensured on both copies (primary and secondary) before the write is considered to be completed for the application.

In Figure 2-2 you can see the sequence of a write operation when operating PPRC in synchronous mode (PPRC-SYNC).

2.4.1 Data consistency at the recovery siteBecause in PPRC-SYNC operation the application does not get the “write complete” condition until the update is synchronously done in both the primary and the secondary volumes (see Figure 2-2), then — from the application perspective — the data at the recovery site secondary volumes is real time data always consistent with the data at the primary volumes.

One remarkable implication of this characteristic is that, in normal PPRC-SYNC operation, dependent writes are applied on the secondary volumes in the same sequence as they are applied in the primary volumes. This is very important from an application consistency perspective at the time of the recovery.

2.4.2 Application availabilityPPRC-SYNC can provide continuous data consistency at the recovery site, without needing to periodically interrupt the application to build consistency checkpoints. From the application perspective this is a non-disruptive way of always having valid data at the recovery location.

2.4.3 Write operations response timePPRC-SYNC increases the response time as compared to a non-PPRC write operation, and this is inherent to the synchronous operation. The overhead comes from the additional steps (steps 3 and 4 in Figure 2-2) that are executed before the write operation is signaled as completed to the application.

Also this PPRC activity between the application site and the recovery site (steps 2 and 3 in Figure 2-2), physically consists of light signals that travel through the fiber links that connect the sites. And because the light needs time to travel, then this overhead — on the response time of the application write operations — will increase proportionally with the distance between sites. So the distance affects the applications’ write response time.


Figure 2-2 PPRC-SYNC mode of operation

2.4.4 Distance factorThe intensity of the light decreases when the light passes through the fiber link — so the longer the fiber, the lower the light intensity. This light intensity attenuation results in distance limitations of the physical link, because the receiver needs a minimum level of light intensity to correctly detect the signals.

The maximum supported distance for synchronous PPRC is 103 km (slightly longer distances may be possible, but an RPQ must be submitted). This distance can be achieved using DWDM (Dense Wave Division Multiplexers) connecting through dark fibers, or using channel extenders over WAN lines. Also, switches or directors can be used for shorter distances.

Please refer to 5.5, “PPRC connectivity” on page 70 for additional distance and connectivity considerations.

2.4.5 PPRC-SYNC PositioningAs summarized in Figure 2-3, synchronous PPRC provides continuous real time consistent data at the secondary site — without any disruptions to the application at the primary site. This gives maximum application availability and minimum recovery times. PPRC-SYNC has a distance limitation of 103 km, and increases the response time of the applications’ write operations.

Recovery siteESS

Application siteESS

secondaryvolume

primaryvolume

Application gets"write completed"Write data

to primary

fiber links

Write data tosecondary

Secondary reports"write completed"

2

3

41

For the application, the write operation is completed once the updates are donein both the primary and the secondaryvolumes

Synchronous mode


Figure 2-3 PPRC-SYNC positioning

Synchronous PPRC is the solution for the environments with non-stoppable applications (the ones that tolerate zero disruption) and with an RPO — recovery point objective (that is the amount of data to be recreated during a recovery) — of zero data loss.

2.5 PPRC volume statesPPRC environments, as compared to non-PPRC environments, bring a new element into consideration: the state in which volumes may be. In this section we review the states in which we may find PPRC volume pairs (see Figure 2-4).

PPRC-SYNC Positioning

The mirrored data is always a consistent real time shadow of the primary data.There is no need to quiesce the primary application in order to maintain the consistency on the mirrored data.This can be used for distances of up to 103 km (an RPQ can be submitted for slightly longer distances).The synchronous way of operation affects the response time of the application write activity. This overhead proportionally increases with the distance.

PPRC-SYNC is the recommended solution for environmentsPPRC-SYNC is the recommended solution for environmentswhere the primary application tolerates no disruption, andwhere the primary application tolerates no disruption, andwhere the recovery point objective (RPO) is zero data losswhere the recovery point objective (RPO) is zero data loss

Note: In this section we describe the traditional PPRC volume states. Later, in 2.7.1, “How PPRC Extended Distance works” on page 18, we present a new volume state, associated with the PPRC Extended Distance method of operation.


Figure 2-4 Volume states in a PPRC environment

As Figure 2-4 explains, at any given time a volume can be in one of the following states:

Simplex The initial state of a volume. A PPRC volume pair relationship has not been established yet between the primary and the secondary volumes.

Pending The initial state of a defined PPRC-SYNC volume pair relationship, when the initial copy of the primary volume to the secondary volume is happening. This state also is found when a PPRC-SYNC volume pair is re-synchronized after it was suspended. During the pending period the volume pair is not in synchronization and PPRC is copying tracks from the primary to the secondary volume.

Duplex The state of a PPRC-SYNC volume pair after PPRC has fully completed the copy operation of the primary volume onto the secondary volume. At this moment the volume pair is in synchronization and all write updates to the primary volume are synchronously applied onto the secondary volume as described in Figure 2-2.

Suspended In this state of the PPRC pair, the writes to the primary volume are not mirrored onto the secondary volume. The secondary volume becomes out of synchronization. During this time PPRC keeps a bit map record of the changed tracks in the primary volume. Later, when the volumes are re-synchronized, only the tracks that were updated will be copied.

A PPRC volume pair will go into suspended state, for instance, when the primary ESS cannot complete a write operation to the recovery system ESS. Also the operator can suspend pairs by command.

Application Site Recovery Site

PrimaryVolumes

SecondaryVolumes

Volume State Synchronization

DuplexDuplex

SimplexSimplex

SuspendedSuspendedDuplexDuplex

DuplexDuplexPendingPending

Data Copy

PPRC pair issuspended,volumes are nomore in sync

Volumes have noPPRC relationship

PPRC pair isestablished, withvolumes in fullsynchronization

Not in sync, butthe PPRC pair isestablished

0

0

10

1

0

0

0

10

0

0

0

1


2.6 Synchronous PPRC using static volumesThere is a particular implementation of PRPC-SYNC that optionally allows you to use it over long distances — beyond the supported 103 km — while taking advantage of the excellent throughput of PPRC when doing an initial copy or a re-synchronization of the volume pairs.

This powerful way of PPRC data copying, combined with the latest microcode improvements in the supported channel extenders (refer to 5.5.2, “PPRC channel extender support” on page 71) makes the static volumes implementation a recommended solution for data migration, for data copy, and for remote backup over very long distances.

This PPRC-SYNC implementation can also be used for disaster recovery solutions over long distances when:

� The applications in consideration can be quiesced.� Solutions are based on switching database log files, and transmitting when inactive.

Static volumeA static volume is a volume that is not receiving any write updates. The very particular use of PPRC-SYNC over long distances requires that the primary volumes must be static volumes.

2.6.1 How PPRC-SYNC can be used over long distancesWhile PPRC is doing an initial copy of a primary volume, for a just established PPRC pair; or while doing an incremental copy, for the re-synchronization of a suspended PPRC pair; the volume pair is in the duplex pending state. While in this state, the copy operation, between primaries and secondaries, is managed by PPRC in track sets. This is a very powerful throughput-oriented method of mirroring. This way of operating — while in duplex pending state, until duplex state is reached — is illustrated in Figure 2-5 for the initial copy of a PPRC-SYNC pair just established.

Figure 2-5 PPRC-SYNC initial copy — duplex pending until duplex is reached

SecondaryVolume

All Trackscopied

0

0

10

1

0

00

100

00

1

1-a. At initial copy, PPRC does a pass across the volume copying all the tracks

3. Now the volume pair is in full duplex mode and all the write updates are mirrored synchronously

0

0

10

1

000

10

0

0

0

1

2-b. When this second step is started, and from then on, any write update on an already copied track is mirrored synchronously as when in full duplex mode

1-b. A bit-map is kept to check for already copied tracks that receive an update before the intial copy pass is finished

PrimaryVolume

Updates areMirroredSynchronously

SecondaryVolume

All Trackscopied

0

0

10

1

0

00

100

00

1

0

0

10

1

000

10

0

0

0

1

PrimaryVolume

Only Changed Tracks Copied

Updates areMirroredSynchronously

2-a. A second pass is done copying just the updated tracks that were checked in the bit-map


For the incremental copy — when PPRC does a re-synchronization (RESYNC) of a suspended pair—, PPRC operates in a similar manner as described in Figure 2-5 — except for phase 1 (the initial copy of the full set of primary volume tracks) that is not executed in this situation. For a re-synchronization, PPRC only sends the tracks that have been updated while the PPRC pair of volumes were suspended — step 2 in Figure 2-5.

The particular implementation of PPRC-SYNC using static volumes — valid for long distances —, consists of a particular setup of these two operations (refer to Figure 2-6):

� Doing an initial copy of a just established PPRC pair, and suspending the pair when the duplex state is reached — this is steps 1 and 2 of Figure 2-5.

� Doing a re-synchronization (RESYNC) of a suspended pair, and again suspending when the duplex state is reached — this is step 2 of Figure 2-5.

Figure 2-6 PPRC-SYNC with static volumes — operation

Because this powerful throughput-oriented implementation of PPRC-SYNC using static volumes can be used over very long distances — well beyond the synchronous 103 km supported distance — it is mandatory that no synchronous copies occur at all. To comply with this, then the primary volumes must be static. And the volumes must be suspended as soon as the duplex state is reached — before the write activity commences. Figure 2-6 summarizes these considerations.

Note: Copying only the updated tracks — when in duplex pending state — is a very throughput-efficient way of having PPRC do the mirroring. In fact, PPRC uses many more sophisticated algorithms — for example, if changed data is in the cache, then PPRC sends only the changed sectors. These details are not covered because they are beyond the scope of this document.

PPRC-SYNC with static volumes working in the duplex pending zone

The operation will start by either:Executing an initial copy of a just established volume pairOr resynchronizing a previously suspended volume pair

The operation will end by:Suspending (or terminating) the pairs when the copy is finished and the volumes are fully synchronized (duplex state)

The volumes must be static during the operation


Figure 2-7 PPRC-SYNC implementation with static volumes

2.6.2 Data consistency at the recovery siteWith the “PPRC-SYNC with static volumes” implementation, you will automatically achieve application global consistency across the application’s set of secondary volumes, because the copy operation is done when the primary set of volumes are static — the application was quiesced.

For example, at an ad-hoc application checkpoint, the application writes are quiesced, so the application set of primary volumes becomes static. Then the PPRC copy operation is done. When the copy is completed (volumes reach the duplex state), then the volumes are immediately suspended so the application writes can be resumed. This way you will be able to get a point-in-time (PiT) version of the data — with application global consistency across all the volumes — at the recovery site.

When you want to recover an application using the remote copied data, you must consider that the copied data will not be a real time version of the current data at the application site. The recovery data will not be reflecting the updates that occurred since the last PPRC copy operation.

In Appendix E, “PPRC with static volumes implementations” on page 149, you will find examples of implementations of this particular solution using PPRC-SYNC.

2.6.3 Application availabilityTo carry on a PPRC copy operation using static volumes, you will be quiescing the application writes at the primary site, in order to have no more updates upon the primary set of application volumes. Also, with the application write activity quiesced, you are automatically getting consistency on your copied data.

A volume pair will be in duplex pending state while:The initial copy of a just established PPRC pair is proceeding

The resynchronization of a suspended PPRC pair is proceeding

While in duplex pending state, the copy operation is done in a very powerful throughput-oriented mannerThis solution can be implemented over very long distances:

Beyond synchronous 103 Km supported distance

Only limited by channel extender and networking capabilities

Synchronous operations cannot occur over long distancesTo avoid synchronous updates, you will be:

Working with primary static volumes (no write updates over the primary volumes)

Suspend the PPRC pairs before any write operation is resumed by the application

PPRC-SYNC copying static volumes while in the duplex pending state


For recovery implementations based on transmitting inactive database log files, you will not require application quiesces.

Please refer to Appendix E, “PPRC with static volumes implementations” on page 149 for examples of implementations of this particular solution using PPRC-SYNC.

2.6.4 Write operations response timeWhen carrying on a PPRC copy operation — using static volumes — the application’s write response time is not under consideration, because the application writes are quiesced while the PPRC copy operation is executed.

2.6.5 Distance factor“PPRC-SYNC using static volumes” is a copy technique appropriate for copying data at very long distances — continental distances. It can be implemented over distances well beyond the synchronous 103 km supported distance. These distances can be achieved using channel extenders over WAN lines (DWDMs are not recommended at very long distances). Also ESCON Directors can be used for shorter distances.

Please refer to Section 5.5, “PPRC connectivity” on page 70 for additional distance and connectivity considerations.

2.6.6 PPRC using static volumes — positioningIn Figure 2-8 we consider the positioning of PPRC using static volumes.

Figure 2-8 “PPRC-SYNC using static volumes” positioning

PPRC using static volumes - Positioning

It can be used over continental distances with excellent thruputOnly limited by channel extenders and network capabilities

Application writes need to be quiesced to perform PPRC copy operation (primary volumes mut be static)

Secondary copy automatically consistent, if checkpoint correctApplications' write operations performance irrelevant

Recovery data is as old as from the last PPRC copy operation

PPRC with static volumes is a recommended solutionPPRC with static volumes is a recommended solutionfor data copy, data migration and offsite backup, over for data copy, data migration and offsite backup, over continental distances with maximum thruput performance.continental distances with maximum thruput performance. It can be used for disaster recovery implementations if It can be used for disaster recovery implementations if the application can be quiesced and non-zero data loss the application can be quiesced and non-zero data loss RPO is admissible.RPO is admissible.


As summarized in Figure 2-8, PPRC with static volumes is a recommended solution for data migration, data copy, transmission of inactive database log files, and off-site backups, using static volumes — over continental distances — with maximum throughput performance. It can also be used for disaster recovery solutions over long distances, if the application can be quiesced.

2.6.7 PPRC with static volumes — implementation examplesFor examples of solutions you can implement when using PPRC with static volumes, please refer to Appendix E, “PPRC with static volumes implementations” on page 149. This appendix describes, at a high level, a series of possible implementations of this PPRC solution.

2.7 PPRC Extended Distance (PPRC-XD)In the PPRC Extended Distance method of operation, PPRC mirrors the primary volumes’ updates — onto the secondary volumes — in a non-synchronous manner, while the application is running. In this way, when in PPRC Extended Distance, the application’s write operations are free of the typical synchronous-like overheads.

The non-synchronous characteristic of PPRC Extended Distance combined with its throughput-efficient technique for copying only track updates, and combined with the latest microcode improvements in the supported channel extenders (refer to 5.5.2, “PPRC channel extender support” on page 71) makes PPRC Extended Distance a recommended option for remote copy solutions at very long distances — even continental distances — and with minimal application performance impact.

PPRC Extended Distance is an excellent solution for:

� Data copy� Data migration� Off-site backup� Transmission of data base logs� Application disaster recovery solutions based on periodic PiT (point-in-time) copies of the

data — if the application tolerates periodic short interruptions (application quiesce)

After reading the next section on how PPRC Extended Distance works, you will better understand why it is a recommended solution for long distance implementations.

2.7.1 How PPRC Extended Distance worksWhen in PPRC-XD way of operation, the volume pairs are in the duplex-pending XD state (see Figure 2-9). This is a new state — in addition to the traditional states, already described in Figure 2-4 on page 13 — in which a volume pair now can also be found.


Figure 2-9 PPRC Extended Distance new duplex-pending XD volume state

While in this duplex-pending XD state, PPRC is doing a non-synchronous mirroring — of the primary volumes’ updates — onto to the secondary volumes. While in this condition, PPRC-XD is periodically cycling through each volume’s bitmap for updated tracks and then, setting the updates in a batch — for transmission to their secondary counterparts. This is a throughput-oriented very efficient method of non-synchronous mirroring.

The volume pair will remain in duplex-pending XD condition, not reaching the duplex state.

This is a very powerful non-synchronous mirroring technique, very efficient for keeping the secondaries updated — far distant, well beyond the 103 km — without any synchronous-like overhead penalty upon the application’s write operations.

When in duplex-pending XD state, a pair will not exit until commanded to do so. The operation to exit the duplex-pending XD state will direct the volume pair either to the duplex state (a go-to-sync operation), or to the suspended state (a suspend pair operation), or to the simplex state (a delete pair operation).

Figure 2-10 shows the basics of the PPRC-XD option, and its associated characteristics.

Note: The efficient extended distance mirroring technique of PPRC Extended Distance is achieved using sophisticated algorithms. For example, if changed data is in the cache, then PPRC sends only the changed sectors. There are also sophisticated queueing algorithms to schedule each volume for processing of their updated tracks; and then the setting of the batches of updates to be transmitted.

These detailed PPRC-XD operation characteristics are not described, because they are beyond the scope of this document.

Application Site Recovery Site

PrimaryVolumes

SecondaryVolumes

Volume State Synchronizationand

consistencyDuplexDuplex

Pending XDPending XDNot in sync

andfuzzy

0

0

1

0

1

0

0

0

10

0

0

0

1

non-sychronousmirroring


Figure 2-10 PPRC-XD — basic operation characteristics

Catch-up (go-to-SYNC) operationPPRC-XD catch-up is the name of the transition that occurs to a PPRC-XD pair when it goes from its normal out-of-sync condition until it reaches a full sync condition. At the end of this transition primary and secondary volumes become fully synchronized.

The catch-up transition can be accomplished by commanding PPRC to go-to-SYNC, so the volume pair leaves the duplex-pending XD state and reaches the duplex state (refer back to Figure 2-10). From this moment on, if the pairs were not immediately suspended, primary write updates would be synchronously transmitted to the recovery site.

Also, as we will see later in the examples, the catch-up transition can be accomplished by just temporarily quiescing the application writes to the primary volumes — and waiting for PPRC-XD to finish the synchronization.

When triggering a catch-up by commanding go-to-SYNC, the rsQueryComplete command (for open systems that support the CLI) and the IEA494 system messages (for z/OS and OS/390 systems) will allow you to detect the moment when the volume pairs reach the duplex state — that is, when the catch-up is completed.

When doing the catch-up transition (from duplex-pending XD, to duplex) by commanding PPRC go-to-SYNC, you will not want any synchronous copy operations to occur — especially if the volumes you are mirroring are separated by long distances, beyond 103 km. For this, there is a new option — when you select the copy options — that allows you to ask PPRC to suspend the pair as soon as it is established. (This option is described later in 3.3.5, “Catch-up (go-to-SYNC) and suspend using the WUI” on page 36.)

suspended simplexfull-duplex

changed tracks'updates only

0

0

10

1

0

0

0

10

0

0

0

1

suspendedsimplex

suspend pair

suspend pair

catch

-up

catch

-up

go-to-syn

c

go-to-syn

cdelete pair

delete pair

inital copyinital copy XD option XD option

incremental incremental resync resyncXD optionXD option

PPRC-XD periodically cyclesthru the volumes for updatedtracks

volumes remain in duplex pending XD state until commanded to exit

New option XD when doing an initial copy or when doing an incrementalresynchronization

Volumes are scheduled forsubsequent transfer of updatesto secondaries

non-synchronous

Non-synchronous mirroring frees application writesfrom synchronous overheads

Secondary fuzzy copy is made consistent with adequate catch-up


How to trigger a catch-up transition by command (go-to-SYNC) is described later in (3.3.2, “Catch-up (go-to-SYNC) using TSO” on page 33, and in 3.3.5, “Catch-up (go-to-SYNC) and suspend using the WUI” on page 36). Also, in the following chapters, you will find examples of PPRC Extended Distance implementations showing how the catch-up transition is used.

Global catch-upWe will be using the generic term global catch-up to denominate any procedure that involves catch-ups of individual XD volume pairs — as part of the process for building a point-in-time copy of the data with global consistency, across all the volumes at the secondary site.

2.7.2 Data consistency at the recovery siteWhile in PPRC-XD state, the volume pairs remain continuously in the duplex-pending XD state — while the application is doing write updates onto the primary volumes. At this time, the recovery site secondary volumes are keeping a fuzzy copy of the data. This means that during this state, the application dependent writes are not assured to be applied on to the secondary volumes in the same sequence as they are written to the primary volumes.

Because of the non-synchronous characteristic of PPRC Extended Distance, at any time there will certain amount of application updated data that will not be reflected at the recovery site secondary volumes. These data corresponds to the tracks that were updated since the last volume bitmap scan was done. These are the out-of-sync tracks.

As we will see later, the amount of out-of-sync tracks (or sectors) can be checked with the CQUERY command (z/OS systems), or with the rsQuery command (for servers supporting the CLI), or clicking the Information Panel button for a selected pair — in the Volumes panel of the ESS Copy Services Web user interface.

The catch-up transition — whether going to duplex state by command (go-to-SYNC), or letting the application catch-up by stopping the writes to the primary volumes — allows you to synchronize the volume pairs in a minimum interval of time. Upon reaching duplex state, the PPRC volume pairs can be temporarily suspended, followed by a FlashCopy of the secondary volumes onto tertiary volumes, and then resuming the PPRC-XD relationships.

With PPRC Extended Distance you can have application global consistency — across all the volumes at the recovery site — if you implement the appropriate checkpoint activities at the application site in order to build this consistency. The catch-up transition will be part of this checkpoint. In the following chapters, you will find examples on how to set up these consistency building checkpoints.

When the recovery of the application is done using a point-in-time global consistent copy of the data, you must remember that, while in a an active PPRC-XD relationship, the secondary volumes always have a current fuzzy copy of the primary volumes. So for a valid application recovery, you must obtain the tertiary volumes where you FlashCopied the last globally consistent catch-up. As you can realize, this tertiary copy of the data will not be reflecting the updates to the primary volumes, since the last global catch-up operation.

Notes: The commanded catch-up transition (go-to-SYNC) should be triggered when the application write activity is low —or preferably none.

Also notice that — in the go-to-SYNC operation — PPRC does an incremental copy of the changed tracks’ updates. This is a very efficient synchronization technique that minimizes the time needed for the catch-up transition.


In the following chapters we present examples on how to use the catch-up procedure to get point-in-time copies of the data at the recovery site, with global consistency across all the volumes (global catch-ups).

2.7.3 Application availabilityPPRC Extended Distance continuously mirrors the primary volumes onto the recovery site secondary volumes — while the application is running. This provides a fuzzy copy at the recovery site.

For application recovery implementations based on point-in-time (PiT) copies of the data, you will have to plan for appropriate checkpoints to briefly quiesce the application and synchronize the volumes pairs (catch-up). This will provide the PiT copy with global data consistency. In the following chapters you will find stepped approaches to minimize the impact on the application when doing this global catch-up operations.

Application recovery implementations based on transmitting inactive database log files, will not require application quiesce.

2.7.4 Write operations response timePPRC Extended Distance uses a non-synchronous mirroring technique, and so it does not incur synchronous-like overheads upon the applications’ write operations. The application’s write operations are finished as soon as the updates are secured in the primary ESS non-volatile storage — like in normal non-PPRC operation.

This characteristic makes PPRC Extended Distance recommended for solutions needing a minimum impact on the application response time — especially when extended distances are in consideration.

2.7.5 Distance factorThe non-synchronous technique of PPRC Extended Distance, relieves the application write operations from the typical synchronous mirroring overhead — which is directly proportional to the distance between the application and the recovery sites.

This characteristic makes PPRC Extended Distance a recommended solution when planning for implementations over extended distances — continental distances, well beyond the synchronous 103 km supported distance. The distance limited only by the capabilities of the network and the channel extenders technologies.

These extended distances can be achieved using channel extenders over WAN lines. Also ESCON Directors can be used for shorter distances. Please refer to 5.5, “PPRC connectivity” on page 70 for additional distance and connectivity information.

2.7.6 PPRC Extended Distance positioningAs summarized in Figure 2-11, PPRC Extended Distance is a recommended solution for data copy, data migration, off-site backup, and transmission of inactive database logs, with excellent performance — particularly relevant when implemented over continental distances. PPRC Extended Distance can also be used for application recovery solutions based on periodic point-in-time backup copies of the data — if the application tolerates periodic brief interruptions (application quiesce).


Figure 2-11 PPRC-XD. Positioning

PPRC Extended Distance operation characteristics — keeping a fuzzy copy of the data, and only needing short catch-ups in order to re-synchronize the volumes — makes application quiesce checkpoints more tolerable, and so the PiT consistent versions of the recovery data will be much more frequent, thus tending to the least non-zero recovery-point-objective.

2.7.7 PPRC-XD implementationsIn the following chapters of this redbook you will find examples of solutions you can implement using PPRC Extended Distance.

PPRC Extended Distance Positioning

It can be used over continental distances with excellent application performance

The distances are only limited by the network and channel extenders capabilitiesApplications' write operations do not have synchronous-like overheadsFuzzy copy of data at the recovery site (sequence of dependent writes may not be respected at the recovery site) Recovery data can become a consistent point in time mirror of the primary data, if appropiate application checkpoints are set to do global catch-ups

Pairs are synchronized with application group consistency Synchronizations can be done more frequently, because of short catch-ups

RPO still non-zero, but improves substantially

PPRC Extended Distance is a recommended solution forPPRC Extended Distance is a recommended solution fordata copy, data migration and offsite backup - over continental data copy, data migration and offsite backup - over continental distances - with excellent application performance.distances - with excellent application performance. It can be used for disaster recovery implementations if It can be used for disaster recovery implementations if application can be quiesced and non-zero data loss RPOapplication can be quiesced and non-zero data loss RPOis admissible.is admissible.


Chapter 3. PPRC Extended Distance characteristics and management

PPRC-XD (Peer-to-Peer Remote Copy Extended Distance) is a non-synchronous remote copy function of the IBM TotalStorage Enterprise Storage Server (ESS) disk storage subsystem.

In the first part of this chapter, we further describe the characteristics of the new PPRC Extended Distance remote copy function, which was introduced in the previous chapter. We also present additional considerations for positioning PPRC Extended Distance.

In the second part of this chapter, we describe the commands and tools for managing PPRC Extended Distance.

3


3.1 PPRC optionsAfter reading the previous chapter, you can see that PPRC Extended Distance (PPRC-XD) brings a new option in the ESS to ESS mirroring capabilities. It adds a new alternative to PPRC — which can now mirror data in a non-synchronous way. So you have three ways of using PPRC:

� Synchronous operation (PPRC-SYNC): This synchronously mirrors the updates done to the primary volumes. Can be used in distances of up to 103 km (an RPQ has to be submitted if slighter longer distances need to be implemented).

� Synchronous operation using primary static volumes: This can be used to move or copy data at very long distances using channel extenders.

� Extended Distance operation (PPRC-XD): This operates non-synchronously and can be used over continental distances, with excellent application performance. When implementing this solution over long distances, you need to use channel extenders.

In the following sections we go further in detail, presenting the characteristics of PPRC Extended Distance that we started introducing in 2.7, “PPRC Extended Distance (PPRC-XD)” on page 18.

3.1.1 Basic characteristicsFigure 3-1 shows the basic characteristics of PPRC Extended Distance.

Figure 3-1 PPRC Extended Distance basic characteristics

When doing the initial copy — when establishing a new pair, or when re-synchronizing (RESYNC) a previously suspended volume pair — you now have two options you can specify:

New option when establishing a volume pair: extended distance (XD)New XD option, in addition to the traditional synchronous option (SYNC)Volume level optionCRIT setting not valid with XD option Requires channel extenders for long distance implementations

Non-synchronous copy to secondary volumes of application updates onto primary volumesWrite updates to primary receive immediate completion while volumes in XD statePrimary ESS keeps record of updated tracks in primary volume bitmapIncremental copy of primary updates periodically sent to secondaryLess demanding bandwidth requiriments compared to SYNC optionNo synchronous-like overhead on applications' response time when volumes in XD statePotentially unlimited distance with channel extenders

Only limited by the channel extender and network technology capabilitiesMaintains a fuzzy copy at secondary volumes until forced to synchronous

No certainty for dependent writes mirroring sequence

channelextender

channelextender

secondary volumeprimary volume

non-syncrhonous PPRC

over long distance


� SYNC option: For synchronous mirroring. In this method of PPRC operation the initial copy of a pair is completed up to the duplex state — and from there on, the updates to the primary volumes are mirrored synchronously onto the secondary volumes. As a consequence, there is a transmission overhead, manageable under 103 km, but which becomes progressively more significant at longer distances. At these longer distances this overhead, typical of a synchronous operation, slows down the performance of the application too much to fulfill reasonable production objectives.

� XD option: For non-synchronous transfer of updates over extended distances. In this new method of PPRC operation — when doing the initial copy of a pair, or when a suspended pair is re-synchronized — the volumes do not reach the duplex state, but remain in the new duplex pending XD state. While in duplex pending XD state, the primary volume updates complete without waiting the mirroring onto the secondary to complete. The primary volumes’ updated tracks are checked in a bitmap — and the updates are periodically sent in batches to the secondary volumes.

In 3.3.1, “Establishing XD pairs using TSO commands” on page 32 and 3.3.4, “Establishing XD pairs using the Web user interface” on page 34, we describe how you set the pairs to work either in XD or in SYNC mode. Also, when reading Appendix A, “ESS Copy Services creation of tasks” on page 95, you will be able to see how you set the pairs to work in either mode.

Fuzzy copy — dependent writesThe XD non-synchronous batch mirroring technique does not guarantee that application dependent writes are copied in the same sequence as they have been applied onto the primary volumes: the secondary copy is fuzzy.

CRIT setting not validAs XD mirroring is done non-synchronously, any CRIT setting is invalid for volume pairs established with XD option.

Long distance efficiencyThe PPRC-XD non-synchronous batch mirroring technique is designed to be highly throughput efficient over extended distances — while preserving the application performance.

For long distance implementations, PPRC Extended Distance requires channel extenders to send the updates remotely over telecommunication lines. PPRC will operate transparently on the telecommunication implementation whatever the distance: the distance will be only limited by the network and channel extender technology capabilities. Please refer to 5.5, “PPRC connectivity” on page 70.

3.1.2 PiT Implementation basicsWhile in an XD relationship, the secondary volumes have a fuzzy copy of the primary volumes, and so are not valid for a point-in-time (PiT) consistent recovery. You will need to synchronize primary and secondary volumes — this is the catch-up transition — and also reach a consistent checkpoint across all the volume pairs in which the application was doing dependent writes (global catch-up).

Finally, you will produce a tertiary copy of the just built point-in-time (PiT) consistent checkpoint copy — because from this tertiary copy you will want to recover your application if necessary to do so. Remember that the secondary volumes will become fuzzy again, as soon as the pairs are reestablished in an XD relationship.

Chapter 3. PPRC Extended Distance characteristics and management 27

The example in Figure 3-2 illustrates two variations of a general procedure to get a consistent PiT copy of the data — over a long distance implementation (beyond the synchronous 103 km supported distance).

Figure 3-2 PPRC Extended Distance basic setup example

The six steps are basically the same, with the difference being that in step 2, if you wait for the application writes to quiesce completely, then in step 3, you can skip the re-synchronization. This path — where you wait for the application writes to quiesce completely — is highlighted in Figure 3-2. In the following sections we discuss both variations of this example.

Note: In this section we briefly introduce you to the basic building blocks of solutions using PPRC Extended Distance — by discussing one possible setup for a long distance implementation. Then in the following chapters, this arrangement —and possible variations, are described in further detail.

The optimum setup — variations of steps, sequence, and timings — will be application dependent, and you will be able to determine it by testing when doing your actual implementation.

Note: Within synchronous supported distances, this procedure can be done differently — as explained in 5.2.2, “PPRC-XD PiT within the synchronous range” on page 62

channelextender

channelextender

FlashCopy

FlashCopy

secondarysecondary

terciaryterciary

primaryprimarynon-syncrhonousnon-syncrhonous PPRC PPRC

over long distance

Recovery SiteRecovery Site

fuzzy copy of data

individual volume pairs synchronize

copy of data globally consistent across volume pairs

5. FlashCopy

Application SiteApplication Site

minimum performance impact

1. Quiesce application updates2. Catch-up (synchronize volume pairs)

go-to-sync / suspendor wait for application writes to quiesce

3. Build consistency on recovery data

resync and freezeor freeze (suspend)

4. Restart application writes as soon as freeze is done6. Reestablish suspended pairs (resync)

consistent tertiarycopy of data


Quiesce application writesThe sequence commences by quiescing the application writes. Then, for the first alternative, you command PPRC go-to-SYNC to trigger the catch-up — before the quiesce is finished. For the second alternative, you wait until there are no more application writes — you wait for quiesce to complete. Please refer back to Figure 3-2 to follow the explanation.

Catch-upFor the first alternative (command PPRC to go-to-SYNC), each of the volume pairs will individually, at different points-in-time, reach the duplex state, and will be synchronized.

Immediately upon reaching the duplex state, the volume pairs should be automatically suspended (refer to 3.3.5, “Catch-up (go-to-SYNC) and suspend using the WUI” on page 36, for a description of how to do this operation). The reason for suspending is to prevent the synchronous copy operations — characteristic of the duplex state — from occurring. Remember that in this alternative, application writes may still be occurring.

For the second alternative, the catch-up (synchronization of the volume pairs) is achieved by waiting until there are no more application writes onto the primary volumes — application completes quiesce. With this alternative, you also get the volume pairs fully synchronized, but without reaching the intermediate duplex state, just remaining in duplex pending XD while the application writes become completely quiesced. By checking PPRC query output information, which tells the amount of tracks out-of-sync, you will be able to detect the moment when the volume pairs are fully synchronized (3.4, “Querying PPRC-XD” on page 37, describes the queries you can use).

At the end of this catch-up step, all the volumes will be individually synchronized. At this time, if the catch-up was a passive one — just waiting for the application to completely quiesce while the volumes remained in duplex pending XD mode — then the secondary volumes are already a PiT copy with global consistency.

Build PiT consistency on recovery dataIf the catch-up was triggered by a go-to-SYNC command — without waiting for the application quiesce to complete — then an additional step is needed: the re-synchronization of the pairs at a moment when there are no more application writes at all (application completely quiesced). Then it is this re-synchronization that will leave the volume pairs in duplex state — and the secondaries with a PiT copy with global application consistency.

From now on, both alternatives follow the same path.

Immediately upon reaching a PiT consistent checkpoint copy at the secondary site, the PPRC relationships should be suspended — so the application can restart its write activity as soon as possible, without affecting the secondary volumes.

The suspension of the volume pairs can be done by doing individual pairs suspensions (one major task grouping several suspension tasks), or in a global consistent manner — which is much more efficient — commanding PPRC to freeze the set of related LSS pairs (freezing is covered later in 4.2, “Building consistency” on page 49).

Recommendation: The catch-up transition should be scheduled when the number of out-of-synchronization tracks — between the primary and secondary volumes — is minimal. The catch-up should be triggered at a moment when the application write activity is low (or none). The best situation is to wait for the application writes to completely quiesce. In the discussions on how to use PPRC-XD, which we present in the following chapters, you will find stepped approaches to minimize the application impact when doing this transition.


Restart application write activityAs soon as the freeze is done, no more updates will be transmitted to the secondary volumes — pairs are suspended and paths are discontinued. So now, the application can resume its write activity — without affecting the consistent PiT checkpoint copy, that the secondaries still hold.

FlashCopy onto tertiariesWith the secondaries holding a consistent PiT checkpoint copy, and the pairs suspended, you can now do FlashCopy —of the consistent checkpoint— onto a set of tertiary volumes. As soon as the FlashCopy relationship is established, you can proceed with the next step.

Reestablish PPRC-XD relationshipsWith the FlashCopy relationship established, you now restart the PPRC-XD relationships by reestablishing the suspended pairs — with the XD option. At the completion of the FlashCopy you will have a tertiary consistent PiT checkpoint copy of the application data.

This final step completes the procedure — with the volumes now back again in a PPRC-XD relationship — and it will be resumed again later, when the next PiT checkpoint is done.

3.2 PPRC-XD states change logicWith the introduction of PPRC Extended Distance the state change logic of PPRC changes slightly. In this section we review the new considerations that come into play. Please refer to Figure 3-3 for the following explanations.

1. Pairs can change from the duplex-pending XD (XD) sate, to the traditional duplex (SYNC) state — when go-to-SYNC is commanded (go-to-SYNC arrow in Figure 3-3).

2. Now, you can also request that a pair be suspended as soon as it reaches the full-duplex state (go-to-SYNC and SUSPEND in Figure 3-3).

3. Pairs cannot change directly from SYNC state to XD state. They need to go through an intermediate suspended state.

4. You can go from suspended state to XD state doing an incremental copy (copying out-of-sync tracks only). This is a similar process as for the traditional transition from suspended state to SYNC state (RESYNC / copy out-of-sync arrow in Figure 3-3).

5. When you initially establish a pair, you now have the option to request that it becomes an XD pair (establish XD arrow in Figure 3-3), or a traditional fully duplexed pair (establish SYNC arrow in Figure 3-3).


Figure 3-3 PPRC state change logic

The dotted lines in Figure 3-3 represent the termination of the pairs. We recommend that when you terminate suspended pairs, always specify the primary LSS. If you specify the secondary LSS, then the primary volumes will remain suspended — because this will be interpreted by PPRC as a recovery operation.

When you query a primary volume of a volume pair:

� If the pair was established as a SYNC pair, then — initially, when the pair has just been established — you will be first informed of the duplex-pending state condition while the pair is synchronizing. After — when the pair has completed the initial synchronization — you will be informed of the duplex state condition — the volumes are fully synchronized. The number of tracks or sectors out of synchronization (out-of-sync), is informed while in duplex-pending — until duplex is reached.

� If the pair was established as an XD pair, then you will be informed of the duplex-pending XD state condition — and the volume will never reach the duplex state condition. The number of tracks or sectors out of synchronization (out-of-sync) is always displayed — even if it is zero — because in fact the volumes never leave the duplex pending state.

The query examples are presented in 3.4, “Querying PPRC-XD” on page 37.

SUSPENDEDSUSPENDED

SIMPLEXSIMPLEX

SYNCSYNC

full DUPLEX

XDXD

DU

PLE

Xpending XD

go-to-SYNC

and SUSPEND

estab

lish X

D

RESYNC /

copy out-of-sync

go-to-SYNC

RESYNC /

copy

out-o

f-syn

c

establish SYNC

SUSPEND

SUSPEND


3.3 Managing PPRC Extended DistanceThe information presented in this section will explain the new options available for managing a PPRC Extended Distance environment. These new options are described for all the three PPRC management tools: TSO commands, Web user interface (WUI), and Command Line interface (CLI).

Establishing an XD pairYou establish a pair in a XD relationship, either from a pair of volumes that are not in a PPRC relationship (primary and secondary are in simplex state), or from a PPRC pair that is now suspended (primary and secondary volumes are in suspended state). For both transitions, the pair of volumes end up in the new duplex pending XD state. Please refer back to Figure 3-3.

Initial copy versus re-synchronization - establish versus reestablishThe transition from simplex to duplex pending XD — is called initial copy. It happens when we establish a new pair, and do the full volume copy — from the primary onto the secondary.

The transition from suspended to duplex pending XD — is called re-synchronization (RESYNC). It happens when we reestablish a suspended pair.

One difference between these two transitions is that — for the re-synchronization — PPRC does an incremental copy, copying only the tracks that were updated while the volumes were in the suspended state. PPRC keeps a bitmap for this purpose, while the pair is in suspended state.

ICKDSFPPRC Extended Distance cannot be managed by the ICKDSF utility alone — as certain PPRC-XD parameters can only be issued from TSO commands, the ESS Copy Services Web user interface (WUI), or the ESS Copy Services command line interface (CLI).

Current PPRC users of ICKDSF will find in Appendix D, “PPRC-XD operation for z/VM and VSE/ESA users” on page 145, a useful aid for transitioning to the WUI and the CLI — for PPRC Extended Distance operation.

3.3.1 Establishing XD pairs using TSO commandsFigure 3-4 shows the establishment of XD pairs.

Figure 3-4 CESTPAIR command with new OPTION (XD/SYNC) parameter

CESTPAIR DEVN(device_number) PRIM(ssid serialno cca lss) SEC(ssid serialno cca lss)

SYNC COPY PACE(15) NOOPTION( XD ) MODE( NOCOPY ) PACE(1-255) CRIT( YES ) RESYNC

XD

NO NO ONLINSEC( YES ) MSGREQ( YES )


The command used for establishing (and reestablishing) a PPRC pair is the CESTPAIR command. As you can see from Figure 3-4, this command brings a new optional parameter: OPTION — with two mutually exclusive values: SYNC or XD. You specify SYNC if you want to establish the pair in the typical synchronous duplex state. You specify XD if you want to establish the pair in the new non-synchronous duplex pending XD state.

For the pair to become an XD pair — in addition to specifying the XD value for the OPTION parameter — you will also specify whether this pair comes from the simplex or from the suspended state (refer to Figure 3-3 on page 31). That is to say, whether this is an initial copy of a newly established pair, or if it is a re-synchronization of a suspended pair. To indicate this, you use the MODE parameter — specifying either COPY or RESYNC, respectively. Figure 3-4 shows the CESTPAIR command with the new XD option shaded.

Figure 3-5 Establishing an XD initial copy pair with CESTPAIR

Figure 3-5 shows an example of how you establish an initial copy pair that will be working in XD mode. For detailed information on the CESTPAIR command syntax and options, you can refer to the publication z/OS DFSMS Advanced Copy Services, SC35-0428.

3.3.2 Catch-up (go-to-SYNC) using TSOAs we already mentioned, PPRC-XD catch-up is the name of the transition for a PPRC-XD pair when it goes from its normal out-of-sync condition — until it reaches a full sync condition (SYNC). Please refer Figure 3-3 on page 31. In other words, the pair goes from the duplex pending XD state to the duplex state. At the end of this transition primary and secondary volumes become fully synchronized, with all their respective tracks identical.

To initiate a catch-up operation, you command PPRC to go-to-SYNC — using the TSO command CESTPAIR. For the go-to-SYNC, the CESTPAIR command must be used with the OPTION(SYNC) and MODE(RESYNC) parameters. Please refer back to Figure 3-4 on page 32.

Note: The CRIT(YES) and the MODE(NOCOPY) parameters are mutually exclusive with the OPTION(XD).

CESTPAIR DEVN(X'F40') PRIM(X'A763' FCA76 X'4F' X'03') SEC(X'A762' FCA76 X'10' X'02') MODE(COPY) OPTION(XD)

Note: Currently the CESTPAIR command does not support the “go-to-SYNC and suspend” operation, described in 3.2, “PPRC-XD states change logic” on page 30.

For these environments — in order to suspend as soon as the duplex state is reached — you have to automate using the state change messages that the system issues.


3.3.3 CESTPAIR command parameter valuesTable 3-1 summarizes the values of the CESTPAIR command parameters, which you will be using when working with XD pairs.

Table 3-1 CESTPAIR parameter values

As already mentioned, the CRIT(YES) and MODE(NOCOPY) parameters are mutually exclusive with the OPTION(XD) parameter value.

3.3.4 Establishing XD pairs using the Web user interfaceWhen using the Web user interface, you commence as usual — by selecting the source and target volumes from the Volumes panel to launch the Task Wizard (Figure 3-6).

Figure 3-6 Selecting the volume pair and launching the Task Wizard

As you can see in Figure 3-7 (with more detail), the Task Wizard now has a new option: Establish Extended Distance PPRC.

Operation Volume pair state CESTPAIR parameter values

from to OPTION MODE

Establish initial copy pair

simplex duplex pending XD XD COPY

Reestablish suspended pair

suspended duplex pending XD XD RESYNC

go-to-SYNCcatch-up

duplex pending XD duplex SYNC RESYNC


Figure 3-7 Establishing the PPRC-XD pair

So, whether to establish an initial copy pair in a XD relationship, or to reestablish a suspended pair in a XD relationship, you select the new Establish Extended Distance PPRC option.

After selecting the Establish Extended Distance PPRC option and clicking the Next button, you will be able to select the copy options from the Select copy options window that pops up (see Figure 3-8). As you can see here, you can either select the option Copy entire volume, or the option Copy out-of-sync cylinders only. You use the first option to do an initial copy of a newly established pair — going from simplex to duplex pending XD state. While you use the second option to re-synchronize (RESYNC) a suspended pair — going from suspended to duplex pending XD state.

Figure 3-8 Selecting PPRC-XD copy options


For complete information on how to use the Web user interface, you can refer to the publication IBM TotalStorage Enterprise Storge Server Web Interface User’s Guide, SC26-7346.

3.3.5 Catch-up (go-to-SYNC) and suspend using the WUIWhen initiating a catch-up operation commanding PPRC to go-to-SYNC — using the Web user interface (WUI) — you select the Establish synchronous PPRC copy pair option from the Task Wizard, as shown in Figure 3-9.

Figure 3-9 Catch-up (go-to-SYNC)

After clicking the Next button, then the Select copy options window pops-up (Figure 3-10). Here, you choose the Copy out-of-sync cylinders only option. In this same window, you also select the Suspend PPRC after establish option; that will immediately suspend the pair as soon as the full duplex condition is reached. As already discussed, this option assures you that synchronous updates are not going to happen.

Note: The legend of the re-synchronization option, in the Select copy options window in Figure 3-8, says Copy out-of-sync cylinders only; but in fact, the copy operation copies only the tracks or sectors — not cylinders — that were changed while the pair was suspended. This is a very powerful incremental technique for the re-synchronization of the suspended volumes, when going from suspended state to either duplex or duplex pending XD state.

Also notice that the Do not copy volume and the Critical volume mode copy options are not valid when establishing an XD pair.

Note: During the go-to-SYNC transition — where the volume pair leaves the duplex pending XD state and reaches the duplex state, to be immediately suspended — the synchronization is done in an incremental manner, transmitting only the updates that are pending. This is a very efficient synchronization technique, which minimizes the time needed for the catch-up transition. Don’t be mislead by the cylinders reference from the Copy out-of-sync cylinders only legend — in the Task WIzard options window, illustrated in Figure 3-10.


Figure 3-10 Suspending as soon as the catch-up is finished

3.3.6 Web user interface optionsTable 3-2 summarizes the options that you will be selecting when working with XD pairs — when using the Web user interface panels.

Table 3-2 Web user interface options

As already mentioned, the Critical volume mode option and the Do not copy volume option — are not valid options when establishing an XD pair.

3.4 Querying PPRC-XDThere have been some modifications in the information displayed by the PPRC queries, in order to support PPRC Extended Distance. The new duplex pending XD status is reported, as well as some changes in the granularity when prompting the out-of-sync tracks/sectors.

Operation Volume pair state WUI options

from to Select task type Select copy options

Establish initial copy pair

simplex duplex pending XD Establish Extended Distance PPRC

Copy entire volume

Reestablish suspended pair

suspended duplex pending XD Establish Extended Distance PPRC

Copy out-of-sync cylinder only

go-to-SYNCcatch-up and suspend

duplex pending XD suspended (as soon as the volumes reach duplex state)

Establish synchronous PPRC copy pair

Copy out-of-sync cylinders only+Suspend PPRC after establish


3.4.1 CQUERY commandThe TSO command CQUERY has not changed, but the output now informs you of the duplex pending XD state, if the volume is part of an XD pair.

Figure 3-11 Output information for a CQUERY on primary XD volume

In Figure 3-11 you can see the output of a CQUERY command addressed to volume x’F40’ — which is the primary volume of an XD pair of volumes. You can see — in the output information— the STATE of the volume is PENDING.XD. As already discussed, the pair will remain in this condition — never reaching the duplex state, unless commanded to do so.

Also from the output information you can check the number of primary updated tracks, that at that moment where not reflected on the secondary volume (TRACKS OUT OF SYNC).

3.4.2 rsQueryComplete commandThe Command Line Interface (CLI) rsQueryComplete command has been enhanced, in order to allow users to more promptly get status information while waiting for a task to complete its PPRC copy initialization. This is especially useful, for example, at the moment you command a pair go-to-SYNC — when you want to detect when the copy task has completed its process.

Previously, users could use the rsQueryComplete with the -m parameter, to specify a whole number of minutes to wait between checks of the copy process completion. The lowest number of minutes was one minute. Now the -m flag takes values in the form of MM:SS so you can enter an interval time less than one minute. For example, a value of 00:30 will report every thirty seconds.

CQUERY DEVN(X'F40')

CQUERY FORMATTED LVL 2VOLUME REPORT************** PPRC REMOTE COPY CQUERY - VOLUME ********************* (PRIMARY) (SECONDARY) ** SSID CCA LSS SSID CCA LSS**DEVICE LEVEL STATE PATH STATUS SERIAL# SERIAL# **------ --------- ---------- ----------- --------- --------- ** 0F40 PRIMARY.. PENDING.XD ACTIVE.. A763 4F 03 A762 10 02 ** CRIT(NO)....... CGRPLB(YES) 0000000FCA76 0000000FCA76** PATHS SAID/DEST STATUS: DESCRIPTION ** ----- --------- ------ ------------------- ** 1 0020 EA02 01 PATH ESTABLISHED... ** ---- ---- 00 NO PATH............ ** ---- ---- 00 NO PATH............ ** ---- ---- 00 NO PATH............ ** IF STATE = PENDING/SUSPEND: TRACKS OUT OF SYNC = 19471 ** TRACKS ON VOLUME = 33390 ** PERCENT OF COPY COMPLETE = 42% *********************************************************************CQUERY COMMAND COMPLETED FOR DEVICE 0F40. COMPLETION CODE: 00

Note: The time specified — in the -m parameter — should not be too short, since this status prompting is causing disruptions to the process.


In Figure 3-12 you can see an example where the rsQueryComplete command has been set to report the initialization complete every 30 seconds.

Figure 3-12 rsQueryComplete reporting at shorter intervals (seconds)

3.4.3 rsQuery commandThe Command Line Interface (CLI) rsQuery command now informs you of the duplex pending XD state — when the volume is part of an XD pair.

Figure 3-13 shows the output of an rsQUERY CLI command, addressed to volume serial number 20221389 — which is the primary volume of an XD pair of volumes. As you can see, the output information reports the Type as extended distance.

Figure 3-13 rsQuery of an XD primary volume

Also, from the output information, you can check the number of primary updated sectors that — at the moment of the rsQuery — were not reflected on the secondary volume — Number of out-of-sync sectors.

C:\Program Files\IBM 2105 CLI>rsQueryComplete /v /s csServerHostName /m 00:30 EST_XDIST_TASKrsWebTest: HeartBeat to the server was successful.rsQueryComplete: Got task manager referencersQueryComplete: --------- Task Name: EST_XDIST_TASK ---------rsQueryComplete: Task EST_XDIST_TASK found by TaskManagerrsQueryComplete: Sampling volumes...rsQueryComplete: Percentage complete = 20rsQueryComplete: waiting 30 seconds...rsQueryComplete: Sampling volumes...rsQueryComplete: Percentage complete = 88rsQueryComplete: waiting 30 seconds...rsQueryComplete: Sampling volumes...rsQueryComplete: Percentage complete = 100rsQueryComplete: ----------------------------------------------rsQueryComplete: Command successful

C:\Program Files\IBM 2105 CLI>rsQuery /v /q 20221389 /s ESS-ServerNamersWebTest: HeartBeat to the server was successful.************************Volume Information************************Volume 20221389 found on 21389:12 as volume number 002PPRC State=source, Type=extended distance, Status=copy_pendingPPRC Number of out-of-sync sectors=01880320FlashCopy_state=none, Size=1.0_GBPPRC Peer=30221389******************************************************************


3.4.4 The Information Panel of the Web user interfaceYou can also get information, from the Information Panel, on the number of tracks or sectors out-of-sync for a selected primary volume. Figure 3-14 shows what happens when a pair of volumes is in an extended distance relationship — and then the primary volume is selected, and the Information Panel button is clicked.

Figure 3-14 Getting the Information Panel output


The Information Panel pops up. Figure 3-15 shows, in more detail, information on the out-of-sync cylinders — Number of out-of-sync cylinders.

Figure 3-15 Information Panel, out-of-sync cylinders

As you can see from the example in Figure 3-15, no matter the out-of-sync cylinders is zero, the PPRC status is still copy pending. This is because the pair is in a XD relationship — and it will never leave the duplex pending XD state, unless commanded to do so.


Chapter 4. PPRC configuration and data consistency

In this chapter we review some of the general PPRC configuration guidelines that also apply when configuring a PPRC Extended Distance environment. We discuss any recommended variations, specific for PPRC Extended Distance, that may be advisable.

We also review some of the things you must consider, as well as the tools that PPRC provides, for building global consistency at the secondary site.

4


4.1 PPRC configurationWhen implementing a PPRC configuration, one important point is the definition of the paths that PPRC is going use. In this section we review this process — and we discuss some considerations you must have in order to plan for consistency.

4.1.1 Establishing conceptsYou establish paths between the application site (primary, or source), logical subsystem (LSS), and the recovery site (secondary, or target) LSS. When you establish a path:

� You identify the source and target LSSs.

� You identify the physical link that will be used between the primary and secondary LSSs.

� Then, optionally, you can set the consistency group option (discussed later in 4.2.1, “Consistency groups” on page 50).

You first establish the paths, before establishing the volume pairs. Each pair (of source and target LSSs) will have a minimum of one path defined. You can define as many paths as physical links exist between the primary and secondary ESSs — with a limit of eight paths per LSS pair.

You do not have to reestablish paths after an IML of ESSs. However, you will have to reestablish paths after a freeze operation (the freeze operation is discussed later in 4.2.4, “Freezing operation” on page 53).

Be aware that each path definition you make will totally replace any existing path definitions for that same pair of LSSs. So, if you are adding a path, then you should redefine all previous path definitions between these two LSSs — in an explicit accumulative way.

Figure 4-1 summarizes these considerations.

Figure 4-1 Establishing paths between LSSs

4.1.2 Defining paths between two LSSsIn this section we review the concepts of links and paths, we review their configuration guidelines, and we discuss what happens when the paths are terminated because of errors.

Links, paths, and channel extendersLinks are the physical connecting elements between the pairs of source and target ESSs. ESCON links have the capability of operating in two ways — but when implemented for PPRC transmission, they operate in either one of those two ways.

Establishing paths between primary and secondary LSSsIdentify source and target LSSsIdentify the physical link for each logical pathOptionally: set the consistency group flag for this pair of LSSsUp to 8 logical paths between a pair of source and target LSSs

Paths definition is kept between IMLs


The path is the one-way utilization of a link by the host adapter interface. An ESCON host adapter port operates in control unit mode, when it is talking to a host. In this mode, an ESCON port can also receive data from a primary ESS, when the ESS port is connected to an ESCON director. An ESCON port is operating in channel mode, when it is used on the primary ESS for PPRC operations onto a secondary ESS.

PPRC is aware of System Adapter IDs (SAIDs), of the ESS ESCON channel interface that is been used for the link, and of ESCON Director (ESCD) ports that are in use in the link to the recovery site. Any intermediate channel extender and network infrastructures, which are in use, are transparent for PPRC.

A primary ESS can have multiple ESCON links (up to the 32 maximum possible per ESS), for connecting it to multiple secondary ESSs. But each primary LSS (an LSS is a logical entity within an ESS) can only connect to up to 4 secondary LSSs from any secondary ESSs. However, a secondary LSS can be connected to any number of primary LSSs.

Path definitionPaths are defined for a primary-to-secondary LSS pair. Up to eight paths can be defined per LSS pair. For each LSS pair, each path is defined over a different physical link. The path definition — by identifying the source and the target LSSs — implicitly sets the direction of the link. A link will operate in only one direction. If you want to have a path in the opposite direction, then you will have to use a different link. Note that if you delete all the paths over a physical link, then you can reuse that link for paths in the opposite direction.

For a pair of ESSs, the physical links between them can be shared with many path definitions from different pairs of LSSs within those ESSs. In practice, many installations configure paths to share the same links for easier management.

For redundancy, and for performance, it is recommended to maximize (up to eight) the number of paths between a pair of source and target LSSs. When a pair of volumes is initially established — or at re-synchronization time of a suspended pair of volumes — PPRC is able to use all the defined paths in parallel.

Path failuresWhen a path fails, a notification is sent to the host server. For open systems servers this will be an SNMP alert. For z/OS systems this will be an IOS581E message. The activity for that pair of LSSs continues — if more paths are available — rerouting the updates transmissions over the alternate paths between that LSS pair.

When the last existing path, between a pair of source and target LSSs, breaks — because some error condition is happening — then this event is considered by PPRC to be a major error condition. In open systems environments, SNMP alerts are sent to the application servers. In z/OS environments, successive messages are issued: IOS581E (link failed reported error), IEA494I (PPRC pair suspended), and IEA491E (communication to secondary failure). And, under this error condition, the pairs will be suspended.

When the consistency group option has been set (at path definition time) then an extended long busy condition will be presented —and this interval can be used by the automation routines to trigger a global freeze (the concept of a freeze is discussed later in 4.2.4, “Freezing operation” on page 53).

Chapter 4. PPRC configuration and data consistency 45

Figure 4-2 summarizes the guidelines and considerations discussed in this section.

Figure 4-2 Path definition, configuration guidelines, and error handling

4.1.3 Displaying path informationIn this section we will see how the paths and link information is displayed, and use this example to illustrate how paths and links relate to each other.

In the example we are presenting, the source and target LSSs are within the same ESS. There are four ESCON interfaces (SAID)): 0000, 0004, 0080, 0084. And we will be using two ESCON links to connect these four interfaces in pairs: 0000 with 0080, and 0004 with 0084.

This connection allows to define up to two paths — out of four choices possible. The four choices that could be possible, are:

0000 ---> 00800080 ---> 00000004 ---> 00840084 ---> 0004

Figure 4-3 shows the information that we get from the Paths panel of the Web user interface — when the four ESCON adapters are selected, and the Information Panel button is clicked.

As you can see in Figure 4-3, when the paths are yet not defined, there are four choices of possible paths — two links, each with two choices of direction. From these four possible choices, you will only materialize two, because each link can operate only in one direction.

Paths and links A primary ESS can have a variable number of ESCON links connecting to secondary ESSs (up to 32) A primary LSS can connect up to 4 seconday LSSsA secondary LSS can connect any number of primary LSSs A path is a one-way utilization of a link between the source and target LSSs

Paths are defined for each primary-to-secondary LSS associationUp to 8 paths per LSS pair, each path mapped onto one different link This path mapping determines the direction of the link

Links cannot be reused for a reverse LSS association -need to use another linkNumber of paths should be maximized for redundancy and establish pair performance

Path failuresWhen one of the paths fails

The I/O in progress is automatically re-routed on an existing alternate pathHost is notified with message IOS581E for z/OS systemsSNMP alert issued for open systems hosts

When all paths failIf host is z/OS then notified with IOS581E, IEA494I, IEA491E messages If host open system then notified with SNMP alertextended long busy condition is presented if consistency group enabled


Figure 4-3 Paths information before the paths are defined


Now when we define two out of the four possible path choices — between the pair of LSSs — then this is what is going to happen:

� We define the 0000 ---> 0080 and 0004 ---> 0084 combination. So when asking for information about these two link-direction sets, we see what is shown in Figure 4-4.

� The 0080 ---> 0000 and 0084 ---> 0004 path combinations are not configurable. So when asking for information about these two link-direction sets, we see what is shown in Figure 4-5.

Figure 4-4 Paths information of valid paths


Figure 4-5 Paths information for links not used

4.2 Building consistencyIn this section we present the topics that have to do with building consistent data at the recovery site.

Global consistencyYou have global consistency at the recovery site, when you have consistent data across all the volumes of an individual secondary LSS, as well as consistent data across several application-related secondary LSSs. The scope of the consistency needed will depend on the application write operations characteristics, as well as on how the data files are distributed over the application site volumes.

When an application does dependent writes, the writes at the primary site volumes are done in a sequence — the dependent write comes upon having successfully completed some previous write operation. In order for the data at the recovery site to be consistent, the sequence of the updates should be the same on the secondary volumes.

This is especially important when error conditions arise that affect only some of the mirrored pairs — or when in a disaster situation, you do not want any dependent write to be mirrored, if the previous write has not been mirrored. This would made the secondary data inconsistent, in the event of needing it for recovery.


4.2.1 Consistency groupsConsistency grouping provides the ability to temporarily queue the write operations — to all PPRC volumes — on a single LSS pairing when an error occurs. If a volume pair error occurs that prevents PPRC from updating the secondary volume, the pair is suspended; and the LSS will enter the extended long busy (ELB) condition when a new write operation is tried. During this short ELB interval, the write operations to the primary LSS volumes are queued. At the same time, notification alerts are sent to the host system. Using automation triggered from the alert messages, during this ELB interval you can freeze the target set of application related LSSs — so you stop all updates at the recovery site, and save at a consistent checkpoint.

This consistency grouping timer function, gives you time to issue automated commands when a pairing has an error — by putting the primary LSS into ELB state. For z/OS systems this ELB state is issued by the Error Recovery Procedures (ERP), and indicated in the IEA494I message, which indicates that a pairing is changing status, and a long busy state has been instigated. As already mentioned, this ELB state temporarily queues all write operations to the PPRC pairs, on the affected source and target LSS pair.

Figure 4-6 summarizes the characteristics of consistency grouping.

Figure 4-6 Consistency group

The ELB time-out value can be displayed and changed from the Logical Subsystems panel of the ESS Copy Services Web user interface, by selecting the desired LSS and then clicking the Properties button. The Logical Subsystems Properties panel is displayed, and there — in the PPRC consistency group time out field — you can check or modify the time-out value.

Consistency grouping:Enables the Extended Long Busy (ELB) condition Enabled when defining the paths between primary and secondary LSSs.Triggers the ELB condition for SYNC pairs when suspended because of errors

Extended Long Busy Errors in a volume pair, result in the pair suspended and the ELB instigatedWhile in ELB all write operations to the affected primary LSS are temporarily queuedMeanwhile PPRC sends alert messages to host operating system, for suspended pairThe alerts can be used to trigger automated operations to create point of consistencyELB time-out (by default 2 minutes) causes the ELB to end, the writes are resumed

Consistency-group-created operation allows writes to resume before ELB time-out expiresRUN option of CGROUP command allows writes to resume before ELB time-out expires

ELB time-out can be modified from the Logical Subsystems panel, when using the ESS Copy Services Web user interface

Consistency grouping is valid to create point of consistency when volumes are in SYNC mirroring. Not necessary when volumes are in XD relationship

PPRC-XD always keeps fuzzy copy at the recovery siteXD pairs become consistent only when all updates are transmitted to secondary by catch-up while application writes are quiescedGives time for multiple freeze operations in order to reach global consistency


On the ESS, the consistency grouping is enabled by selecting the corresponding option — when the paths between source and target LSSs are defined. This is explained in the following sections.

4.2.2 Defining consistency groups using TSOWhen working with TSO, you will be defining the paths —between source and target LSSs— by means of the CESTPATH command. At this time, you optionally enable the consistency grouping for the LSS pair — specifying the parameter value CGROUP(YES).

Figure 4-7 CGROUP parameter when establishing a path

In Figure 4-7 you can see the syntax of the CESTPATH command, and the CGROUP parameter highlighted, with its two mutually exclusive values: YES or NO. For complete information on the CESTPATH command and its parameters, please refer to the publication z/OS DFSMS Advanced Copy Services, SC35-0428.

4.2.3 Defining consistency groups using the Web user interfaceWhen working with the ESS Copy Services Web user interface (WUI), you commence from the Paths panel. From this panel — once the target and source of the PPRC path have been selected — you right-click the highlighted target LSS, to pop-up the Task Wizard. In the Task Wizard you select the task you want to perform (in this case, you are establishing a path). So you select Establish paths and then you click Next (see Figure 4-8).

Note: With PPRC-XD there is no need to enable the consistency group option, in order to build consistent point-in-time copies of the data at the recovery site. Please refer to 4.2.4, “Freezing operation” on page 53 for further discussion.

CESTPATH DEVN(device_number) PRIM(ssid serialno lss) SEC(ssid serialno lss)

LINK( linkaddr )

NO NO CGROUP( YES ) RESETHP( YES )


Figure 4-8 Establishing a PPRC path

Then the Select path options window pops up, and here you check the PPRC consistency group option (see Figure 4-9).

Figure 4-9 Establish PPRC path with consistency group

For complete information on how to define consistency groups using the ESS Copy Services Web user interface, please refer to the publication IBM TotalStorage Enterprise Storge Server Web Interface User’s Guide, SC26-7346.


4.2.4 Freezing operationWith a freeze operation you can stop the write activity — on all the active PPRC primary and secondary volumes — of a given source and target LSS pair. This function enables you to maintain secondary volume update consistency. It affect all the LSS volumes that are in a PPRC state of active copy process: duplex, duplex pending SYNC, or duplex pending XD states. It does not affect the suspended and simplex volumes that may be under the LSS.

The freeze function operates on LSS pairs that have been defined with (or without) the consistency group option enabled. The freeze operation has three effects:

� The paths that connect the pair of LSSs being frozen are terminated

� The active volumes under the frozen LSS pair are suspended. This state transition (to suspended) is then communicated to the host with alert messages. These alert messages can be used by automation routines to trigger other recovery operations.

� If the consistency group option was enabled at path definition time, then the ELB condition is instigated, so the write activity to the primary LSS is temporarily queued. During this ELB interval, other automated operations can be triggered; for example, freezing other application-related LSS pairs.

When several freeze operations are executed within the ELB time-out interval — for all the application-related LSS pairs — then you can insure a global consistent checkpoint at the recovery site. For this to be possible, you should have defined all these related LSS pairs associations with the consistency group option.

When you suspend a pair of volumes — whether specifying the source or the target LSS — you will observe that both volumes of the pair, will appear as suspended after the operation. Both volumes are still linked by existing logical paths. But this is not the case when you do a freeze operation. For this situation, only the primary volume shows as suspended. The secondary volumes will appear as an active secondary.

When freezing volumes in duplex state, the secondaries will be a consistent copy. When freezing volumes in duplex pending XD — or duplex pending SYNC state — then the secondaries will be in an indeterminate point of consistency.

After a freeze operation — on a consistency group enabled LSS — in order to remove the resulting ELB condition, and restart the updates to the primary volumes, you have to execute a consistency-group-created task (when using the WUI), or a CGROUP command with the RUN option (when using TSO commands). These operations close the time-out window, and allow the writes to the primary volumes to resume. But the affected pairs will remain suspended, and the paths will not be automatically reestablished. This has to be done.

Before reestablishing the paths, in order to reestablish the pairs, you may perform a FlashCopy of the secondary volumes —because at this moment, they are holding a globally consistent point-in-time copy of the data. Figure 4-10 summarizes the characteristics of the freeze operation.

Note: With PPRC-XD there is no need to enable the consistency group option. As you will see later in the examples, when building a consistent point-in-time copy on the XD secondaries, you will need to have the application writes already quiesced — and this makes the ELB time-out window unnecessary.


Figure 4-10 Freeze and resume operations characteristics

4.2.5 Freeze operation with TSOWhen using TSO, the freeze operation —and the end-of-freeze operation— are done with the CGROUP command —using either the FREEZE parameter or the RUN parameter, respectively.

In Figure 4-11 you can see the syntax of the CGROUP command with the two mutually exclusive parameter values: FREEZE or RUN highlighted. For complete information on the CGROUP command and its parameters, please refer to the publication z/OS DFSMS Advanced Copy Services, SC35-0428.

Figure 4-11 FREEZE and RUN options of the CGROUP command

The freeze operation works on an LSS basis: Duplex, duplex pending XD, and duplex pending SYNC volumes are frozenNon PPRC (simplex), and suspended pairs in the same LSS are not frozen

The primary LSS stops propagation of updates to the secondary LSS:Removes the paths between the primary and secondary LSS pair Suspends the volumes pairs

Secondary volumes that were in duplex state will have a consistent point of dataSecondary volumes that were in duplex pending XD or duplex pending SYNC will have an indeterminate consistency

If LSSs were consistency group enabledThe LSS is put in the ELB condition when the first write comesThe ELB prevents updates to the primary volumes during the default 2 minutes time-out interval. Updates are temporarily queued.

Time-out interval will end if consistency-group-created task is invokedTime-out interval will end if CGROUP command with RUN option is executedTime-out interval value (2 min) can be changed with the Web user interface

To resume the mirroring activity to the secondary volumes:The paths must be reestablishedThe pairs must be reestablished

CGROUP DEVN(device_number) PRIM(ssid serialno lss) SEC(ssid serialno lss)

FREEZE

RUN


4.2.6 Freeze operation with the Web user interfaceWhen using the ESS Copy Services Web user interface, the freeze operation is done by specifying the Freeze PPRC Consistency Group task option from the Task Wizard — with the LSS pairing already selected, from the Logical Subsystems panel (see Figure 4-12).

Figure 4-12 Freezing using the Logical Subsystems panel of the Web user interface

You end the freeze condition — that is, you thaw the freeze — by selecting the PPRC Consistency Created task option (see Figure 4-12).

For complete information on how to define consistency groups using the ESS Copy Services Web user interface, please refer to the publication IBM TotalStorage Enterprise Storge Server Web Interface User’s Guide, SC26-7346.


Chapter 5. PPRC Extended Distance implementation and use

In this chapter we explain various ways of exploiting the PPRC Extended Distance capabilities. We also discuss some proposed implementations of solutions, and provide a further positioning of this new function.

5


5.1 PPRC-XD point-in-time consistencyThe XD option brings more functionality into the PPRC environment, changing the paradigm that PPRC is only synchronous. This new capability of PPRC will be used in different implementation scenarios than those of the traditional synchronous PPRC.

With PPRC Extended Distance, the secondary volumes are a continuous fuzzy copy of the primary volumes, where the application updates are being done. This fuzzy condition of the secondary volumes demands that — in order to use them for application recovery — you need to make them point-in-time consistent at both the individual volume and the application level.

Volume and application level consistencyConsistency at the volume level means that — for a selected point in time — the secondary volume mirrors exactly what the primary volume was at that point in time. Consistency at the application level means global consistency across all the volumes upon which the application does updates. This is to say that — for a selected point in time — the whole set of application related secondary volumes will mirror exactly what the primary set of application related volumes were at that point in time.

5.2 PPRC-XD for application recoveryAs a general approach, you will find that automation will be of great help, when managing mirroring environments — such as a PPRC-XD based solution. With PPRC-XD, software automation will help you control the stepped go-to-SYNC transition that you will be doing when going from XD to SYNC.

The go-to-SYNC catch-up transition requires that you either temporarily quiesce all the write updates to the primary volumes — or do it when the application write activity is low. The recommended approach is to quiesce the application write activity, and wait until there are no more writes before the catch-up is triggered. However, consider that at some point, you will have to bring all the application writes to a stop, in order to build point-in-time consistency.

Consequently, when scheduling the quiesce of your application writes, you will also be introducing some lead time, to build consistency on the recovery data. This planned outage can be minimized when properly automated, and the whole process can prove to be very efficient when recovering from a disaster situation.

When making your secondary volumes fully synchronous with their respective primaries, the commanded go-to-SYNC approach is not the only one you can use. As already mentioned, you can just quiesce the application writes and let PPRC-XD normal operation to do the catch-up, while you query PPRC to check for no more out-of-sync tracks.

5.2.1 Starting with a catch-up of the volumesThis example shows you the sequence of steps — when commencing with the catch-up of the application set of volumes. The result will be a point-in-time copy with global consistency across all the secondary volumes.

Note: The following sections present possible setups for getting a consistent point-in-time copy of your data at the recovery site. Nevertheless, bear in mind that the optimum setup — variations of steps, sequence and timings — will be application dependent, and you will be able to determine them by testing when doing your actual implementation.


Catch-up for all volumes and suspendPlease refer to Figure 5-1 to follow the explanation presented in this section.

Figure 5-1 Consistent PiT copy, starting with the catch-up

As the application is running and making updates onto the primary volumes, you may commence the catch-up by first doing a “go-to-SYNC and then suspend” of the application volumes. In order to suspend immediately upon reaching the duplex state — to avoid any synchronous update over the long distance — you can use the new option Suspend PPRC after establish —if you are working with the Web user interface (this new option, and how to use it, is described in 3.3.5, “Catch-up (go-to-SYNC) and suspend using the WUI” on page 36).

For TSO users, the “go-to-SYNC and suspend” operation should be automated. Because the volumes will be transitioning — from the duplex pending XD state to the duplex state — this change of status will be notified to the host with the IEA494I message. These messages can be intercepted and acted upon by automation routines to suspend the volumes as they reach the duplex state.

The command to go-to-SYNC is described in 3.3.2, “Catch-up (go-to-SYNC) using TSO” on page 33, and in 3.3.5, “Catch-up (go-to-SYNC) and suspend using the WUI” on page 36).

You can monitor the volumes to detect when there are no more out-of-sync tracks. The queries to PPRC report the number of out-of-sync tracks/sectors for the volume pairs. If you are working with TSO, you can use the CQUERY command (please refer to 3.4.1, “CQUERY command” on page 38).

primaryprimary

primaryprimary

primaryprimary

primaryprimary

B4B4

A3A3

A4A4

B3B3secondarysecondary

B2B2

secondarysecondary

A1A1

secondarysecondary

B3B3

secondarysecondary

A3A3

secondarysecondary

B3B3

secondarysecondary

A3A3

t 2t 2tim

e

t 1t 1t i t i mm e e

reestablishpaths and XD pairs(resync)

catch-up window

terciaryterciary

A3A3

terciaryterciary

B3B3

t 4t 4quiescewindow

updated tracks onlyre synchronize(copy out-of-sync only)

susp

ende

d

non-synchronousmirroring isresumed

susp

ende

d

andfreeze

application writes can be resumed as soon as the freeze is done

dupl

exdu

plex

secondarysecondary

B4B4

secondarysecondary

A4A4

dupl

exdu

plex

pend

ing

XD

pend

ing

XD

individual catch-up

go-to-SYNCoperation

susp

ende

dsu

spen

ded

t 3t 3

volumes finishing the catch-up at differenttimes - the application still doing writes toprimaries - secondary volumes not globallyconsistent

secondary volumes witha globally consistentpoint-in-time copy

FlashCopyFlashCopy

+ + t 3 t 3

t 4t 4

t 3t 3

Chapter 5. PPRC Extended Distance implementation and use 59

If you are using the Web user interface, then you can use the rsQuery command (please refer to 3.4.3, “rsQuery command” on page 39). Also, the rsQueryComplete command can now be used, with a finer granularity, to detect more accurately when the SYNC condition has been reached (please refer to 3.4.2, “rsQueryComplete command” on page 38).

Upon finishing this “go-to-SYNC and suspend” step, you have a set suspended secondary volumes, which individually have a consistent copy of their primary suspended counterpart. But as you can see from Figure 5-1, the point-in-time is not consistent across all the application secondary volumes. The application was doing write updates to the primary volumes, and these updates — as volumes were individually finishing their catch-up, and suspending at different moments — were not mirrored, but were tracked by PPRC on the primary volume’s bitmaps.

Quiesce application writesIn order to reach global consistency across all the secondary volumes that make the application set — that is, a consistent point-in-time (PiT) — you should quiesce the application writes. This quiesce will allow for a new re-synchronization of the volumes, but this time reaching a global consistent point-in-time (PiT).

The need of a quiesce, in order to build consistency at the recovery site, is a trade off against the write performance penalty inherent to a long distance synchronous implementation, which would not require the quiesce.

Build consistent PiT and restart application writesWith the application not doing any writes, you now re-synchronize (copy out-of-sync tracks only / RESYNC) the suspended pairs.

One alternative could be to re-synchronize the pairs, and then suspend them again, immediately upon reaching the duplex state. This operation can be implemented creating a major task that groups several individual suspend tasks, if managing with the CLI — or, by creating a TSO command list, that executes several individual suspend commands, one for each of the volume pairs in consideration.

A more efficient way of doing this is to re-synchronize the pairs and — once they reach the duplex state — then freeze the respective LSS volume set. This will simultaneously suspend all the volume pairs within the affected LSS, and also terminate the paths between the LSSs.

Either of the two alternatives leaves the secondary volumes with a globally consistent PiT copy of the primary volumes — and also, either of the two alternatives leaves the volumes suspended. This suspended state enables you to restart the application write activity onto the primary volumes, while you FlashCopy the secondary volumes that now have a consistent PiT copy.

Restart application writesThe application write activity can be restarted as soon as the freeze is done. These updates will not be propagated to the secondary volumes; volume pairs are suspended, and LSS paths are discontinued. In the meanwhile, PPRC will keep track of the primary updated tracks — with the associated bitmaps — to be able to do an incremental re-synchronization, when the XD pairs are later reestablished.

Note: At PPRC synchronous supported distances, this quiesce could be skipped. Refer to 5.2.2, “PPRC-XD PiT within the synchronous range” on page 62 for further explanation.


FlashCopy consistent checkpoint dataWith the PPRC pairs suspended — and with the application write activity already resumed — the consistent PiT secondary copy can be FlashCopied onto a tertiary set of volumes.

Re-establish the XD pairsOnce the FlashCopy relationship is established, you can bring the primary and secondary volumes out of the suspended state. Bringing them into the duplex pending XD state — with an incremental re-synchronization (copy out-of-sync only) — will resume the mirroring onto the secondary volumes.

Before reestablishing the volume pairs XD relationships, you must redefined the LSS paths that were terminated by the freeze operation.

Now the procedure is complete — the consistent PiT copy is on the tertiary volumes — and we are ready for the next checkpoint.

Figure 5-2 summarizes the characteristics of the steps involved in the previously described setup for consistency building.

Figure 5-2 Building global consistency considerations

Catch-up will leave secondary volumes with individual volume consistency Start a "go-to-SYNC and suspend" operation to synchronize the volume pairsQuery PPRC to detect when the duplex-then-suspended state has been reached

Quiesce application writesPrevents any synchronous writes over long distances while building consistencyQuiesce step can be skipped if operating within synchronous supported rangeQuery PPRC for out-of-sync tracks: when none, volumes are synchronized and no more updates are been done

Build global consistency on secondary volumes and freezeRe synchronize the pairs

incremental copy of the changed tracks onlyquery PPRC to detect when the duplex state is reachednow the secondaries have a consistent PiT

Immediately freeze so the pairs are suspended and paths terminated

Application writes can be resumedPairs are suspended, no propagation of updates to secondariesPairs are suspended, PPRC keeps track of primary updates

FlashCopy saves this secondary consistent PiT copy

Resume PPRC-XD mirroringPaths are reestablishedXD pairs are reestablished


5.2.2 PPRC-XD PiT within the synchronous rangeAt PPRC synchronous supported distances, the previous setup arrangement could be done differently — skipping the quiesce and the re-synchronization steps. If you trade-off some temporary application write performance, then the procedure would be limited to:

1. Trigger a catch-up, commanding PPRC-XD go-to-SYNC, but not suspending. This should be done when the application write activity is low — the catch-up will be short, and the application write performance impact less notorious.

2. Query for all volumes in duplex state — secondary copy becomes globally consistent.

3. Immediately upon all the volume set reaches the duplex state — freeze.

4. Now you have a secondary consistent PiT copy, that you can FlashCopy. Also at this moment, the application is already relieved of the write penalty, because volumes were left suspended when the freeze was done.

5. Finally you reestablish the paths, reestablish the XD pairs — and now, the non-synchronous mirroring starts again until the next catch-up.

For this particular implementation —where the application is not quiesced— you may consider to enable the consistency group option of the related LSSs. Whether to enable —or not— this option, will depend on the particular application write activity characteristics.

5.2.3 Starting with the quiesce of the applicationAs already mentioned, you can setup different arrangements in order to build global data consistency at your recovery site. At the time of actual implementation at your installation, you will be testing to determine which is the most efficient and less disruptive implementation.

The example presented in Figure 5-3 illustrates an arrangement where you first start by waiting for the application writes to quiesce completely — and then, let PPRC-XD complete the catch-up of the volumes.

The now more granular PPRC queries will allow you to determine with more accuracy, when there are no more out-of-sync tracks. At this moment your secondary volumes have a global PiT consistent copy of the primary application volumes.

As you see in this example — because you already achieved a globally consistent checkpoint at the recovery site — there is no need to re-synchronize the pairs, as in the previous example. You proceed to freeze the LSS, which still had the volumes in duplex pending XD state, so you can resume the application as soon as possible. Also, upon completing the freeze, you proceed to do a FlashCopy onto the tertiary volumes.


Figure 5-3 Consistent PiT copy, waiting for application writes to quiesce

As soon as the FlashCopy relationship is established, you can now re-initiate the XD mirroring onto the secondary volumes. To do so, you reestablish the paths and the suspended pairs.

5.3 PPRC-XD solutions and positioningIn this section we further position PPRC Extended Distance, from a functional and application performance perspective.

5.3.1 Functional positioningPPRC Extended Distance brings the possibility of doing PPRC mirroring over extended distances, thus preserving the applications write operations’ performance. Its non-synchronous characteristic — with some periodic go-to-sync operations to build consistency PiT copies at the recovery site — allows you to better exploit your available bandwidth capacity, and to protect a larger part of your data.

With PPRC Extended Distance, the ESSs hosting the primary and secondary LSSs can be separated by distances well beyond the 103 km supported with PPRC synchronous transmissions: continental distances, only limited by the channel extender and network technology capabilities. As synchronous mirroring cannot occur over long distances, without a serious application performance impact, the best trade-off is to accept the periodical brief application quiesces, in order to build consistent PiT copies of the data.

primaryprimary

primaryprimary

primaryprimary

primaryprimary

B3B3

A2A2

A3A3

B2B2secondarysecondary

B2B2

secondarysecondary

A2A2

secondarysecondary

B2B2

secondarysecondary

A2A2

t 2t 2time

t 1t 1t i t i mm e e

reestablishpaths andXD pairs(resync)

quiesce window query for zero out-of-sync

re synchronization step not needed

non-synchronousmirroring is resumed

freeze

dupl

exdu

plex

pend

ing

XD

pend

ing

XD

volumes catch-upto the same pointin time

terciaryterciary

A2A2

terciaryterciary

B2B2

wait for applicationwrites toquiesce

susp

ende

dsu

spen

ded

dupl

exdu

plex

pend

ing

XD

pend

ing

XD

secondarysecondary

B3B3

secondarysecondary

A3A3

t 3t 3

secondary volumes with a globally consistent point intime copy of the primaries

FlashCopyFlashCopyapplication writes can be resumed as soon as the freeze is done

t 2t 2

++t 2t 2


PPRC Extended Distance provides a fuzzy copy of the data while application updates are happening. As a consequence, it can be considered a non-synchronous data mover, with the least time to catch-up. PPRC Extended Distance recovery implementations — where the mirrored data is fuzzy, and consistency checkpoints have to be built — differ from recovery implementations based on the continuous availability of real-time consistent recovery data.

AutomationAutomated software procedures should be implemented, in order to efficiently execute the steps involved in building global consistency at the recovery site, in either of the following circumstances:

� For quiescing application writes and then resuming — when the ESSs are at distances larger than the acceptable synchronous range

� When trading-off a slowdown of the application write performance — if the ESSs are within the synchronous supported distance

Please refer to 5.4.3, “Automation” on page 69 for further discussion on automation.

PPRC Extended Distance can be effectively used for:

� Data copy� Data migration� Off-site backups� Remote transmission of inactive database logs� Application recovery (if application can be periodically quiesced to create consistent PiT)� Mirroring in a mixed environment (sync and non-sync)

Figure 5-4 summarizes the functional characteristics that position PPRC Extended Distance.

Figure 5-4 PPRC Extended Distance functional positioning

Non-synchronous transfer of PPRC-XD enables current backup technologies to be implemented over continental with excellent performance

PPRC-XD introduces a non-synchronous mirroring technique in the open environments

PPRC XD provides a fuzzy copy at the recovery siteWrite sequence on the secondary volumes may not be the same as application sequence PPRC-XD can catch-up when consistent PiT copy needed

Application quiesce needed and thenEither command PPRC-XD "go-to-SYNC and suspend"Or wait for PPRC-XD to catch-up

Query PPRC-XD for out-of-sync tracksAutomation is recommended to make efficient implementations

Good for data migration; transmission of database logs; offsite backups; large data center migrations (combined with tapes); disaster recovery implementations (when quiesce of the application and a non-zero RPO are acceptable)

All with a good exploitation of the installed telecommunication bandwidth

Mixed option (at PPRC-SYNC range) to mirror more volumes for a given bandwidth Non-critical volumes mirrored with XD option with periodic catch-upsCritical volumes mirrored with SYNC option


5.3.2 Remote data migrationData migrations to a remote site can now be done with PPRC Extended Distance in a much shorter time, with the minimum possible disruption to the production process.

The application site volumes to be migrated can be receiving updates from the application, while keeping a fuzzy copy at the remote site. When an appropriate application checkpoint is reached, then the application writes are quiesced for a very short time, to let the secondaries catch-up. Eventually, the application will also have to be stopped — if it has to restart using what will be the migrated files — to re-initialize itself pointing to the migrated files in their new volume locations. In either case, as soon as the count of out-of-sync tracks is zero, the application can resume the writes onto the primary site volumes — or it can restart and switch to the just-migrated version of the volumes. Figure 5-5 illustrates a very simple setup for migration of an application’s data to a remote data center.

Figure 5-5 Data center migration

Using tapesA more sophisticated use of PPRC Extended Distance — complementing it with tapes — can be used when needing to migrate a huge amount of data (for instance, 100 TB) from a production site to a remote site, in an overnight schedule. The idea is to capture a fuzzy copy of the data onto tapes — after a synchronous relationship (PPRC-SYNC) with the NOCOPY option has been established, and immediately suspended. Tapes are sent to the remote site, and there, they are restored on the terminated secondaries. Then a PPRC-XD relationship is established between the volume pairs, to send the updates that occurred since the tape captures were done. An interesting characteristic of this solution is that it does not demand a huge telecommunication bandwidth, and has no application outage. This solution is explained with more detail in B.1, “Data migration for a large data center” on page 124.

Start Up PPRC-XDEstablish all volumes

At beginning of data center cutover windowQuiesce applications but allow PPRC-XD to continue to run

Allow PPRC-XD to catch up (time would be short)

Once PPRC-XD caught up, data is ready at remote location

Significantreduction in work and effortrequired to migratedata centers overlong distance

B

L

X

C

M

Y

ESS 2:MirroredData

Production Servers

Backup / AdminServers

ESS 1:ProductionData


5.3.3 Transmission of database logsDatabase recovery solutions based on switching active database logs — and transmitting the then inactive log and the bootstrap data set (BSDS) — can be efficiently implemented with PPRC Extended Distance. Once at the remote site, the log is applied onto the shadow database (see Figure 5-6).

Figure 5-6 Transmission of database logs

5.3.4 Off-site backupsEither for the backup of volumes after your installation’s batch window processing — or for point-in-time backups, that your installation may issue at appropriate application checkpoints — PPRC Extended Distance provides an efficient solution with less bandwidth demand.

Batch window volumes backupThe backup steps included in any production processing environment can now be done to remote sites — and in a much faster manner — using PPRC Extended Distance. The volumes holding the batch production output files can be mirrored with PPRC-XD while the processing jobs are updating them. As each job finishes, it stops doing updates, so — in a very short time — the job’s set of volumes will automatically catch-up by their own. You can query these volumes to check when there are no more out-of-sync tracks, and then proceed with a FlashCopy in order to have a tertiary backup copy. This use of PPRC Extended Distanceof will allow you to substantially shrink your batch processing window.

Point-in-time backups and split mirroringFor point-in-time (PiT) backups and split mirroring implementations, PPRC Extended Distance minimizes the application quiesce interval — just for the short XD catch-up — as compared to other implementations. At the same time, PPRC Extended Distance is a solution with minimum telecommunication bandwidth requirements, as compared to synchronous techniques. These characteristics make it possible to do more frequent PiT backups than with other implementations; and so, the recovery point objective (RPO) improves substantially.

#3: wait for catch-upquery for ceroout-of-sync

duplex pending XD

ESS 2:MirroredData

Production Servers



log1

log2

log3

log1'

log2'

log3'

D#3: Shadow process applies log1' to shadow database

#1: Application dynamically switches from log1 to log2


Figure 5-7 summarizes the basic high level setup for these types of implementations, when used over long distances.

Figure 5-7 PPRC-XD split mirror implementation

The PiT backups and split mirroring implementations remain the same — but with PPRC Extended Distance, there are some additional benefits. Besides sending the copied data to remote locations, you will also make better use of your existing bandwidth; and so, the amortization of your communication costs will be done sooner.

5.3.5 Mixed environment (XD and SYNC)The performance of the application write operations is tightly related to the available bandwidth of the telecommunications infrastructure, and this is more critical when the transmission is synchronous. Because of this, you might consider using this expensive bandwidth resources to mirror a mix of — a few critical volumes using the SYNC option — while the rest of the data is mirrored with the XD option. This consideration can be extended for the case where you need to cover more volumes, under the PPRC protected range, while not being able to increase your bandwidth.

5.3.6 Application performance positioningAs the distance between ESSs increases, synchronous PPRC response time is proportionally affected — and this negatively impacts the application performance. When implementations over extended distances are needed, PPRC Extended Distance becomes an excellent trade-off solution.

You can estimate PPRC Extended Distance application impact, as that of the application when working with PPRC suspended volumes. For the ESS, there is some more work to do with the XD volumes — as compared to the suspended volumes — because with PPRC-XD, the changes have to be sent to the remote ESS. But this is a negligible overhead for the application, as compared with the typical synchronous overhead.

Establish volume pair A-B in PPRC-XD relationship

Run in XD mode until PiT consistent copy needed

When consistent copy needed, quiesce the application

Wait for PPRC-XD to catch-up and then freeze (or suspend)

Allows pairs to synchronize

Query for zero out-of-sync tracks/sectors

With the pairs suspended the application can resume

FlashCopy secondary B to tertiary C (no impact to host)

When FlashCopy is initiated, resume XD mirroring

Go back to Step 2.

SECONDARYSECONDARY

PRIMARYPRIMARY

PPRC OVER

EXTENDED

DISTANCE

CHAN

EXT

CHAN

EXT Vol B

FlashCopy

Vol C

Vol A

Sequence of operation


Figure 5-8 illustrates how the different PPRC methods are positioned, in respect to the distance factor. PPRC Extended Distance is a non-synchronous solution for implementations over long distances — continental distances, well beyond the synchronous 103 km supported distance. With PPRC Extended Distance, in order to get consistent PiT copy of the data, you do a catch-up transition. The traditional PPRC (PPRC-SYNC) is a synchronous solution, for implementations within metropolitan areas.

Figure 5-8 Flexible continuum of PPRC and PPRC-XD capabilities

5.3.7 Initial establish for a large number of volumesWhen having to establish several copy pairs at once (in synchronous mode), you may find PPRC Extended Distance a very helpful solution.

If your PPRC setup involves several volume pairs that need to be established in synchronous mode at the same time, then, when triggering this process, some volume pairs will become synchronized before others. In this situation, where some volumes are already synchronized while others are in the process of synchronization, the performance of the whole transition (and of the application) can be affected. If this is the case, you may find a good solution by initially establishing the PPRC volume pairs as extended distance pairs (XD option), and once the bulk transfer is near to completion, then you transition the volume pairs to synchronous (SYNC option).

5.4 PPRC-XD implementation considerationsThe PPRC Extended Distance option brings new possibilities for mirroring implementations. Its unique technique brings also some considerations when planning its implementation that we review in this section. Figure 5-9 summarizes these considerations.

PPRCSYNCdist 0

km

PPRCXDAny

distance

PPRCSuspended

No PPRC

PPRCSYNC

dist < 103 km

CATCH-UP methodology:

PPRC-XD transitions to PPRC-SYNC

< = = No data transmission to remote = = => Native Throughput

Select PPRC-SYNC and/or PPRC-XD depending on distance and data synchronization requirements

<= = = = = = Sync Data Copy = = = = = = > Zero Data Loss Metropolitan area Application impact

<= = Non-synch = => Long Dist Copy Minimal application impact

<= = PPRC tracks = => incremental changes for later transmission


Figure 5-9 Implementation considerations

5.4.1 More data can be PPRC protectedWhen planning your data integrity requirements, you now can think of more data to be protected. PPRC Extended Distance has less bandwidth demand, and minimum application write performance impact — independent of the implemented distance. This allows you to include more of your data under the PPRC protection range.

5.4.2 Consistency managementWith PPRC Extended Distance, the management of consistency is done differently than with synchronous PPRC. With PPRC Extended Distance, you now have remote fuzzy copies of the primary volumes, and so the dependent writes on the remote volumes are not necessarily in the same sequence, as they were done by the application on the primary volumes.

This characteristic makes you plan for the applications’ recovery implementations in a different manner. For example, you have to plan for application checkpoints to build consistency at the recovery site data; whereas in a synchronous implementation, you will not need these checkpoints.

5.4.3 AutomationAs already mentioned, to build consistency at the recovery site, you may be doing short planned outages of the application, in order to quiesce the write activity upon the primary volumes. These data consistency building windows involve several operations, on the selected set of volumes. To make this procedure more efficient, we recommend the implementation of automation routines.

When implementing automation scripts — on open servers with CLI support — you will find the rsQueryComplete command very useful, as it signals the completion of the task. Please refer to page 86 for a discussion and illustration on the rsQueryComplete command, and to 3.4.2, “rsQueryComplete command” on page 38.

In z/OS and OS/390 environments, when automating, you will find very useful the state change messages that the system issues when PPRC relationships transition from one state to a different one. The IEA494I and IEA494E messages can be detected by automation routines, and used to suspend secondary PPRC operations, using the CGROUP command with the FREEZE parameter.

5.4.4 Bandwidth managementPPRC Extended Distance uses the available bandwidth in an adaptive and flexible way, more efficiently exploiting the available resources. You should plan the bandwidth based upon the amount of write operations, and the minimum time needed for the go-to-SYNC transition.

Think PPRC-XD protection for a wider data scope

Now two consistency management objectives: SYNC and XD

Automation recommended for efficient catch-up windows

Bandwidth management more flexible


5.5 PPRC connectivityIn this section we review the connectivity and distance considerations for PPRC implementations. These considerations are summarized in Figure 5-10.

Figure 5-10 Connectivity, distance and support considerations

PPRC connectivity capabilitiesPPRC can be used with the following connectivity technologies:

� ESCON Directors� Channel Extenders over Wide Area Network (WAN) lines� Dense Wave Division Multiplexors (DWDM) on dark fibre

5.5.1 PPRC supported distancesThe supported distances vary according to the method of PPRC mirroring.

Synchronous transmissionWhen PPRC is doing synchronous transmissions (PPRC-SYNC), the supported distance is 103 km. Slighter longer distances than 103 km are possible. For this an IBM RPQ (Request for Program Quotation) must be submitted.

Non-synchronous transmissionWhen PPRC is doing non-synchronous transmissions (PPRC-XD), the distances can be continental distances — only limited by the channel extender and network technology capabilities. Channel extenders are needed to implement continental distances.

PPRC can use the following connection capabilities:ESCON DirectorsChannel Extenders over Wide Area Network (WAN) linesDense Wave Division Multiplexors (DWDM) over dark fibers

Synchronous (SYNC) maximum supported distance is 103 KmFor slighter longer distances an RPQ must be submitted

Non-synchronous (XD) can be used over continental distancesOnly limited by the network and channel extender technology capabilities

The connectivity infrastructure is transparent to PPRC

Evaluation, qualification, approval and support of PPRC configurations using channel extender products, is the sole responsibility of the channel extender vendor

The channel extender vendors and DWDM vendors should be consulted regarding prerequisites when using their products

A complete and current list of PPRC supported environments, configurations, networks, and products is available at:

http://www.ibm.com/storage/hardsoft/products/ess/supserver.htm


5.5.2 PPRC channel extender supportChannel extender vendors connect ESS PPRC systems via a variety of Wide Area Network (WAN) connections, including Fibre Channel, Ethernet/IP, ATM-OC3 and T1/T3.

When using channel extender products with PPRC, the channel extender vendor will determine the maximum distance supported — between the primary and secondary ESS. The channel extender vendor should be contacted for their distance capability, line quality requirements, and WAN attachment capabilities.


http://www.storage.ibm.com/hardsoft/products/ess/supserver.htm

The channel extender vendor should be contacted regarding hardware and software prerequisites — when using their products in an ESS PPRC configuration. Evaluation, qualification, approval and support of PPRC configurations using channel extender products is the sole responsibility of the channel extender vendor.

5.5.3 PPRC Dense Wave Division Multiplexor (DWDM) supportWave Division Multiplexing (WDM) and Dense Wave Division Multiplexing (DWDM) is the basic technology of fibre optical networking. It is a technique for carrying many separate and independent optical channels on a single dark fibre.

A simple way to envision DWDM is to consider that at the primary end, multiple fibre optic input channels — such as ESCON, Fibre Channel, FICON, or Gbit Ethernet — are combined by the DWDM into a single fibre optic cable. Each channel is encoded as light of a different wavelength. You might think of each individual channel as an individual color: the DWDM system is transmitting a “rainbow”. At the receiving end, the DWDM fans out the different optical channels. DWDM — by the very nature of its operation — provides the full bandwidth capability of the individual channel. As the wavelength of light is — from a practical perspective — infinitely divisible, DWDM technology is only limited by the sensitivity of its receptors, as the total aggregate bandwidth possible.



The DWDM vendor should be contacted regarding hardware and software prerequisites — when using their products in an ESS PPRC configuration.

5.6 PPRC-XD requirementsThe ESS ships with IBM Licensed Internal Code (LIC) that is licensed for use by a customer on a specific machine, designated by serial number, under the terms and conditions of the IBM Customer Agreement or the IBM Agreement for Licensed Internal Code.

PPRC Extended Distance is supported on the ESS models F10 and F20, and requires LIC level 1.5.2 or later. PPRC Extended Distance is provided with the PPRC optional feature on the ESS — feature code 182x.




It is managed via the ESS Copy Services Web user interface, provided through the IBM TotalStorage Enterprise Storage Server Specialist. It is also managed using the ESS Copy Services command line interface (CLI) — in selected open systems environments — and, using TSO commands, in the z/OS and OS/390 environments.



You should refer to this site for the most current and complete support information, when planning to use PPRC Extended Distance.



Chapter 6. Using the WUI and the CLI to manage PPRC-XD

In this chapter we provide an example for doing a consistent point-in-time (PiT) backup, with PPRC Extended Distance, using the following ESS Copy Services interfaces:

� ESS Copy Services Web user interface (WUI)� ESS Copy Services command line interface (CLI)

Examples using TSO commands are provided in Appendix C, “PPRC-XD Managing in the TSO environment” on page 131.

This chapter describes how the sequence of ESS Copy Services tasks is executed. Then the creation of the corresponding tasks is detailed in Appendix A, “ESS Copy Services creation of tasks” on page 95.

6


6.1 ESS SpecialistThe IBM TotalStorage Enterprise Storage Server Specialist (ESS Specialist) is the initial interface to the ESS, when using a Web browser (see Figure 6-1).

Figure 6-1 Selecting the ESS Copy Services interface

The ESS Specialist can be accessed by any Web browser, as long as it supports Java Virtual Machine (JVM) — at a level which complies with the Java applets provided by the ESS, currently JVM 1.1.8.

For detailed support information and instructions on how to use the ESS Web user interface, please refer to the publication IBM TotalStorage Enterprise Storge Server Web Interface User’s Guide, SC26-7346. You can get this user’s guide from the Web by clicking Reference Information at:


This will take you to the technical support page of the ESS, from where you can download the publication.

To connect to the ESS Specialist interface, you can use the ESS Master console — or, provided your ESS clusters are connected to your intranet — you can access the ESS interfaces from any workstation on the intranet.



6.2 Creating ESS Copy Services tasksTo create an ESS Copy Services task, start by launching ESS Copy Services, as shown in Figure 6-1 on page 74. Then, when the ESS Copy Services applet is initialized, you will get the ESS Copy Services welcome panel (see Figure 6-2) — from where you can start configuring the ESS Copy Services tasks.

Figure 6-2 ESS Copy Services interface - Welcome panel

In 6.3, “Tasks for a PiT backup with PPRC-XD” on page 76, we present the sequence of ESS Copy Services tasks involved in the creation of a consistent PiT backup — when using PPRC Extended Distance. Later, in 6.4, “Using the CLI for a PiT backup with PPRC-XD” on page 82, we describe how this series of tasks are executed using the CLI.

In Appendix A, “ESS Copy Services creation of tasks” on page 95, you will find the description of how these tasks were created.

Note: When working with the Web user interface, all windows are applets. Therefore we recommend that you restart your browser periodically — especially after each completed task — to ensure the information displayed is current.

Also if you intend to change window size, do so prior to selecting ESS Copy Services. We recommend that you do not open additional browser windows while using the Web interface.

Note: Prior to creating tasks, make sure that the target volumes you define in the task are not already in use — for example, by another host. This will prevent data loss, or data corruption.

Chapter 6. Using the WUI and the CLI to manage PPRC-XD 75

As already described, the new duplex pending XD status — associated with PPRC Extended Distance — is represented by the ESS Specialist with a new set of icons, as you can see in Figure 6-3.

Figure 6-3 Source and target PPRC-XD volume pair icons

For a description of the whole set of icons that the ESS Specialist presents, please refer to the publication IBM TotalStorage Enterprise Storge Server Web Interface User’s Guide, SC26-7346.

6.3 Tasks for a PiT backup with PPRC-XDIn this section we present the sequence of ESS Copy Services tasks involved in the creation of a consistent PiT backup — when using PPRC Extended Distance over a long distance implementation (beyond the synchronous 103 km supported distance).

Then, 6.4, “Using the CLI for a PiT backup with PPRC-XD” on page 82 illustrates how this sequence of tasks is invoked using the CLI.

In this example, the sequence of steps to do a consistent PiT backup — using PPRC Extended Distance — is as follows:

1. Establish paths — no consistency group needed [EST_PATHS_V1].

2. Establish PPRC XD pair — initial copy of entire volume [EST_PPRC_XD_V1].

3. Establish PPRC SYNC pair (go-to-SYNC) and suspend — incremental copy of out-of-sync tracks only, and suspend [PPRC_SYN_SUSP_V1].

4. Quiesce the application writes — stop write activity on application volumes.

5. Re-synchronize PPRC pair — incremental copy out-of-sync tracks [RESYN_PPRC_V1].

6. Freeze — suspend LSS volume pairs, all at once, and drop paths [FREEZE_LSS12_13].

7. Resume application — write activity is resumed.

8. FlashCopy PPRC target — secondary copied onto tertiary [EST_FC_V1_W1].

9. Reestablish Paths — paths withdrew during freeze, now restored [EST_PATH_12_13].

10.Reestablish PPRC XD pair — copy out-of-sync tracks only [REEST_PPRC_XD_V1].

Let us now look at each step in more detail.

When reading this section:

To follow the sequence of steps illustrated in this section, you may find useful to refer to the descriptions in 5.2, “PPRC-XD for application recovery” on page 58, and in 5.3.4, “Off-site backups” on page 66. These previous sections discuss in detail the considerations involved in each of the steps of a consistent PiT backup procedure.

For the steps in this section — where an ESS Copy Services task is executed — the step is listed with the corresponding task name (in brackets) as when the task was created. This will help you correlate this information with that in Appendix A, “ESS Copy Services creation of tasks” on page 95


6.3.1 Establish pathsBefore commencing the PPRC-XD relationship —between the volume pairs— the paths between the corresponding LSSs must be defined.

Because application writes will not be happening when the moment arrives to build the consistent PiT copy —as you will see later when the freeze step is described— then this paths definitions do not need to have the consistency group option enabled. There is no need for ELB conditions.

In general, the consistency group option will not be necessary when working with PPRC-XD pairs —even when planning for unexpected error conditions— because the secondaries are always having a fuzzy copy.

The Establish paths task, establishes the paths between the LSS pairs. In Figure 6-4 the paths are established — illustrated by the three blue asterisks underneath the physical connection.

Refer to “Reestablish paths” on page 111 for a description of the creation of the task.

Figure 6-4 Establish paths without consistency group

6.3.2 Establish PPRC-XD pair — initial copyDuring this step (see Figure 6-5), the PPRC Extended Distance relationship between primary and secondary volumes is established, doing the initial full copy of the primary volumes — option Copy entire volume when selecting the copy options.

Figure 6-5 Establish PPRC-XD — Copy entire volume —

Upon execution of this task, the PPRC-XD pairs are established, and the volumes reach the duplex pending XD state. While in this state, the secondary volumes will hold a fuzzy copy of the primary volumes.

The amount of time needed to complete the initial copy of the volumes will be a function of the size of the volumes, the amount of volumes been copied, and the available bandwidth on the physical links.

Refer to “Establish PPRC XD pair — initial copy” on page 98 for a description of the creation of the task.


6.3.3 Establish PPRC-SYNC pair — go-to-SYNC and suspendWith the volume pairs already in an XD relationship (since some previous time, when initially established) with the secondaries holding a fuzzy copy of the primaries, then — at a scheduled point-in-time, preferably when the application write activity is low (or none) — a catch-up is commanded (go-to-SYNC), as shown in Figure 6-6.

Figure 6-6 Establish PPRC-SYNC pair — go-to-SYNC, and suspend

During this operation, PPRC does a fast incremental copy — that is, copy the out-of-sync tracks’ updates only. As soon as the volume pairs reach the duplex state, they will suspend, because you don’t want synchronous updates to happen.

This task transitions the state of the PPRC pairs from non-synchronous (duplex pending XD) to synchronous (first, duplex pending, and then, duplex state) followed by an immediate suspended state. This transition is illustrated in Figure 6-6, where the duplex state is shown in parentheses, because you will not be able to monitor this state — as the pair will be immediately suspended upon reaching the duplex state.

At the end of this task, the volume pairs are individually synchronized, and are in suspended state. But this is still not a consistent PiT checkpoint, from the application perspective.

Refer to “Establish PPRC SYNC pair — go-to-SYNC and suspend” on page 102 for a description of the creation of the task.

6.3.4 Quiesce the application write activityThe characteristics of this step depend solely on your applications’ quiesce peculiarities. When you can proceed with the quiesce checkpoints? How long they can be? How frequently they can be done? These characteristics will be totally application dependent. Even for some applications — like DB2 using the set log suspend — you may temporarily quiesce the write activity without having to stop the application.

In our example, the quiesce of the application write activity is needed for two reasons. One reason is, to be able to reach a global consistent point-in-time; this will happen when we execute the next two steps, the re-synchronization and the freeze. The other reason is that, during the next re-synchronization step, we do not want any synchronous transmission to occur — this is because in our example the implementation is over an extended distance.

Waiting for the application writes to quiesce: As already discussed, an alternative is to start the catch-up by quiescing the application write activity. In this alternative the synchronization can be accomplished just by leaving the pairs in XD mode, and waiting for PPRC-XD to finish its synchronization — that is very efficient. You will be querying to detect when the out-of-sync value is zero. At this point you would already have a consistent PiT copy, so the re-synchronization step — done later in our example — would not be necessary. Please refer to the discussions in 5.2, “PPRC-XD for application recovery” on page 58, and in 3.1.2, “PiT Implementation basics” on page 27.


If we were within the synchronous supported distance, then we could trade off some application write performance, and avoid the application quiesce. For further discussion please refer to Section 5.2.2, “PPRC-XD PiT within the synchronous range” on page 62.

6.3.5 Re-synchronize PPRC pairThe Re-sync PPRC pair task re-synchronizes the pair. When doing this re-synchronization, the pairs goes from the suspended state they were in — through an intermediate duplex pending state — and finally reaching the duplex state, with full synchronization.

In this process, only the modified tracks are copied — the tracks that changed in the interval that went from the end of the first catch-up, until the application quiesce was complete.

Figure 6-7 illustrates this transition of the pair, from suspend state (as they were left at the end of the first go-to-SYNC), through duplex pending (when the modified tracks only are copied), and then reaching the duplex state (with the volume pairs fully synchronized).

Figure 6-7 Re-synchronization of the pairs

In this step, there is no chance that synchronous transmissions occur when the duplex state is reached, because the application writes are already quiesced. So, an automatic suspend would not be required for this purpose. But, for the purpose of restarting the application write activity immediately, you will want to leave the just synchronized pairs in a suspended state — as the next step will do.

Refer to “Re-synchronize PPRC pair” on page 104 for a description of the creation of the task.

6.3.6 FreezeFreezing an LSS pair terminates the paths between the respective LSSs, and leaves the primary volumes within the affected LSS in a suspended state (see Figure 6-8). This transition affects — all at once — the whole set of volumes of the affected LSS. Please refer to 4.2.4, “Freezing operation” on page 53 for further discussion of the freeze operation.

Figure 6-8 Freeze

This operation is very efficient, because it allows you to manage the application set of volumes all together, taking action upon the related set of LSSs. If you are working with the WUI or the CLI, it is best implemented by creating and saving tasks — for all the related LSS pairs — and then grouping the tasks. If you are working with TSO commands, you may set up a command list with the set of freeze commands.


The illustration in Figure 6-8 on page 79 shows how the freeze operation works upon the LSS pair — dropping the paths between them — and putting all the volumes of the primary LSS in the suspended state. Because the freeze operation discontinues the paths between the affected LSSs — without any notification to the secondary LSS — then, for this secondary LSS, its volumes will not be able to reach a normal suspended state. Rather, they will remain in the duplex state they were in, before the freeze operation took effect, as was illustrated in Figure 6-8 on page 79.

If, instead of the freeze, we had issued a group of suspend tasks for each of the volume pairs, then the secondary LSS and volumes would have been notified — and would both show suspended as Figure 6-9 illustrates. Also, the LSSs paths would remain in effect.

Figure 6-9 Suspend PPRC pairs

Now that the freeze operation is done, you have reached a consistent secondary PiT checkpoint at the recovery site. At this moment you can proceed to resume the application write activity, and to do a copy of the secondary volumes onto a tertiary set of volumes.

Refer to “Freeze” on page 106 for a description of the creation of the freeze task.

6.3.7 Resume application write activityWith the secondary volumes already holding a consistent PiT checkpoint, and with the primary volumes in suspended state, you can immediately restart the application write activity. The secondary checkpoint is still saved, because no updates will be propagated — the primary volumes are suspended, and paths are discontinued from the freeze operation.

Besides, the updates on the primary volumes will be later transmitted to the secondaries, when the pairs are reestablished, because PPRC is keeping track of the changed tracks while volumes are suspended.

6.3.8 FlashCopy PPRC targetWith the FlashCopy PPRC target task, we generate a tertiary copy of the PiT consistent secondary volumes. This tertiary copy can then be used for backup to tape, or for other purposes, for example, data mining. This FlashCopy task can be immediately invoked, upon completion of the freeze step, and while the application writes are resumed.

The illustration in Figure 6-10 shows the PPRC target volume — still in the duplex state as from the previous freeze operation — that becomes the source for a FlashCopy operation. Also the PPRC primary volume shows suspended, as from the previous freeze operation.


Figure 6-10 FlashCopy of the PPRC target

Refer to “FlashCopy PPRC target” on page 108 for a description of the creation of the FlashCopy task.

6.3.9 Reestablish pathsSInce the paths were dropped by the freeze operation, now they must be reestablished — previous to reestablishing the PPRC-XD volume pairs.

The Reestablish Paths task, reestablishes the paths between the LSS pairs. In Figure 6-11 the paths are reestablished — illustrated by the three blue asterisks underneath the physical connection.

Figure 6-11 Paths are reestablished

Refer to “Reestablish paths” on page 111 for a description of the creation of the task.

6.3.10 Reestablish PPRC-XD pairsWith the FlashCopy relationship already established — the tertiary copy is already saved — and the paths already reestablished, we can now resume the update activity onto the secondary volumes. For this, we have to reestablish the XD relationship between the volume pairs, as shown in Figure 6-12.

Note: A FlashCopy pair is terminated either by completion of the background copy, or by invoking the Withdraw FlashCopy pair task. So, when no background copy is requested, then an additional step — for withdrawing the FlashCopy pair — must be executed as part of the procedure.


Figure 6-12 Reestablish PPRC-XD pair - incremental copy

The Reestablish PPRC XD pair task reestablishes the volume pairs XD relationships, doing an incremental copy of the out-of-sync tracks only. It copies only the tracks that were changed while the volumes were suspended — Copy out-of-sync cylinders only copy option.

This way, the volume pairs will again reach the duplex pending XD state — ready for the following PiT backup checkpoint cycle. The transitioning of this task is illustrated in Figure 6-11 on page 81, where the PPRC pairs transition to the duplex pending XD state — with the secondary volume been a FlashCopy source.

Refer to “Reestablish PPRC XD pairs” on page 111 for a description of the creation of the task.

6.3.11 Additional tasksIt may be useful to create a few additional tasks while you are creating the tasks needed for the PiT backup.

Withdraw FlashCopy pairCreate a Withdraw FlashCopy pair task, needed to terminate the FlashCopy pair when not doing a background copy. Refer to “Withdraw FlashCopy pair” on page 113 for a description of the creation of the task.

Suspend PPRC pairFor suspending PPRC pairs. Refer to “Suspend PPRC pair” on page 114 for a description of the creation of the task.

Terminate PPRC pairFor terminating PPRC pairs. Refer to “Terminate PPRC pair” on page 115 for a description of the creation of the task.

Consistency-group-createdFor resuming the write activity upon a primary LSS after a freeze operation —if the LSS pair had the consistency group option enabled. Refer to “Consistency-group-created” on page 117 for a description of the creation of the task.

6.4 Using the CLI for a PiT backup with PPRC-XDIn this section, the sequence set presented in the previous section is now executed from the CLI. For detailed information on how to use the CLI, you can refer to IBM TotalStorage Enterprise Storage Server Copy Services Command-Line Interface Reference, SC26-7434.


As already mentioned, this an example procedure for getting a consistent PiT backup, using PPRC Extended Distance. This procedure, and the description of the tasks, were discussed in more detail in 6.3, “Tasks for a PiT backup with PPRC-XD” on page 76.

The creation of these tasks is described in Appendix A, “ESS Copy Services creation of tasks” on page 95. The tasks can be executed either from the tasks panels using the WUI, or — as we will describe in the present section — using the CLI.

Using the CLI gives you the possibility of automating the process in a script. Also some of the steps in this example will be best implemented, if you define and save the individual tasks — and then group them in a major task. In our description the execution of each step is individually illustrated, and the output at completion is displayed.

6.4.1 Establish pathsWhen this task is executed, paths between the primary and secondary LSSs are established. Please refer to 6.3.1, “Establish paths” on page 77, for further discussion of this step.

The task we invoke is the predefined task EST_PATH_12_13 as shown in Figure 6-13. Once this task is executed, the paths are defined, and then we can proceed to define the pairs’ XD relationships.

Figure 6-13 rsExecuteTask EST_PATH_12_13

This task does not enable the consistency group option for the pair of LSSs logically connected — because this option was not selected when the task was created.

Please refer to Appendix A.1.9, “Reestablish paths” on page 111 for a description of the creation of this task.

Note: Remember that when executing tasks from the CLI, task names are case sensitive.

Note: In our example, ESS-ServerName represents the DNS registered name (or IP address) of the primary ESS server. Also the password protection option is disabled (on the ESS), therefore password credentials are not supplied with the commands.

For details on how to specify password credentials — either directly or by pointing to a security file — see the IBM TotalStorage Enterprise Storage Server Copy Services Command-Line Interface Reference, SC26-7434. We recommend that you always have the password option enabled in a production environment.

C:\Program Files\IBM 2105 CLI>rsExecuteTask /v /s ESS-ServerName EST_PATH_12_13rsAppletStatusConnection: Successfully connected to Applet Status ServerrsStatusMessenger: Status Messanger startedrsExecuteTask: *************Finding the tasks****************rsExecuteTask: Task EST_PATH_12_13 found by task managerrsExecuteTask: *************Scheduling the tasks****************rsExecuteTask: Task EST_PATH_12_13 scheduled with copy services serverrsExecuteTask: *************Monitoring the tasks****************rsExecuteTask: Waiting on server...rsExecuteTask: Task EST_PATH_12_13 completed successfullyrsStatusMessenger: Status Checker endingrsExecuteTask: Command successful


6.4.2 Establish PPRC-XD pair — initial copyWhen this task is executed, the XD relationship between the primary and secondary volumes is established. Please refer to 6.3.2, “Establish PPRC-XD pair — initial copy” on page 77, for further discussion of this step.

In Figure 6-14 the rsExecuteTask command executes the predefined EST_PPRC_XD_V1 task, successfully establishing the PPRC-XD pair.

Figure 6-14 rsExecuteTask EST_PPRC_XD_V1

Please refer to A.1.2, “Establish PPRC XD pair — initial copy” on page 98 for a description of the creation of this task.

Querying the state of the volumesIn Figure 6-15 you can see a query to the source volume. As you can notice, the output information shows the “PPRC Number of out-of-sync sectors=01880320” for that volume pair.

Figure 6-15 rsQuery PPRC source volume

Note: In our example, the detailed rsRegister information from the CLI output is left out.

C:\Program Files\IBM 2105 CLI>rsExecuteTask /v /s ESS-ServerName EST_PPRC_XD_V1

rsWebTest: HeartBeat to the server was successful.01-03-2002 09:24:15 rsRegister: CLI client going to register with server01-03-2002 09:24:15 rsRegister: registerClient request string 01-03-2002 09:24:15 rsRegister: registerClient response string "102,1,30001007f,691369970,0"01-03-2002 09:24:15 rsRegister: CLI client registered successfullyrsExecuteTask: Got task manager referencersAppletStatusConnection: Successfully connected to Applet Status ServerrsStatusMessenger: Status Messanger startedrsExecuteTask: *************Finding the tasks****************rsExecuteTask: Task EST_PPRC_XD_V1 found by task managerrsExecuteTask: *************Scheduling the tasks****************rsExecuteTask: Task EST_PPRC_XD_V1 scheduled with copy services serverrsExecuteTask: *************Monitoring the tasks****************rsExecuteTask: Waiting on server...rsExecuteTask: Task EST_PPRC_XD_V1 completed successfullyrsStatusMessenger: Status Checker endingrsExecuteTask: Command successful

C:\Program Files\IBM 2105 CLI>rsQuery /v /q 20221389 /s ESS-ServerNamersWebTest: HeartBeat to the server was successful.************************Volume Information************************Volume 20221389 found on 21389:12 as volume number 002PPRC State=source, Type=extended distance, Status=copy_pendingPPRC Number of out-of-sync sectors=01880320FlashCopy_state=none, Size=1.0_GBPPRC Peer=30221389******************************************************************


When querying for out-of-sync sectors, you must query the source volume — since this information is only associated to the primary volumes. The out-of-sync sectors are not displayed when querying the target volumes, as you can see in Figure 6-16.

Figure 6-16 rsQuery target volume

6.4.3 Establish PPRC-SYNC pair — go-to-SYNC and suspendIn this step, the pairs go synchronous and suspend — which, in our example, we do before quiescing the application. Please refer to 6.3.3, “Establish PPRC-SYNC pair — go-to-SYNC and suspend” on page 78, for further discussion of this step.

At a scheduled point-in-time — preferably when the application write activity is low, or none — a catch-up is commanded (go-to-SYNC) by invoking the PPRC_SYN_SUSP_V1 predefined task (see Figure 6-17). This task leaves the pairs suspended and individually synchronized at a point-in-time. But this is still not the application consistent PiT checkpoint.

The creation of this task is described in Appendix A.1.3, “Establish PPRC SYNC pair — go-to-SYNC and suspend” on page 102.

Figure 6-17 rsExecuteTask PPRC_SYN_SUSP_V1

Note: The number of out-of-sync sectors — displayed when querying a volume — is a point-in-time value.

C:\Program Files\IBM 2105 CLI>rsQuery /v /q 30221389 /s ESS-ServerNamersWebTest: HeartBeat to the server was successful.************************Volume Information************************Volume 30221389 found on 21389:13 as volume number 002PPRC State=target, Type=extended distance, Status=copy_pendingFlashCopy_state=none, Size=1.0_GBPPRC Peer=20221389******************************************************************

C:\Program Files\IBM 2105 CLI>rsExecuteTask /v /s ESS-ServerName PPRC_SYN_SUSP_V1rsWebTest: HeartBeat to the server was successful.rsExecuteTask: Got task manager referencersAppletStatusConnection: Successfully connected to Applet Status ServerrsStatusMessenger: Status Messanger startedrsExecuteTask: *************Finding the tasks****************rsExecuteTask: Task PPRC_SYN_SUSP_V1 found by task managerrsExecuteTask: *************Scheduling the tasks****************rsExecuteTask: Task PPRC_SYN_SUSP_V1 scheduled with copy services serverrsExecuteTask: *************Monitoring the tasks****************rsExecuteTask: Waiting on server...rsExecuteTask: Task PPRC_SYN_SUSP_V1 completed successfullyrsStatusMessenger: Status Checker endingrsExecuteTask: Command successful


No matter that the rsExecuteTask command output is showing a “command successful” line of information — this does not mean that the task has completed. To check for completion of the task, we issue the rsQueryComplete command — shown in Figure 6-18. As you can see from the output of the command in Figure 6-18, the task was initially not completed at the beginning of the query —and then the verification is reissued 60 seconds later, showing 100% complete.

As already mentioned, the time interval between query prompts can now be smaller — because of the more granularity you have when specifying the - m parameter of the rsQueryComplete command (please refer to Section 3.4.2, “rsQueryComplete command” on page 38).

Figure 6-18 rsQueryComplete PPRC_SYN_SUSP_V1

While the rsExecuteTask command is used for triggering the task execution, the rsQueryComplete command is very useful for automation scripts — because it signals the completion of the task.

With the rsQueryComplete task indicating completion of the PPRC_SYN_SUSP_V1 task, we now have the volume pair synchronized and suspended — ready for the quiesce of the application.

Query the state of the volumesAs we query the source (Figure 6-19) and target volumes (Figure 6-20) we are able to verify that the pair is effectively suspended.


C:\Program Files\IBM 2105 CLI>rsQueryComplete /v /s ESS-ServerName PPRC_SYN_SUSP_V1rsWebTest: HeartBeat to the server was successful.rsQueryComplete: Got task manager referencersQueryComplete: --------- Task Name: PPRC_SYN_SUSP_V1 ---------rsQueryComplete: Task PPRC_SYN_SUSP_V1 found by TaskManagerrsQueryComplete: Sampling volumes...rsQueryComplete: Percentage complete = 3rsQueryComplete: waiting 60 seconds...rsQueryComplete: Sampling volumes...rsQueryComplete: Percentage complete = 100rsQueryComplete: ----------------------------------------------rsQueryComplete: Command successful

C:\Program Files\IBM 2105 CLI>rsQuery /v /q 20221389 /s ESS-ServerNamersWebTest: HeartBeat to the server was successful.************************Volume Information************************Volume 20221389 found on 21389:12 as volume number 002PPRC State=source, Type=synchronous, Status=suspendedFlashCopy_state=none, Size=1.0_GBPPRC Peer=30221389******************************************************************


Figure 6-20 rsQuery PPRC target volume

Also, as you can check from Figure 6-19 and Figure 6-20, there is no out-of-sync information displayed. This is because the out-of-sync information is only a consideration when the volume pairs are in a duplex pending condition —or duplex pending XD. But now, our volumes are in a suspended state —after having reached the duplex state (full synchronization) for a moment.

6.4.4 Quiesce the application write activityFor this step, the quiesce of the application writes must be scripted according to the application peculiarities.

6.4.5 Re-synchronize PPRC pairIn this step the pairs are re-synchronized — for the updates that occurred in the interval since the completion of the individual go-to-SYNCs, and the completion of the application quiesce. Please refer to 6.3.5, “Re-synchronize PPRC pair” on page 79, for further discussion of this step.

As already mentioned, in the alternative that you first quiesce the application — and wait for the termination of all application writes — then this re-synchronization can be skipped.

We do the re-synchronization by executing the predefined task RESYN_PPRC_V1, as shown in Figure 6-21. The creation of the task is described in A.1.5, “Re-synchronize PPRC pair” on page 104.

C:\Program Files\IBM 2105 CLI>rsQuery /v /q 30221389 /s ESS-ServerNamersWebTest: HeartBeat to the server was successful.************************Volume Information************************Volume 30221389 found on 21389:13 as volume number 002PPRC State=target, Type=synchronous, Status=suspendedFlashCopy_state=none, Size=1.0_GBPPRC Peer=20221389******************************************************************

Note: If we are within the PPRC synchronous supported distance (103 km), and we can temporarily tolerate the toll of synchronous mirroring — increased write response time — then we can avoid the quiesce of the application. Please refer to 6.3.4, “Quiesce the application write activity” on page 78, for further discussion of this step


Figure 6-21 rsExecuteTask RESYN_PPRC_V1

Again, the rsQueryComplete command is issued to verify the completion of the task, as shown in Figure 6-22.

Figure 6-22 rsQueryComplete RESYN_PPRC_V1

Upon completion of this step, we proceed to step 5, where we freeze the write activity.

6.4.6 FreezeIn this step — by executing the predefined task FREEZE_LSS12_13 — we act at once upon all the volumes in the LSS. The volumes of the affected primary LSS are suspended, and the paths of the affected LSS pair are terminated. Please refer to 6.3.6, “Freeze” on page 79, for further discussion of this step.

Remember that, if you are using the WUI and the CLI, the freeze operation is best implemented by creating and saving tasks for each of the application related LSS pairs, and then grouping the tasks in a major task. For TSO users, you may set up a command list with the freeze commands for the set of related LSS pairs that you want to freeze.

C:\Program Files\IBM 2105 CLI>rsExecuteTask /v /s ESS-ServerName RESYN_PPRC_V1rsWebTest: HeartBeat to the server was successful.rsExecuteTask: Got task manager referencersAppletStatusConnection: Successfully connected to Applet Status ServerrsStatusMessenger: Status Messanger startedrsExecuteTask: *************Finding the tasks****************rsExecuteTask: Task RESYN_PPRC_V1 found by task managerrsExecuteTask: *************Scheduling the tasks****************rsExecuteTask: Task RESYN_PPRC_V1 scheduled with copy services serverrsExecuteTask: *************Monitoring the tasks****************rsExecuteTask: Waiting on server...rsExecuteTask: Task RESYN_PPRC_V1 completed successfullyrsExecuteTask: Command successful

C:\Program Files\IBM 2105 CLI>rsQueryComplete /v /s ESS-ServerName RESYN_PPRC_V1rsWebTest: HeartBeat to the server was successful.rsQueryComplete: Got task manager referencersQueryComplete: --------- Task Name: RESYN_PPRC_V1 ---------rsQueryComplete: Task RESYN_PPRC_V1 found by TaskManagerrsQueryComplete: Sampling volumes...rsQueryComplete: Percentage complete = 100 (PPRC Type = synchronous)rsQueryComplete: ----------------------------------------------rsQueryComplete: Command successful


When this task is completed, we have the consistent PiT checkpoint at the recovery site. You can see in Figure 6-23 the execution of the predefined task FREEZE_LSS12_13. The creation of the task is described in A.1.6, “Freeze” on page 106.

Figure 6-23 rsExecuteTask FREEZE_LSS12_13

As a consequence of the freeze operation, the primary volume will become suspended (see rsQuery output in Figure 6-24).


However, the secondary volume will stay as it was, prior to the freezing — in our example “Status=fullcopy”. See the rsQuery output in Figure 6-25.

Figure 6-25 rsQuery PPRC target volume

C:\Program Files\IBM 2105 CLI>rsExecuteTask /v /s ESS-ServerName FREEZE_LSS12_13rsWebTest: HeartBeat to the server was successful.rsExecuteTask: Got task manager referencersAppletStatusConnection: Successfully connected to Applet Status ServerrsStatusMessenger: Status Messanger startedrsExecuteTask: *************Finding the tasks****************rsExecuteTask: Task FREEZE_LSS12_13 found by task managerrsExecuteTask: *************Scheduling the tasks****************rsExecuteTask: Task FREEZE_LSS12_13 scheduled with copy services serverrsExecuteTask: *************Monitoring the tasks****************rsExecuteTask: Waiting on server...rsExecuteTask: Task FREEZE_LSS12_13 completed successfullyrsExecuteTask: Command successful

C:\Program Files\IBM 2105 CLI>rsQuery /v /q 20221389 /s ESS-ServerNamersWebTest: HeartBeat to the server was successful.************************Volume Information************************Volume 20221389 found on 21389:12 as volume number 002PPRC State=source, Type=synchronous, Status=suspendedFlashCopy_state=none, Size=1.0_GBPPRC Peer=30221389******************************************************************

C:\Program Files\IBM 2105 CLI>rsQuery /v /q 30221389 /s ESS-ServerNamersWebTest: HeartBeat to the server was successful.************************Volume Information************************Volume 30221389 found on 21389:13 as volume number 002PPRC State=target, Type=synchronous, Status=fullcopyFlashCopy_state=none, Size=1.0_GBPPRC Peer=20221389******************************************************************


6.4.7 Resume application write activityOnce the freeze is done, we can now resume the write activity onto the primary volumes. With the freeze operation, the consistent checkpoint at the remote site has already been taken.

As for this step, the application write activity will be able to resume onto the primary volumes. But these updates will not be mirrored onto the secondary volumes — primary volumes are still suspended, and paths are still discontinued. Please refer to 6.3.7, “Resume application write activity” on page 80, for further discussion of this step.

6.4.8 FlashCopy PPRC targetThe secondary volumes still have not been updated — and they do have a consistent PiT checkpoint — so it is safe to do a copy now before they start to be updated again. Please refer to 6.3.8, “FlashCopy PPRC target” on page 80, for further discussion of this step.

A FlashCopy — of the secondary volumes onto the tertiary volumes — is done by executing the task EST_FC_V1_W1 as shown in Figure 6-26. The creation of the task is described in A.1.8, “FlashCopy PPRC target” on page 108.

Figure 6-26 rsExecuteTask EST_FC_V1_W1

You can check the completion of the FlashCopy operation by querying the source (Figure 6-27) and target (Figure 6-28) volumes.

Figure 6-27 rsQuery FlashCopy source

C:\Program Files\IBM 2105 CLI>rsExecuteTask /v /s ESS-ServerName EST_FC_V1_W2rsWebTest: HeartBeat to the server was successful.rsAppletStatusConnection: Successfully connected to Applet Status ServerrsStatusMessenger: Status Messanger startedrsExecuteTask: *************Finding the tasks****************rsExecuteTask: Task EST_FC_V1_W1 found by task managerrsExecuteTask: *************Scheduling the tasks****************rsExecuteTask: Task EST_FC_V1_W1 scheduled with copy services serverrsExecuteTask: *************Monitoring the tasks****************rsExecuteTask: Waiting on server...rsExecuteTask: Task EST_FC_V1_W1 completed successfully

C:\Program Files\IBM 2105 CLI>rsQuery /v /q 30221389 /s ESS-ServerName************************Volume Information************************Volume 30221389 found on 21389:13 as volume number 002PPRC State=target, Type=synchronous, Status=fullcopyFlashCopy_state=source, Size=1.0_GBPPRC Peer=20221389******************************************************************


Figure 6-28 rsQuery FlashCopy target

As soon as the FlashCopy pair is established, we can proceed to the step described in the following section.

6.4.9 Reestablish pathsIn this step, the paths that were terminated in the freeze operation are reestablished. Please refer to 6.3.9, “Reestablish paths” on page 81, for further discussion of this step.

To do this, the predefined task EST_PATH_12_13 is invoked, as shown in Figure 6-29. The creation of the task is described in Appendix A.1.9, “Reestablish paths” on page 111.

Figure 6-29 rsExecuteTask EST_PATH_12_13

With the paths reestablished, we can now proceed to the next step —reestablish the PPRC-XD pairs.

6.4.10 Reestablish PPRC-XD pairsNow the XD mirroring over the secondary volumes is reestablished, by reestablishing the XD pair relationships. Please refer to 6.3.10, “Reestablish PPRC-XD pairs” on page 81, for further discussion of this step.

C:\Program Files\IBM 2105 CLI>rsQuery /v /q 30521389 /s ESS-ServerName ************************Volume Information************************Volume 30521389 found on 21389:13 as volume number 005PPRC State=simplex, Status=noneFlashCopy_state=target, Size=1.0_GBPPRC Peer=None******************************************************************

C:\Program Files\IBM 2105 CLI>rsExecuteTask /v /s ESS-ServerName EST_PATH_12_13rsAppletStatusConnection: Successfully connected to Applet Status ServerrsStatusMessenger: Status Messanger startedrsExecuteTask: *************Finding the tasks****************rsExecuteTask: Task EST_PATH_12_13 found by task managerrsExecuteTask: *************Scheduling the tasks****************rsExecuteTask: Task EST_PATH_12_13 scheduled with copy services serverrsExecuteTask: *************Monitoring the tasks****************rsExecuteTask: Waiting on server...rsExecuteTask: Task EST_PATH_12_13 completed successfullyrsStatusMessenger: Status Checker endingrsExecuteTask: Command successful


To do this, the predefined task REEST_PPRC_XD_V1 is invoked, as shown in Figure 6-30. The creation of the task is described in Appendix A.1.10, “Reestablish PPRC XD pairs” on page 111.

Figure 6-30 rsExecuteTask REEST_PPRC_XD_V1

Now the secondary volumes begin again to hold a fuzzy copy of the primary volumes — ready for the following PiT backup checkpoint cycle.

6.4.11 Consistency group createdIn our example, the LSS pair was not enabled for consistency grouping. It was unnecessary, because at the freeze step, the application was not making any write activity. As we already discussed, for PPRC-XD relationships, there is no need of having consistency grouping enabled — the secondary volumes always have a fuzzy copy of the primaries. Please refer to “Establish paths” on page 77 for further discussion.

In the event that you implement PPRC-XD within the synchronous range — and find it acceptable to operate short times in the synchronous (SYNC) mode, trading off some application write performance — then you might be interested in having the consistency group option enabled.

When you have defined the LSS pair with the consistency group option enabled, then — after you perform a freeze operation onto the LSS pair — any subsequent write activity attempted onto the primary LSS volumes will trigger an ELB (extended long busy) condition of the primary LSS. During this interval (default 2 minutes), all write activity onto the primary volumes of this LSS is queued, and the LSS remains in this condition until the time expires, or until a consistency-group-created operation is done — the “thaw” of the freeze.

C:\Program Files\IBM 2105 CLI>rsExecuteTask /v /s ESS-ServerName REEST_PPRC_XD_V1rsExecuteTask: Got task manager referencersAppletStatusConnection: Successfully connected to Applet Status ServerrsStatusMessenger: Status Messanger startedrsExecuteTask: *************Finding the tasks****************rsExecuteTask: Task REEST_PPRC_XD_V1 found by task managerrsExecuteTask: *************Scheduling the tasks****************rsExecuteTask: Task REEST_PPRC_XD_V1 scheduled with copy services serverrsExecuteTask: *************Monitoring the tasks****************rsExecuteTask: Waiting on server...rsExecuteTask: Task REEST_PPRC_XD_V1 completed successfullyrsStatusMessenger: Status Checker endingrsExecuteTask: Command successful


Task CG_CRE_LSS12_13 does a consistency-group-created operation. Figure 6-31.illustrates the execution of the task.

Figure 6-31 rsExecuteTask CG_CRE_LSS12_13

The creation of the task is described in Appendix A.1.14, “Consistency-group-created” on page 117.

Remember that — like the freeze operation — the consistency-group-created operation is best implemented by creating and saving tasks for each of the application related LSS pairs, and then grouping the tasks in a major task.

6.4.12 Creating the scriptWhen creating the script for your PiT backup solution with PPRC XD, you can use the example presented in this chapter for your reference. However, you must plan for the appropriate checkpoints. You might want to monitor the workload on your application, to determine which are the appropriate times to do the PiT checkpoints — unless you already have a predefined backup window in place.

C:\Program Files\IBM 2105 CLI>rsExecuteTask /v /s ESS-ServerName CG_CRE_LSS12_13rsExecuteTask: Got task manager referencersAppletStatusConnection: Successfully connected to Applet Status ServerrsStatusMessenger: Status Messanger startedrsExecuteTask: *************Finding the tasks****************rsExecuteTask: Task CG_CRE_LSS12_13 found by task managerrsExecuteTask: *************Scheduling the tasks****************rsExecuteTask: Task CG_CRE_LSS12_13 scheduled with copy services serverrsExecuteTask: *************Monitoring the tasks****************rsExecuteTask: Waiting on server...rsExecuteTask: Task CG_CRE_LSS12_13 completed successfullyrsStatusMessenger: Status Checker endingrsExecuteTask: Command successful


Appendix A. ESS Copy Services creation of tasks

In this appendix we explain how the ESS Copy Services tasks were created for the PiT example presented in 6.3, “Tasks for a PiT backup with PPRC-XD” on page 76. How these tasks were run using the CLI was covered in 6.4, “Using the CLI for a PiT backup with PPRC-XD” on page 82.

These tasks were created using the ESS Copy Services Web user interface (WUI). For detailed information on how to use the ESS Specialist for the creation of tasks, please refer to the publication IBM TotalStorage Enterprise Storge Server Web Interface User’s Guide, SC26-7346.

This appendix also includes an example of how to install the CLI — specifically, the CLI that used to run the created tasks of our example.

A


A.1 ESS Copy Services tasks: step-by-stepThe following description corresponds to the creation of tasks relative to one pair of volumes, on open systems LSSs — FBA LSSs. The steps are similar if you are creating tasks for multiple volumes. In this case you will probably want to group the individual tasks into major tasks.

In the following sections, the name between brackets corresponds to the task name. This makes more easier the reference to the previous chapters — that describe the implementation and execution of these tasks.

A.1.1 Establish paths

Task [EST_PATHS_V1]This task defines the paths between the primary and secondary LSSs. The paths are defined without consistency grouping.

In the ESS Copy Services Paths panel, you first select the source LSS from the scroll-down menu — and then select the SAIDs and the target LSS for launching the Task Wizard. Then, in the Task Wizard, the Establish paths option is selected (see Figure A-1).

Figure A-1 Establish paths


Then, from the Select path options window, the path options are selected — as shown in Figure A-2. The definition does not enable consistency grouping for this pair of LSSs.

Figure A-2 Path options

Finally, the task definition is completed by naming the task — EST_PATH_V1 — and putting in a description — as shown in Figure A-3.

Figure A-3 Define task ‘EST_PATHS_V1’.

Once the task is saved, it will appear in the Tasks panel of the ESS Copy Services Web user interface.

Note: Remember that when defining a path over a physical link, this link becomes unidirectional. This means that the physical link cannot be shared with paths that associate LSSs in the reverse direction. Please refer to 4.1.2, “Defining paths between two LSSs” on page 44 for more information on path configuration.

Appendix A. ESS Copy Services creation of tasks 97

A.1.2 Establish PPRC XD pair — initial copy

Task [EST_PPRC_XD_V1]This task establishes the XD relationship between the primary and secondary volumes — and does the initial source volume full copy.

The source and target LSSs are selected from the Volumes panel window — as shown in Figure A-4 Selecting Source and Target LSSs.

Figure A-4 Selecting source and target LSSs


Then you select the source volume by left-clicking on the volume (volume ID turns blue) followed by a right-click on the target volume (volume ID turns red) — as shown in Figure A-5.

Figure A-5 Selecting the volume pair

With an additional right-click on the target volume you pop-up the Task Wizard — as shown in Figure A-6. From the Task Wizard you specify the task to be executed.


Figure A-6 Launching the Task Wizard

The Task Wizard is shown with more detail in Figure A-7. Since our objective is to establish a PPRC-XD pair relationship, then we select the Establish Extended Distance PPRC task — as Figure A-7 shows.

Figure A-7 Establish Extended Distance PPRC

Tip: To select multiple volume pairs in the same task, click the button “Enter Multiple Selection Mode”.


This task will establish the XD pair — and do the initial copy of the source volume — therefore the copy option Copy entire volume is selected — as shown in Figure A-8.

Figure A-8 Copy entire volume

Finally the task definition is completed by naming the task — EST_PPRC_XD_V1 — and writing a description — as shown in Figure A-9.

We recommend that you use a naming convention, where the task name is descriptive of the task function. This will make it easy at management time.

These are some rules to remember when coding task names:

� The task name can only contain alphanumeric characters.� The task name cannot contain blanks.� The task name can contain the underscore (_) or hyphen (-) characters.� The task name cannot exceed 16 characters.


Figure A-9 Define task ‘EST_PPRC_XD_V1’

You can save the created task or run it immediately. We recommend you always save a task prior to execution — this gives an opportunity to reuse the task and do better troubleshooting if the task fails.

When the task is saved it appears in the ESS Copy Services Tasks panel. It can then be executed from there — or from the CLI.

A.1.3 Establish PPRC SYNC pair — go-to-SYNC and suspend

Task [PPRC_SYN_SUSP_V1]This task transitions the volume pair from the duplex pending XD state to the duplex state — go-to-SYNC command — and then suspends the pair. When executing this task — PPRC does an incremental copy of the updates to the changed tracks only. This is the very fast and efficient way of PPRC to do the catch-up.

To create this task, the Task Wizard is launched — by selecting the source and target volumes. In the Task WIzard window, the Establish synchronous PPRC copy pair option is selected — as you can see in Figure A-10.

Tip: The tasks described in this section are created by selecting the same volume pair while launching the task wizard. A faster and less error prone way to do this is to go back and select the first task created, and modify it to choose a different task. In this case, the volume pairs are inherited from the original task.


Figure A-10 Establish synchronous PPRC copy pair

Then — when selecting the copy options — select Copy out-of-sync cylinders only, so PPRC does an incremental copy of the updates to the changed tracks only. And also select Suspend PPRC after establish — as shown in Figure A-11. The reason for selecting these copy options is that we want to synchronize the pair, and then leave the volumes in suspended state — so no synchronous updates are transmitted to the secondary volumes.

Figure A-11 Suspend PPRC after establish

Note: During the go-to-SYNC transition, the synchronization is done in an incremental manner — transmitting only the updates that are pending. This is a very efficient synchronization technique, that minimizes the time needed for the catch-up transition — don’t be mislead by the cylinders reference from the Copy out-of-sync cylinders only legend — in the Task WIzard options window, that was illustrated in Figure A-11.


Finally, the task definition is completed by naming the task — PPRC_SYN_SUSP_V1 — and writing a description — as shown in Figure A-12.

Figure A-12 Define task ‘PPRC_SYN_SUSP_V1’

Once the task has been saved, it appears in the Tasks panel of the ESS Copy Services Web user interface.

A.1.4 Quiesce the application write activityThe characteristics of this task is completely dependent on your application peculiarities. The objective of this task is to stop all write activity from your application — onto the primary volumes. You must create a script which carries out this action.

A.1.5 Re-synchronize PPRC pair

Task [RESYN_PPRC_V1]The objective of this task is to re-synchronize the pair of volumes, and leave them in duplex state. The execution of this task transitions — the volume pair — from the suspended state to the duplex state. In this transition, and incremental copy of the changed tracks is done.

The creation of the task is similar to the PPRC_SYN_SUSP_V1 — but in this occasion we do not need to suspend the pair. This is because there is no application write activity, and because we will be doing a freeze operation of the entire LSS in the next step.


When the Task Wizard pops-up, we select the option Establish synchronous PPRC copy pair — as Figure A-13 shows.

Figure A-13 Re-sync synchronous PPRC copy pair.

When selecting the copy options, select Copy out-of-sync cylinders only — as shown in Figure A-14. This way the task will be doing an incremental copy of the updated tracks only.

Figure A-14 Re-SYNC, copy out-of-sync cylinders only

Note: The legend of the re-synchronization option — in the Select copy options window in Figure A-14 — says Copy out-of-sync cylinders only, but in fact, the copy operation copies only the tracks or sectors — not cylinders — that were changed while the pair was suspended. This is a very powerful incremental technique for the re-synchronization of suspended volumes — when going from suspended state to either duplex or duplex pending XD state.


Finally, the task definition is completed by naming the task — RESYN_PPRC_V1 — and writing a description — as shown in Figure A-15.

Figure A-15 Define task RESYN_PPRC_V1

As you can see, this task was actually created by modifying a previous created task — therefore it is saved by clicking New on the applet.


A.1.6 Freeze

Task [FREEZE_LSS12_13]This task has the objective of freezing the LSS pair, so —all at once— the primary volumes are suspended — and no more updates can be propagated onto the secondary volumes, until some action is taken. At this moment the pairs are-synchronized and — as soon as the freeze command is done — the isolated secondaries hold a consistent PiT checkpoint of the data.

To define this task, you first select the LSS pair from the ESS Copy Services Logical Subsystems panel. Then right-clicking the target LSS, the Task WIzard window pops-up — as shown in Figure A-16.

As you can see in Figure A-16, the Freeze PPRC Consistency Group option is then selected. The freeze operation can be done on an LSS pair with (or without) consistency grouping enabled — in our example, consistency grouping was not enabled.


Figure A-16 Freeze PPRC consistency group

Finally the, task definition is completed by naming the task — FREEZE_LSS12_13 — and writing a description — as shown in Figure A-17.


Figure A-17 Define task ‘FREEZE_LSS12_13’


A.1.7 Resume application write activityWith the primary volumes suspended, and the paths between LSSs terminated — as a result of the freeze operation — we can now resume the application write activity — without propagating the updates onto the PiT consistent secondary volumes. The restart of the application write activity can be executed immediately after the freeze is done.

Meanwhile, PPRC will keep track of the primary updated tracks — for a subsequent re-synchronization step.

A.1.8 FlashCopy PPRC target

Task [EST_FC_V1_W1]The objective of this task is to create a tertiary copy of the secondary PiT consistent copy. This copy is done with a FlashCopy operation — initiated when the secondary is not receiving updates — LSS paths are not available, and primary volumes are still suspended.

The source and target volumes are selected similarly as in the previous tasks definitions — the only difference is that the FlashCopy target must be within the same LSS as the source. The volume selection is shown in Figure A-18 — where the selection of the volumes is done working on the left pane of the window (source side).

Note: When you want to freeze multiple LSS pairs in the same step, you must create a task for each pair and then group them to one major task. Grouped tasks are executed in parallel.


Figure A-18 Selecting source and target FlashCopy volumes

The Task Wizard is then launched with an additional (right) click on the target volume. As shown in Figure A-19, the task Establish FlashCopy Pair is then selected.

Figure A-19 Establish FlashCopy pair


Selecting either — to perform background copy — or not, is completely dependent upon your specific setup and recovery requirements. In our example, we chose the accelerated destage mode option (see Figure A-20). In either case — as soon as the FlashCopy relationship is established — then the secondary is in conditions of starting to receive updates.

Figure A-20 Selecting FlashCopy options

Finally, the task definition is completed by naming the task — EST_FC_V1_W1 — and writing a description — as Figure A-21 shows.

Figure A-21 Define Task ‘EST_FC_V1_W1’.

Note: Permit establish if target is online is not a valid option for open systems — therefore it is grayed out.


A.1.9 Reestablish paths

Task [EST_PATHS_V1]This task is the same task already illustrated in “Establish paths” on page 96 — please refer to that section for description and considerations. We are now reestablishing the paths again — because they were terminated by the freeze operation.

This step is necessary to be able to resume the volume pairs XD relationship.

A.1.10 Reestablish PPRC XD pairs

Task [REEST_PPRC_XD_V1]This task reestablishes the PPRC-XD relationship — between the primary and secondary volumes.

To create this task, the Task Wizard is launched by selecting the source and target volume — as in previously task definitions. Then the task Establish Extended Distance PPRC is selected — as Figure A-22 shows.

Figure A-22 Establish Extended Distance PPRC

Since we are reestablishing the volume pair, from its suspended state, we do not need to copy the entire volume — but only the out-of-sync tracks. Therefore, we select the incremental Copy out-of-sync cylinders only copy option — as shown in Figure A-23.


Figure A-23 Copy out-of-sync tracks only

Finally, the task definition is completed by naming the task — REEST_PPRC_XD_V1 — and writing a description — as shown in Figure A-24.

Figure A-24 Define task ‘REEST_PPRC_XD_V1



A.1.11 Withdraw FlashCopy pair

Task [WDRAW_FC_V1_W1]To create this task —which has the objective of withdrawing the FlashCopy pair— the source and target volumes are selected — as described in “FlashCopy PPRC target” on page 108.

Once the Task Wizard is launched, then the task Withdraw FlashCopy pair is selected — as shown in Figure A-25.

Figure A-25 Withdraw FlashCopy pair

Finally, the task definition is completed by naming the task — WDRAW_FC_V1_W1 — and writing a description — as shown in Figure A-26.

Figure A-26 Define task ‘WDRAW_FC_V1_W1



A.1.12 Suspend PPRC pair

Task [SUSP_PPRC_V1_S]To create this task — which has the objective of suspending the PPRC pair — the Task Wizard is launched by selecting the source and target volume — as in previous task definitions.

You then select the Suspend PPRC copy pair task — as shown in Figure A-27.

Figure A-27 Suspend PPRC copy pair

Then you specify whether the task is scheduled with the target — or the source LSS. In Figure A-28 we selected the task to be scheduled with the source LSS.

Figure A-28 Specify LSS where task is scheduled.

Note: You can define the suspend task to be scheduled on either the source LSS — or at the target LSS. When both ESSs are operational — and the physical links are not down — there is no difference whether the task is executed from the source or the target LSS.


Finally, the task definition is completed by naming the task — SUSP_PPRC_V1 — and writing a description — as shown in Figure A-29.

Figure A-29 Define task ‘SUSP_PPRC_V1_S


A.1.13 Terminate PPRC pair

Task [TERM_PAIR_V1_S]To create this task —which has the objective of terminating the PPRC pair— the Task Wizard is launched by selecting the source and target volumes — as in previous task definitions.

You then select —from the Task Wizard options—the Terminate PPRC copy pair task — as Figure A-30 shows.

Figure A-30 Terminate PPRC pair


Similarly to when you suspend a PPRC pair, for the termination of a PPRC pair — you also specify whether the task is scheduled with the target or the source LSS. The same considerations apply. In our example we selected to schedule with the source LSS — as Figure A-31 shows.

Figure A-31 Specify LSS where task is scheduled

Finally, the task definition is completed by naming the task — TERM_PAIR_V1_S — and writing a description — as shown in Figure A-32.

Figure A-32 Define task ‘TERM_PAIR_V1_S



A.1.14 Consistency-group-created

Task [CG_CRE_LSS12_13]This task will be used if the LSSs — that you freeze — are consistency group enabled. This task is the “thaw” for the freeze. Once the consistency-group-created task is executed, the primary volumes leave the ELB time-out window — and start receiving updates.

The task is created by selecting the LSS pair, as when creating the freeze task, but choosing the PPRC Consistency Created option — as Figure A-33 shows.

.

Figure A-33 PPRC Consistency Created

As with the freeze, a consistency-group-created operation is best implemented by creating and saving tasks — for all of the application-related pairs — and then grouping the task into a major task.

Finally, the task definition is completed by naming the task — CG_CRE_LSS12_13 — and writing a description of the task — as Figure A-34 shows.


Figure A-34 Define task ‘CG_CRE_LSS12_13’

A.2 Installing the CLIIn this section we describe how the CLI is installed in a WIndows 2000 system.

A.2.1 CLI supported systemsCurrently the ESS Copy Services command line interface (CLI) is supported on the following selected open system servers:

� Hewlett-Packard-UX� IBM AIX� IBM NUMA-Q� Linux (Red Hat and SuSE, 32-bit, Intel-Based)� Microsoft Windows 2000� Microsoft Windows NT4.0� Novell Netware� Sun Solaris

The list of CLI supported servers is updated periodically, as more systems are added. For the most updated information on the CLI supported systems, please refer to:




A.2.2 Installing procedureTo install the CLI from the IBM TotalStorage Enterprise Storage Server Mega CD, you must be logged on to the respective platform with the highest administrative privilege (root, administrator or supervisor).

This installation procedure was done on a Windows 2000 system. For installation details see the information — pertaining to your platform — on the publication IBM TotalStorage Enterprise Storage Server Copy Services Command-Line Interface Reference, SC26-7434 — from the Mega CD [\\Mega CD\cliReadmes\f2acli01.pdf].

The Mega CD is inserted into the CD drive D:, and the setup file for Windows is executed — as Figure A-35 shows.

Figure A-35 Executing the CLI setup file

The InstallShield Wizard launches — as shown in Figure A-36 and in Figure A-37.

Figure A-36 Preparing the InstallShield Wizard


Figure A-37 IBM 2105 CLI InstallShield Wizard

On the InstallShield Wizard welcome panel (Figure A-37) click Next — to continue the installation.

Then — when prompted for the destination folder (see Figure A-38) — either click Browse (to change it), or click Next (to install in the default path C:\Program Files\IBM 2105 CLI\).


Figure A-38 Choosing the destination folder

Then the binaries are installed — as Figure A-39 illustrates.

Figure A-39 Installing


When the installation is completed, you get the window shown in Figure A-40.

Figure A-40 CLI has completed the installation

You should now click Finish — to terminate the InstallShield Wizard (see Figure A-40) — and proceed to reboot.

After rebooting, verify the installation by opening the Add/Remove Programs applet to verify that the IBM 2105 CLI is listed —as Figure A-41 illustrates.

Figure A-41 Verifying the installation


Appendix B. Additional examples of PiT and data migration solutions

In this appendix we present additional examples of solutions using PPRC Extended Distance for PiT backup and data migration implementations.

B


B.1 Data migration for a large data centerThe following section describes the use of PPRC Extended Distance, in combination with tapes, for the data migration of a large mainframe center, where a large volume of data needs to be migrated over a long distance.

B.1.1 Planning the migrationA large z/OS mainframe center has to be moved from one site to a remote location, at a long distance. This migration requires the detailed planning of many activities. In our example, we are going to set up a migration procedure using PPRC Extended Distance — and tapes.

Using PPRC Extended Distance for this remote migration adds more flexibility to a classic tape-only migration approach — that otherwise, is the recommended approach when planning to move a huge amount of data at a reasonable cost.

Having an objective of doing this migration as fast as possible, with no outage for the applications, PPRC Extended Distance offers added value as follows:

� There is no outage of the application for migrating the data — just the shutdown that was planned for switching the application to the remote site.

� There is minimal impact on the applications’ response time, while updates are transferred non-synchronously to the remote location.

� PPRC Extended Distance has an adaptive behavior, which optimizes the utilization of the available network bandwidth.

These are some other environmental considerations for this example:

� The centers are connected by channel extenders — compatible with PPRC Extended Distance.

� The secondary ESSs are installed, and the PPRC-XD relationships (between primaries and secondaries) are established.

� The tape library resources can sustain intensive copy operations, while also attending normal production demands.


Initial considerations and initial migration activities for this environment are summarized in Figure B-1.

Figure B-1 Large z/OS center remote data migration (1)

B.1.2 Preparing the volume pairs and dumping the dataThe relationship between primary and secondary volumes must be prepared, so the remote volumes can later catch-up the updates that the tapes missed — while on their way to the remote site.

For this initial phase of the data migration, the paths between the respective LSSs must be operational, and the volumes’ relationships must be established. This first phase of the migration activity is summarized in Figure B-1.

It will be very useful to setup major ESS Copy Services tasks, grouping the individual tasks that are involved in this procedure. This way, the activations and transitions — between PPRC states — can be made quickly and globally. If using TSO commands, then command lists can be set up, grouping the individual commands.

B.1.2.1 Establish paths First you have to define the logical paths between the local and remote LSS pairs. Because the pairs will be in a PPRC-XD relationship — secondaries holding a fuzzy copy — then consistency grouping is not needed.

B.1.2.2 Establish PPRC-SYNC pairs — NOCOPY optionEstablishing pairs in synchronous mode allows using the NOCOPY option. This way of establishing the volume pair will build a PPRC relationship — but will not duplicate the primary volume onto the secondary site. This transition leaves the volume pairs in the duplex state.

The environment:ESS to ESS remote migration of large amount of data - over an extended distance

Secondary disks installed, mapping one to one with primariesTape resources can sustain large amount of data dump/restores

Tapes used for fast migration with less telecom costNetwork bandwidth requirements low

Preparing the volume pairs:CLI can be used to invoke tasks

Major tasks can group individual activation/transitioning PPRC tasksEstablish paths between primary and secondary LSSs at both centersEstablish pairs with NOCOPY mode option

Immediately suspend Suspending the PPRC pairs

Can use "Suspend PPRC after establish" new copy optionPPRC keeps track of primary updates

Dump of all primary volumes onto tapes:Fuzzy copies are good because PPRC is keeping record of the updates that occur on the primary volumes

Applications are still running - with no interruptions - at the primary site

Appendix B. Additional examples of PiT and data migration solutions 125

B.1.2.3 Automatically suspend the pairsAs soon as the volumes reach the duplex state, any updates onto the primary volumes will be synchronously transmitted — to the remote site. You do not want this to happen because the volumes are completely out-of-sync — data-check error conditions can occur due to incompatible formats — and because you do not want synchronous transfers over extended distances, such as in this center migration.

So the volumes must be — automatically and immediately — suspended upon reaching the duplex state condition, at the end of the previous step. The best way of implementing this is by using the new Suspend PPRC after establish copy option of the ESS Copy Services Web user interface.

While the volumes remain in the suspended state, PPRC keeps track of the application primary updates. This will later allow a later transmission of the updated tracks, when a re-synchronization is executed.

Because PPRC starts to track the updated tracks before the volumes are dumped into tape — this will result in some more tracks been transmitted than necessary.

B.1.2.4 Dump primary volumes onto tapesNow the primary volumes are dumped to the tapes — that will be sent to the remote site.

This dump of the primary volumes can be a fuzzy one — because PPRC is keeping record of the updated tracks. So the dump process can occur at any time — while the applications are running

B.1.3 Restoring the data and re-synchronizing the pairsIn this second phase of the migration procedure, the tapes — with the volumes’ dumps — are sent to the remote location to be restored. Then after restoring, the updates — that occurred while the tapes were delivered — will be applied onto the secondaries using PPRC Extended Distance. This second phase characteristics are summarized in Figure B-2.


Figure B-2 Large z/OS center remote data migration (2)

B.1.3.1 Overnight delivery of tapes to secondary siteThe standard classical Pickup Tape Access Method (PTAM) is used to send the tapes to the remote center.

B.1.3.2 Terminate PPRC secondary volumesThe volume pairs were in the suspended state. In order to make the secondary volumes available for use at the remote site, they must be returned to the simplex state —so they become addressable by the z/OS commands and utilities

B.1.3.3 Restore tapes onto secondary volumesHaving now the secondary volumes in the simplex state, we can run the utilities necessary for restoring the tapes onto disk. These restored data is still not valid for running the application — because it is a fuzzy copy of the primary data. In fact, the application at the primary site is still running — and has not stopped its write activity at any moment.

B.1.3.4 Establish XD pairsNow the pairs are re-synchronized — copy out-of-sync tracks only— in XD mode. This will make PPRC send the updated tracks to the secondary volumes. As PPRC started recording before the tape captures — suspended state reached before tape dumps were issued — then more updates than necessary will be transmitted.

B.1.3.5 Stop application at the primary centerNow the application at the primary center must be stopped — this is the necessary cutover of the application, needed to restart it at the remote location.

Overnight delivery of the tapes to the remote location

Terminate PPRC secondary volumes Secondary volumes become simplex and can be used in the remote site

Restore tapes onto the secondary volumesStill not usable because they have a fuzzy copy of the primary volumes

Establish pairs with XD option (copying out-of-sync tracks only)All updates since the first suspension sent to secondariesSome extra data is sentNo synchronous overhead for the application write activity

Stop application at primary centerNecessary to switch application to the remote locationWill leave the secondaries with a PiT checkpoint - ready to restart

Catch-up and terminateSynchronize the pairs (go-to-SYNC) and then terminate

Restart application at the remote site using the migrated data

No application disruption for the migration of the data. Catch-up donewith the planned application cutover shutdown


This shutdown of the application is used by PPRC-XD to do the catch-up of the volumes

B.1.3.6 Catch-up of the secondary volumes and terminateNow, with no more application writes, PPRC-XD will automatically do a catch-up of the volume pairs. At the end of this short catch-up, the secondaries will hold a PiT copy of the data — as of when the application was stopped for cutover.

Now the application can restart at the remote site, with the migrated data. This data migration — involving a large volume of data — has been accomplished with no disruption to the application processing.

B.2 PiT backup solutionIn this section we further describe the possibilities and discuss the considerations you must take into account when planning to use PPRC Extended Distance for PiT backup solutions.

B.2.1 Considerations when using PPRC-XDPPRC Extended Distance gives the possibility of mirroring data while the application is running — and without the distance restrictions and bandwidth requirements, typical of synchronous mirroring. This also means that PPRC Extended Distance provides additional flexibility when operated within the synchronous supported range (103 km).

This enhanced flexibility — within the synchronous PPRC range — means that you can have some volume pairs operating in synchronous way (PPRC-SYNC), and others operating in non-synchronous way (PPRC-XD).

The following considerations will help you when planning for implementations using PPRC Extended Distance for PiT backup solutions.

B.2.2 Distance considerationsAs mentioned previously, using PPRC Extended Distance within the synchronous range (103 km) — versus using PPRC beyond this distance limit — are two different situations that require different considerations.

One very important point — when using PPRC Extended Distance for PiT backup solutions — is reaching and verifying the duplex state condition — prior to the FlashCopy that will provide the tertiary PiT copy (see Figure B-3).

.

Figure B-3 PiT backup with PPRC XD

PPRC-XDnon-synchronous

mirroring

PrimaryPrimary SecondarySecondary TertiaryTertiary


Within synchronous PPRC rangeWhen using PPRC Extended Distance within the 103 km range, you can do a consistent PiT backup — without quiescing the application. The reason is that — within the 103 km range — you can transition PPRC states, from the PPRC-XD duplex pending XD state, to the PPRC-SYNC duplex state, without the liability of heavy write penalty — due to synchronous overhead over long distances.

When doing a freeze operation (once the duplex state has been reached), you get a consistent PiT copy of the data at the secondary. With this consistent secondary copy, you can then do a FlashCopy (onto tertiary volumes) and then the PPRC Extended Distance relationship can be resumed — between primaries and secondaries.

You are trading off some performance degradation on your applications’ write operations —while in synchronous mode— for not quiescing the application while doing the catch-up.

Exceeding synchronous PPRC rangeWhen using PPRC Extended Distance over distances beyond the 103 km range, you should be quiescing your application write activity when transitioning to duplex state —commanded go-to-SYNC transition— to fully synchronize the pairs.

This temporary quiesce will allow you to avoid the typical synchronous overhead on the primary volumes write operations. With this approach you will be able to get consistent PiT copies of the data — at remote locations over continental distances.

Though PPRC Extended Distance does not give you a high availability solution, it allows you to have frequent PiT copies of your data at a remote location — with which you can restart your applications within a short period of time and with minimal loss of data (very good RTO and RPO).

Trade off is, you need to quiesce the application write activity while going into synchronous mode — catch-up transition. to build consistent PiT checkpoint.

B.2.3 Other planning considerationsWhen planning to use PPRC Extended Distance for PiT backup solutions, you must also regard the following considerations:

� If you are going to have tertiary copies, then within the target LSS you should have an available set of volumes ready to become the FlashCopy target.

� If your next step is to dump the tertiary volumes onto tapes, then you must ensure that the tape resources are capable of handling this dump operations in between the PiT checkpoints — unless you have additional sets of volumes ready to become alternate FlashCopy targets within the secondary LSSs.

B.2.4 Impending disastersFor disaster situations where there is some time available before the forced shutdown must be done, you can proceed with PPRC Extended Distance as when producing PiT copies of your applications’ data.

For this situation, you would stop your applications (those for which you were going to do so anyway) before the forecasted disaster hit your primary center, and proceed with a catch-up before completely shutting down the systems. This will allow you to have — in a very short time — all your applications’ data replicated at the secondary site.


Appendix C. PPRC-XD Managing in the TSO environment

In this appendix we illustrate how the management of PPRC Extended Distance is done when using TSO commands.

In the first part, we illustrate a typical operation of PPRC Extended Distance using TSO commands. To further understand the setup sequence done during this operation, you can refer to the description in 3.1.2, “PiT Implementation basics” on page 27.

In the second part, we show a management situation when path failures occur.

C


C.1 Managing with TSO commandsThis section illustrates a typical operation of PPRC Extended Distance in a TSO environment — when a consistent PiT checkpoint is done.

To further understand the setup sequence executed with this operation, you can refer to the description in 3.1.2, “PiT Implementation basics” on page 27.

C.1.1 PPRC-XD Initialization and monitoringIn this first step of the operation, the paths are established and the pairs are established —in a PPRC Extended Distance relationship. Each operation is monitored doing the appropriate queries.

This PPRC environment is initialized as follows:

� One logical path is established between the primary and secondary LSSs.

� The LSSs’ 4-digits subsystem identifiers (SSID) are A763 and A762, respectively.

� The logical subsystem numbers for the LSSs are: primary X’03’, secondary X’02’.

� The path between the LSSs is defined without consistency grouping.

� The logical path uses the system adapter ID (SAID) X’0020’, using the ESCD exit port X’EA’, to then reach the secondary LSS X’02’.

The the PPRC Extended Distance pair relationship is established between volumes x’0F40’ and x’0F50’. When establishing the pair, the other options used are:

� OPTION(XD) — we are establishing the pair in a PPRC Extended Distance relationship.

� The default CRIT(NO) — CRIT(YES) is not valid in PPRC Extended Distance relationship.

� MODE(COPY), because we are doing an initial copy — MODE(NOCOPY) is not valid when establishing a PPRC Extended Distance relationship.

Figure C-1 TSO commands for establishing paths and pairs

Note: In this example, the operation of an individual volume pair is illustrated. In a situation where you have many volume pairs, automation procedures will be in place.

Establish path LSS 03 --> LSS 02 without consistency grouping

CESTPATH DEVN(X'0F40') PRIM(X'A763' FCA76 X'03') SEC(X'A762' FCA76 X'02') LINK(X'0020EA02') CGROUP(NO)

CQUERY DEVN(X'0F40') PATHS

Query the paths status

Establish PPRC Extended Distance pairs

CESTPAIR DEVN(X'0F40') PRIM(X'A763' FCA76 X'4F' X'03') SEC(X'A762' FCA76 X'10' X'02') MODE(COPY) OPTION(XD)


Figure C-1 illustrates the TSO commands given for establishing the paths — and for establishing the PPRC Extended Distance pair. Note the redundancy of specification information for the target LSS — in both SEC and LINK parameters.

Then we verify the result of these initial activities by querying the paths status (Figure C-2) and the volumes status (Figure C-3).

Figure C-2 CQUERY Paths

When querying the volume, you can see in Figure C-3 the PENDING.XD value of STATE — indicating that this volume is now in a PPRC Extended Distance relationship.

CQUERY DEVN(X'F40') PATHS

ANTP0095I CQUERY FORMATTED LVL 2PATHS REPORT*************** PPRC REMOTE COPY CQUERY - PATHS ********************* PRIMARY UNIT: SERIAL#= 0000000FCA76 SSID= A763 SS= 2105 LSS= 03 ** FIRST SECOND THIRD FOURTH ** SECONDARY SECONDARY SECONDARY SECONDARY **SERIAL NO: 0000000FCA76 ............ ............ ............ ** SSID LSS: A762 02 ....... ....... ....... ** PATHS: 1 0 0 0 ** SAID DEST S* SAID DEST S* SAID DEST S* SAID DEST S* ** --------- -- --------- -- --------- -- --------- -- ** 1: 0020 EA02 01 ---- ---- 00 ---- ---- 00 ---- ---- 00 ** 2: ---- ---- 00 ---- ---- 00 ---- ---- 00 ---- ---- 00 ** 3: ---- ---- 00 ---- ---- 00 ---- ---- 00 ---- ---- 00 ** 4: ---- ---- 00 ---- ---- 00 ---- ---- 00 ---- ---- 00 ** ** S* = PATH STATUS: ** 00=NO PATH 01=ESTABLISHED 02=INIT FAILED ** 03=TIME OUT 04=NO RESOURCES AT PRI 05=NO RESOURCES AT SEC** 06=SERIAL# MISMATCH 07=SEC SSID MISMATCH 08=ESCON LINK OFFLINE ** 09=ESTABLISH RETRY 0A=PATH ACTIVE TO HOST 0B=PATH TO SAME CLUSTR** 10=CONFIGURATION ERROR *********************************************************************

ANTP0001I CQUERY COMMAND COMPLETED FOR DEVICE 0F40. COMPLETION CODE: 00

Appendix C. PPRC-XD Managing in the TSO environment 133

Figure C-3 CQUERY Secondary Volume

As the primary volumes start to receive write activity, you then — when querying the primary volumes — commence to see the changing value of the TRACKS OUT OF SYNC information (refer to Figure C-4 and Figure C-5).

Figure C-4 CQUERY on primary XD volume - Out-of-sync tracks 19471

CQUERY DEVN(X'F50')

ANTP0090I CQUERY FORMATTED LVL 2VOLUME REPORT************** PPRC REMOTE COPY CQUERY - VOLUME ********************* (PRIMARY) (SECONDARY) ** SSID CCA LSS SSID CCA LSS**DEVICE LEVEL STATE PATH STATUS SERIAL# SERIAL# **------ --------- ---------- ----------- --------- --------- ** 0F50 SECONDARY PENDING.XD ACTIVE.. A763 4F 03 A762 10 02 ** ............... ........... ............ 0000000FCA76** PATHS SAID/DEST STATUS: DESCRIPTION ** ----- --------- ------ ------------------- ** 0 ---- ---- 00 NO PATH............ ** ---- ---- 00 NO PATH............ ** ---- ---- 00 NO PATH............ ** ---- ---- 00 NO PATH............ *********************************************************************ANTP0001I CQUERY COMMAND COMPLETED FOR DEVICE 0F50. COMPLETION CODE: 00

CQUERY DEVN(X'F40')

CQUERY FORMATTED LVL 2VOLUME REPORT************** PPRC REMOTE COPY CQUERY - VOLUME ********************* (PRIMARY) (SECONDARY) ** SSID CCA LSS SSID CCA LSS**DEVICE LEVEL STATE PATH STATUS SERIAL# SERIAL# **------ --------- ---------- ----------- --------- --------- ** 0F40 PRIMARY.. PENDING.XD ACTIVE.. A763 4F 03 A762 10 02 ** CRIT(NO)....... CGRPLB(YES) 0000000FCA76 0000000FCA76** PATHS SAID/DEST STATUS: DESCRIPTION ** ----- --------- ------ ------------------- ** 1 0020 EA02 01 PATH ESTABLISHED... ** ---- ---- 00 NO PATH............ ** ---- ---- 00 NO PATH............ ** ---- ---- 00 NO PATH............ ** IF STATE = PENDING/SUSPEND: TRACKS OUT OF SYNC = 19471 ** TRACKS ON VOLUME = 33390 ** PERCENT OF COPY COMPLETE = 42% *********************************************************************CQUERY COMMAND COMPLETED FOR DEVICE 0F40. COMPLETION CODE: 00


Figure C-5 CQUERY on primary XD volume. Out-of-sync tracks 3549

C.1.2 Catch-up transitionNow it is time to get a consistent point-in-time copy of your data at the remote site. Then at a scheduled point-in-time — when the application write activity is low (or none) — you trigger a go-to-SYNC operation (refer to Figure C-6).

CQUERY DEVN(X'F40')

ANTP0090I CQUERY FORMATTED LVL 2VOLUME REPORT************** PPRC REMOTE COPY CQUERY - VOLUME ********************* (PRIMARY) (SECONDARY) ** SSID CCA LSS SSID CCA LSS**DEVICE LEVEL STATE PATH STATUS SERIAL# SERIAL# **------ --------- ---------- ----------- --------- --------- ** 0F40 PRIMARY.. PENDING.XD ACTIVE.. A763 4F 03 A762 10 02 ** CRIT(NO)....... CGRPLB(YES) 0000000FCA76 0000000FCA76** PATHS SAID/DEST STATUS: DESCRIPTION ** ----- --------- ------ ------------------- ** 1 0020 EA02 01 PATH ESTABLISHED... ** ---- ---- 00 NO PATH............ ** ---- ---- 00 NO PATH............ ** ---- ---- 00 NO PATH............ ** IF STATE = PENDING/SUSPEND: TRACKS OUT OF SYNC = 3549 ** TRACKS ON VOLUME = 33390 ** PERCENT OF COPY COMPLETE = 90% *********************************************************************ANTP0001I CQUERY COMMAND COMPLETED FOR DEVICE 0F40. COMPLETION CODE: 00


Figure C-6 Catch-up using TSO

Figure C-7 illustrates the CESTPAIR command —and a subsequent QUERY command— used to command PPRC to do the catch-up transition — go-to-SYNC.

Figure C-7 CESTPAIR to command the go-to-SYNC transition (catch-up)

Application write activity should be minimum (or none) for this transition

If within synchronous supported distance (103 Km) you can trade off write performance to avoid application quiesce

CQUERY DEVN(X'0F40')

Check for low "tracks out-of-sync" value to go-to-SYNC

CSUSPEND DEVN(X'0F40') PRIM(X'A763' FCA76 X'4F' X'03')SEC(X'A762' FCA76 X'10' X'02')

Establish pair relationships with OPTION(SYNC)

CESTPAIR DEVN(X'0F40') PRIM(X'A763' FCA76 X'4F' X'03')SEC(X'A762' FCA76 X'10' X'02') MODE(RESYNC) OPTION(SYNC)

Immediately upon reaching the duplex state, volume pairs are suspended

CESTPAIR DEVN(X'F40') PRIM(X'A763' FCA76 X'4F' X'03') SEC(X'A762' FCA76 X'10' X'02') MODE(RESYNC) OPTION (SYNC)ANTP0001I CESTPAIR COMMAND COMPLETED FOR DEVICE 0F40. COMPLETION CODE: 00

CQUERY DEVN(X'F40')ANTP0090I CQUERY FORMATTED LVL 2VOLUME REPORT************** PPRC REMOTE COPY CQUERY - VOLUME ********************* (PRIMARY) (SECONDARY) ** SSID CCA LSS SSID CCA LSS**DEVICE LEVEL STATE PATH STATUS SERIAL# SERIAL# **------ --------- ---------- ----------- --------- --------- ** 0F40 PRIMARY.. DUPLEX.... ACTIVE.. A763 4F 03 A762 10 02 ** CRIT(NO)....... CGRPLB(YES) 0000000FCA76 0000000FCA76** PATHS SAID/DEST STATUS: DESCRIPTION ** ----- --------- ------ ------------------- ** 1 0020 EA02 01 PATH ESTABLISHED... ** ---- ---- 00 NO PATH............ ** ---- ---- 00 NO PATH............ ** ---- ---- 00 NO PATH............ *********************************************************************ANTP0001I CQUERY COMMAND COMPLETED FOR DEVICE 0F40. COMPLETION CODE: 00


C.1.3 Build consistency and freezeTo get a consistent PiT copy of the data at the recovery site, we quiesce the application writes and then re-synchronize the pairs — followed by a freeze (see Figure C-8).

The freeze operation is an efficient way of — all at once — stopping the propagation of updates onto the secondary volumes — by suspending the primaries and terminating the paths.

Figure C-8 Build global consistency at the recovery site

C.1.4 Resume application write activityNow with the secondaries holding an consistent PiT checkpoint, and with the volume pairs still suspended — no propagation of updates onto secondaries — then the application write activity can be resumed. Refer to Figure C-9.

Figure C-9 Resuming write updates onto primary volumes

C.1.5 Save checkpoint and resume PPRC-XD Before resuming the PPRC Extended Distance mirroring onto the secondaries, these have to be saved into a tertiary set of volumes. This can be accomplished making a FlashCopy.

Once the FlashCopy establish relationship is done, then the paths between primary and secondary LSSs can be redefined — they were terminated by the freeze operation — and the PPRC Extended Distance mirroring relationships can be resumed. This sequence is summarized in Figure C-10.

From there on, the secondaries will again start to hold a fuzzy copy of the primaries — until the next checkpoint.

Re synchronize the pairs so a PiT consistent checkpoint can be reached

When all volumes in duplex state, then freeze the application volume setfreeze of the application-related LSS pairs

CGROUP DEVN(X'0F40') PRIM(X'A763' FCA76 X'03') SEC(X'A762' FCA76 X'02') FREEZE

Quiesce application write activity

CESTPAIR DEVN(X'0F40') PRIM(X'A763' FCA76 X'4F' X'03')SEC(X'A762' FCA76 X'10' X'02') MODE(RESYNC) OPTION(SYNC)

Application write activity resumedPrimaries receiving write updates - in suspended statePPRC keeping record of updated primary tracksSecondaries not receiving updates - volume pairs are still suspended


Figure C-10 Save secondary copy and resume PPRC-XD copy

There are two ways for launching the FlashCopy operation:

� With a TSO command� With the z/OS DFSMS/dss utility.

Figure C-11 illustrates how FlashCopy is invoked using the DFSMS/dss utility.

Figure C-11 z/OS DFSMS/dss FlashCopy invocation

Figure C-12 and Figure C-13 show the JES2 JOB LOG with the execution of the FlashCopy job.

//FCEXAMPL JOB ,'MCKINNON',MSGLEVEL=(1,1),TIME=(5,0),REGION=4096K, // MSGCLASS=A,CLASS=A //* //STEPT001 EXEC PGM=ADRDSSU //SYSPRINT DD SYSOUT=* //SYSIN DD * COPY - FULL - INDYNAM (D9S3S1) - OUTDYNAM (D9S3S0) - COPYVOLID /*

FlashCopy saves secondary consistent checkpoint on tertiary TSO command or z/OS DFSMS/dss COPY FULL

fcestabl sdevn(x'0F50') tdevn(x'0F51') mode(copy)

Application writes were already updating primaries, but still not secondariesWhen the FlashCopy is established, then the updates onto secondaries can be resumed

FlashCopy establishes relationship before physical copy endsDFSMS/dss issues ADR806I message on SYSOUT (not SYSLOG console)

FlashCopy job step ends when FlashCopy established

CESTPATH DEVN(X'0F40') PRIM(X'A763' FCA76 X'03') SEC(X'A762' FCA76 X'02') LINK(X'0020EA02') CGROUP(NO)

Establish PPRC Extended Distance pairs

CESTPAIR DEVN(X'0F40') PRIM(X'A763' FCA76 X'4F' X'03') SEC(X'A762' FCA76 X'10' X'02') MODE(RESYNC) OPTION(XD)

Paths are reestablished between LSSs


Figure C-12 z/OS DFSMS/dss FlashCopy Execution (1)

J E S 2 J O B L O G -- S Y S T E M 3 0 9 0 -- N O D E T P C 9 5 3 08.02.23 JOB00903 ---- WEDNESDAY, 06 MAR 2002 ---- 08.02.23 JOB00903 $HASP373 FCEXAMPL STARTED - INIT 1 - CLASS A - SYS 3090 08.02.23 JOB00903 IEF403I FCEXAMPL - STARTED - TIME=08.02.23 08.02.24 JOB00903 DFPWTX00 IEAVMXIT REPLYING TO WTOR. 08.02.24 JOB00903 REPLY 30,U 08.02.24 JOB00903 REPLY 30,U 08.02.24 JOB00903 DFPWTX00 IEAVMXIT REPLYING TO WTOR. 08.02.24 JOB00903 REPLY 30,U 08.02.24 JOB00903 REPLY 30,U 08.02.24 JOB00903 *30 ADR369D AUTHORIZE FOR WRITE ACCESS A VTOCIX DATA SET ON D9S3S0, FCEXAMPL, STEPT001, REPLY U OR T 08.02.27 JOB00903 ADR320I (001)-SBRTN(01), VOLUME SERIAL D9S3S0 ON UNIT 0F50 IS CHANGED 066 066 TO D9S3S1 08.02.27 JOB00903 VARY 0F50,OFFLINE DFDSS INTERNAL VARY 08.02.27 JOB00903 ADR344I (001)-SBRTN(01), VOLSER ON UCB 0F50 IS A DUPLICATE. VOLUME MADE 068 068 UNAVAILABLE. 08.02.27 JOB00903 FCEXAMPL STEPT001 ADRDSSU 0000 08.02.27 JOB00903 IEF404I FCEXAMPL - ENDED - TIME=08.02.27 08.02.27 JOB00903 $HASP395 FCEXAMPL ENDED ------ JES2 JOB STATISTICS ------ 06 MAR 2002 JOB EXECUTION DATE 21 CARDS READ 77 SYSOUT PRINT RECORDS 0 SYSOUT PUNCH RECORDS 4 SYSOUT SPOOL KBYTES 0.06 MINUTES EXECUTION TIME 1 //FCEXAMPL JOB ,'DSSY',MSGLEVEL=(1,1),TIME=(5,0),REGION=4096K, JOB00903 // MSGCLASS=A,CLASS=A //*------------------------------------------------*/ //* TEST ID: TSTRCP2B PART 1 OF 1 (6525557) */ //* SOURCE ID: DSSY AT SJFEVMX */ //* TARGET ID: DSSY AT SJFEVMX */ //* VERSION: 5.1, Jul 7, 1987 */ //* WHEN SENT: 03/06/02 07:05:57 */ //*------------------------------------------------*/ //* JOBPARM SYSAFF not altered by PU /*ROUTE PRINT SJFEVMX/DSSZ //* 2 //STEPT001 EXEC PGM=ADRDSSU 3 //SYSPRINT DD SYSOUT=* 4 //SYSIN DD *


Figure C-13 z/OS DFSMS/dss FlashCopy Execution (2)

Note (in Figure C-13) the ADR806I message indication of logical completion of the FlashCopy. This message only appears in the joblog — not in the console syslog.

IEF236I ALLOC. FOR FCEXAMPL STEPT001 IEF237I JES2 ALLOCATED TO SYSPRINT IEF237I JES2 ALLOCATED TO SYSIN IEF237I 0F51 ALLOCATED TO SYS00001 IEF237I 0F50 ALLOCATED TO SYS00002 DFPWTX00 IEAVMXIT REPLYING TO WTOR. DFPWTX00 IEAVMXIT REPLYING TO WTOR. IEF285I SYS02065.T080224.RA000.FCEXAMPL.R0106930 KEPT IEF285I VOL SER NOS= D9S3S1. IEF285I SYS02065.T080224.RA000.FCEXAMPL.R0106931 KEPT IEF285I VOL SER NOS= D9S3S0. IEF142I FCEXAMPL STEPT001 - STEP WAS EXECUTED - COND CODE 0000 IEF285I ++++++++.FCEXAMPL.JOB00903.D0000102.? SYSOUT IEF285I ++++++++.FCEXAMPL.JOB00903.D0000101.? SYSIN IEF373I STEP/STEPT001/START 2002065.0802 IEF374I STEP/STEPT001/STOP 2002065.0802 CPU 0MIN 00.04SEC SRB 0MIN 00.01SEC VIRT 1756K SYS 268K EXT 560K SYS 11480K IEF375I JOB/FCEXAMPL/START 2002065.0802 IEF376I JOB/FCEXAMPL/STOP 2002065.0802 CPU 0MIN 00.04SEC SRB 0MIN 00.01SEC PAGE 0001 5695-DF175 DFSMSDSS V1R3.0 DATA SET SERVICES 2002.065 08:02 COPY - FULL - INDYNAM (D9S3S1) - OUTDYNAM (D9S3S0) - COPYVOLID ADR101I (R/I)-RI01 (01), TASKID 001 HAS BEEN ASSIGNED TO COMMAND 'COPY ' ADR109I (R/I)-RI01 (01), 2002.065 08:02:24 INITIAL SCAN OF USER CONTROL STATEMENTS COMPLETED. ADR016I (001)-PRIME(01), RACF LOGGING OPTION IN EFFECT FOR THIS TASK ADR006I (001)-STEND(01), 2002.065 08:02:24 EXECUTION BEGINS ADR241I (001)-DDTFP(01), TARGET VTOC BEGINNING AT 000000:0001 AND ENDING AT 000001:0014 IS OVERLAID ADR806I (001)-T0MI (02), VOLUME COPIED USING A FAST REPLICATION FUNCTION. ADR320I (001)-SBRTN(01), VOLUME SERIAL D9S3S0 ON UNIT 0F50 IS CHANGED TO D9S3S1 ADR344I (001)-SBRTN(01), VOLSER ON UCB 0F50 IS A DUPLICATE. VOLUME MADE UNAVAILABLE. ADR006I (001)-STEND(02), 2002.065 08:02:27 EXECUTION ENDS ADR013I (001)-CLTSK(01), 2002.065 08:02:27 TASK COMPLETED WITH RETURN CODE 0000 ADR012I (SCH)-DSSU (01), 2002.065 08:02:27 DFSMSDSS PROCESSING COMPLETE. HIGHEST RETURN CODE IS 0000


C.2 Path failures scenarioThis section illustrates a situation where path failures occur —in a TSO environment.

The PPRC environment in our example has four PPRC links, each with a path defined. PPRC is shadowing four volumes — an IEBDG utility is permanently making updates onto each volume.

Figure C-14 shows the query output information when two of the links — SAID x’0029’ and SAID x’0089’ — are already interrupted.

Figure C-14 Initial path status, with two paths already interrupted

Figure C-15 shows the resulting messages when the third link —SAID x’0009’— is interrupted.

Figure C-15 The third link is interrupted

The message IOS581E (in Figure C-15) is issued by the z/OS Input/Output Supervisor — to keep trace of the error on this attempted I/O operation. The I/O operation will be automatically retried over the remaining paths — without sending a unit-check to the application if it is successfully completed.

ANTP0095I CQUERY FORMATTED LVL 2 572 PATHS REPORT *************** PPRC REMOTE COPY CQUERY - PATHS ******************** * PRIMARY UNIT: SERIAL#= 000000013895 SSID= 0023 SS= 2105 * * FIRST SECOND THIRD FOURTH * * SECONDARY SECONDARY SECONDARY SECONDARY * *SERIAL NO: 000000013881 ............ ............ ............ * * SSID: 0043 .... .... .... * * PATHS: 4 0 0 0 * * SAID DEST S* SAID DEST S* SAID DEST S* SAID DEST S* * * --------- -- --------- -- --------- -- --------- -- * * 1: 0009 0006 01 ---- ---- 00 ---- ---- 00 ---- ---- 00 * * 2: 0029 0006 09 ---- ---- 00 ---- ---- 00 ---- ---- 00 * * 3: 0089 0006 09 ---- ---- 00 ---- ---- 00 ---- ---- 00 * * 4: 00A9 0006 01 ---- ---- 00 ---- ---- 00 ---- ---- 00 * * * * S* = PATH STATUS: * * 00=NO PATH 01=ESTABLISHED 02=INIT FAILED * * 03=TIME OUT 04=NO RESOURCES AT PRI 05=NO RESOURCES AT SEC* * 06=SERIAL# MISMATCH 07=(RESERVED) 08=(RESERVED) * * 09=(RESERVED) 10=CONFIGURATION ERROR * ********************************************************************

IOS581E LINK FAILED REPORTING CHPID=8C 574 INCIDENT UNIT TM=002105/F20 SER=IBMBM-013895 IF=0009 IC=03 ATTACHED UNIT TM=002105/F20 SER=IBMBM-013881 IF=0009 IOS581E LINK FAILED REPORTING CHPID=8F 575 INCIDENT UNIT TM=002105/F20 SER=IBMBM-013881 IF=0009 IC=03


Figure C-16 illustrates the query output information when the third link is not working.

Figure C-16 CQUERY path status report after third link was interrupted

ANTP0095I CQUERY FORMATTED LVL 2 576 PATHS REPORT *************** PPRC REMOTE COPY CQUERY - PATHS ******************** * PRIMARY UNIT: SERIAL#= 000000013895 SSID= 0023 SS= 2105 * * FIRST SECOND THIRD FOURTH * * SECONDARY SECONDARY SECONDARY SECONDARY * *SERIAL NO: 000000013881 ............ ............ ............ * * SSID: 0043 .... .... .... * * PATHS: 4 0 0 0 * * SAID DEST S* SAID DEST S* SAID DEST S* SAID DEST S* * * --------- -- --------- -- --------- -- --------- -- * * 1: 0009 0006 09 ---- ---- 00 ---- ---- 00 ---- ---- 00 * * 2: 0029 0006 09 ---- ---- 00 ---- ---- 00 ---- ---- 00 * * 3: 0089 0006 09 ---- ---- 00 ---- ---- 00 ---- ---- 00 * * 4: 00A9 0006 01 ---- ---- 00 ---- ---- 00 ---- ---- 00 * * * * S* = PATH STATUS: * * 00=NO PATH 01=ESTABLISHED 02=INIT FAILED * * 03=TIME OUT 04=NO RESOURCES AT PRI 05=NO RESOURCES AT SEC* * 06=SERIAL# MISMATCH 07=(RESERVED) 08=(RESERVED) * * 09=(RESERVED) 10=CONFIGURATION ERROR * ******************************************************************** ANTP0001I CQUERY COMMAND COMPLETED FOR DEVICE 20C0. COMPLETION CODE: 00


Finally — when we drop the last remaining link — the system sends the error messages shown in Figure C-17.

Figure C-17 Interrupting the last path — link failure messages

Notice message IOS581E for the last link failure — one per channel attached to the ESS. Then there is a state change interrupt message for each of the volume pairs — IEA494I informing the volumes became suspended. And finally, there are four unit-check messages — IEA491E.

IOS581E LINK FAILED REPORTING CHPID=EB 588 INCIDENT UNIT TM=002105/F20 SER=IBMBM-013881 IF=00A9 IC=03IOS581E LINK FAILED REPORTING CHPID=E8 589 INCIDENT UNIT TM=002105/F20 SER=IBMBM-013895 IF=00A9 IC=04 ATTACHED UNIT TM=002105/F20 SER=IBMBM-013881 IF=00A9 IEA494I 2080,RS2080,PPRC PAIR SUSPENDED,SSID=0022,CCA=00 IEA494I 20C0,RS20C0,PPRC PAIR SUSPENDED,SSID=0023,CCA=00 *IEA491E RS20C0,PPRC SUSPENDED, COMMUNICATION_TO_SECONDARY_FAILURE, (PRI)SER=01BM-13895,CCA=00 (SEC)SER=0100-13881,CCA=00 CONTINUATION OF IEA491E SNS=10901000000000FB3200040000363905740036470023 FE32000060800B000000 *IEA491E RS20C1,PPRC SUSPENDED, COMMUNICATION_TO_SECONDARY_FAILURE, (PRI)SER=01BM-13895,CCA=01 (SEC)SER=0100-13881,CCA=01 CONTINUATION OF IEA491E SNS=10901000010000FB3201040000363905740036470023 FE32000060800B000000 IEA494I 20C1,RS20C1,PPRC PAIR SUSPENDED,SSID=0023,CCA=01 *IEA491E RS2080,PPRC SUSPENDED, COMMUNICATION_TO_SECONDARY_FAILURE, (PRI)SER=01BM-13895,CCA=00 (SEC)SER=0100-13881,CCA=00 CONTINUATION OF IEA491E SNS=10901000000000FB3200040000363905740036470022 FE32000060820B000000 *IEA491E RS2081,PPRC SUSPENDED, COMMUNICATION_TO_SECONDARY_FAILURE, (PRI)SER=01BM-13895,CCA=01 (SEC)SER=0100-13881,CCA=01 CONTINUATION OF IEA491E SNS=10901000010000FB3201040000363905740036470022 FE32000060820B000000 IEA494I 2081,RS2081,PPRC PAIR SUSPENDED,SSID=0022,CCA=01


Figure C-18 shows the output information we obtain when querying the final status of the paths.

Figure C-18 Final status path report — none of the paths are operative

ANTP0095I CQUERY FORMATTED LVL 2 602 PATHS REPORT *************** PPRC REMOTE COPY CQUERY - PATHS ******************** * PRIMARY UNIT: SERIAL#= 000000013895 SSID= 0023 SS= 2105 * * FIRST SECOND THIRD FOURTH * * SECONDARY SECONDARY SECONDARY SECONDARY * *SERIAL NO: 000000013881 ............ ............ ............ * * SSID: 0043 .... .... .... * * PATHS: 4 0 0 0 * * SAID DEST S* SAID DEST S* SAID DEST S* SAID DEST S* * * --------- -- --------- -- --------- -- --------- -- * * 1: 0009 0006 09 ---- ---- 00 ---- ---- 00 ---- ---- 00 * * 2: 0029 0006 09 ---- ---- 00 ---- ---- 00 ---- ---- 00 * * 3: 0089 0006 09 ---- ---- 00 ---- ---- 00 ---- ---- 00 * * 4: 00A9 0006 09 ---- ---- 00 ---- ---- 00 ---- ---- 00 * * * * S* = PATH STATUS: * * 00=NO PATH 01=ESTABLISHED 02=INIT FAILED * * 03=TIME OUT 04=NO RESOURCES AT PRI 05=NO RESOURCES AT SEC* * 06=SERIAL# MISMATCH 07=(RESERVED) 08=(RESERVED) * * 09=(RESERVED) 10=CONFIGURATION ERROR * ******************************************************************** ANTP0001I CQUERY COMMAND COMPLETED FOR DEVICE 20C0. COMPLETION CODE: 00


Appendix D. PPRC-XD operation for z/VM and VSE/ESA users

PPRC Extended Distance is managed using the ESS Copy Services Web user interface (WUI) and — for selected servers — using the ESS Copy Services command line interface (CLI). In the z/OS and OS/390 environments, PPRC Extended Distance can be managed using TSO commands.

Users who currently require ICKDSF to manage PPRC will need to use the ESS Copy Services Web user interface, for some activities and transitions related to PPRC Extended Distance operation.

This appendix will aid current ICKDSF users in identifying and correlating the different ICKDSF command functions versus the ESS Copy Services Web user interface functions and terms.

D


D.1 ICKDSF PPRC command functionsUsers who require ICKDSF to control and manage Peer-to-Peer Remote Copy (PPRC) command functions, are required to use the ESS Copy Services Web user interface capabilities — for managing PPRC Extended Distance.

The information in this appendix will aid in identifying the different ICKDSF command functions and terms vs. the ESS Copy Services Web user interface (WUI) functions and terms.

The documents referenced in this appendix are:

� The new IBM TotalStorage Enterprise Storage Server PPRC Extended Distance, SG24-6568 redbook

� and the IBM TotalStorage Enterprise Storge Server Web Interface User’s Guide, SC26-7346

The format in the following explanations shows the ICKDSF PPRCOPY command functions, and then the reference to the corresponding description in the Web Interface users' guide, and then the reference to those in the redbook.

D.2 PPRCOPY ESTPATHEstablishing paths between the primary and secondary LSSs.

Web interface users' guide:Chapter 3: Using ESS Copy Services

Section: Using the Paths panelSubsection: Establishing paths

PPRC Extended Distance redbook

Appendix A, “ESS Copy Services creation of tasks” on page 95A.1, “ESS Copy Services tasks: step-by-step” on page 96

A.1.1, “Establish paths” on page 96

D.3 PPRCOPY DELPATHDeleting paths between the primary and secondary LSSs.

Web interface users' guide:

Chapter 3: Using ESS Copy ServicesSection: Using the Paths panel

Subsection: Removing pathsSection: Using the Logical Subsystems panel

Subsection: Removing orphaned paths

Note: MSGREQ is not required for ESS Copy Services WUI support.


D.4 PPRCOPY ESTPAIREstablishing pairs between primary and secondary disks.


Chapter 3: Using ESS Copy ServicesSection: Using the Volumes panel

Subsection: Establishing a synchronous PPRC copy pairSubsection: Establishing an extended distance PPRC copy pairSubsection: Converting an extended distance PPRC pair to a synchronous pairSubsection: Resynchronizing PPRC copy pairs

Section: Using the Logical Subsystems panelSubsection: Establishing PPRC copy pairs for all volumes in a source LSS and

all volumes in a target LSSSubsection: Resynchronizing PPRC copy pairs for al volumes in a source LSS

and all volumes in a target LSS



A.1.2, “Establish PPRC XD pair — initial copy” on page 98A.1, “ESS Copy Services tasks: step-by-step” on page 96

A.1.3, “Establish PPRC SYNC pair — go-to-SYNC and suspend” on page 102

Note: MSGREQ and PACE are not required for ESS Copy Services WUI support

D.5 PPRCOPY DELPAIRDeleting pairs between primary and secondary disks.



Subsection: Terminating a PPRC copy pairSubsection: Terminating a PPRC copy pair when only one volume is visible to

the ESS Copy Services clientSection: Using the Logical Subsystems panel

Subsection: Terminating PPRC copy pairs for all volumes in a source LSS and all volumes in a target LSS



A.1.13, “Terminate PPRC pair” on page 115

Note: MSGREQ is not required for ESS Copy Services WUI support.

D.6 PPRCOPY SUSPENDSuspending pairs of disks.

Appendix D. PPRC-XD operation for z/VM and VSE/ESA users 147



Subsection: Suspending a PPRC copy pairSubsection: Suspending a PPRC copy pair when only one volume is visible to

the ESS Copy Services clientSection: Using the Logical Subsystems panel

Subsection: Suspending PPRC copy pairs for all volumes in a source LSS and all volumes in a target LSS



A.1.12, “Suspend PPRC pair” on page 114

Note: PRIMAINT is not supported by the ESS Copy Services WUI.

D.7 PPRCOPY RECOVERRecovering disks on the secondary system.



Subsection: Terminating a PPRC copy pair when only one volume is visible to the ESS Copy Services client

Section: Using the Logical Subsystems panelSubsection: Terminating PPRC copy pairs for all volumes in a source LSS and

all volumes in a target LSSChapter 4: Disaster Recovery

Note: VERIFY/NOVERIFY is not supported by the ESS Copy Services WUI. You cannot modify the volume labels using the WUI.

D.8 PPRCOPY QUERYQuerying the status of volumes and paths.

With PATHS parameter, Web interface users' guide:

Chapter 3: Using ESS Copy ServicesSection: Using the Paths panel

Subsection: Viewing information about paths

Without PATHS parameter, Web interface users' guide:

Chapter 3: Using ESS Copy ServicesSection: Using the Logical Subsystems panel

Subsection: Viewing information about a volume


Appendix E. PPRC with static volumes implementations

In this appendix we present some general examples — not detailed implementations— of solutions using PPRC-SYNC with static primary volumes.

E


PPRC with static primary volumesAs already discussed in 2.6, “Synchronous PPRC using static volumes” on page 14, synchronous PPRC, using static primary volumes, is a powerful solution for data copy, data migration, and off-site backup of static volumes — at very long distances, with excellent throughput performance.

In this section we present a high level overview of the possible solutions that can be implemented — when using PPRC with static volumes.

Remote data migrationFigure E-1 shows a remote data migration scenario, using synchronous PPRC with static volumes. As the primary volumes are static (not receiving write updates), the PPRC pairs can be established — doing the initial full volume copy — to migrate the static primary volumes.

Figure E-1 PPRC-SYNC with static volumes for remote data migration

PPRCESTABLISH copy at long

distance

B

L

X

C

M

Y

ESS 2:MirroredData

Production Servers



volumes are not changing


Transmission of database log filesAs you can see from the example in Figure E-2, the application switches the active log from log1 volume to log2 volume — leaving log1 volume static, and available for transmission. Then an initial establish of the pair is done — to send log1 over the network to its secondary pair. Meanwhile the application is using the active log2. Once the PPRC copy is completed, the secondary — log1’ —copy is available to be applied on the shadow database. This procedure is cycled, successively switching the active logs.

Figure E-2 PPRC-SYNC with static volumes for transmission of database log files

#2: PPRCRemote Copy ESTABLISH

to send LOG1 to remote site

log1

log2

log3

log1'

log2'

log3'ESS 2:MirroredData

Production Servers



#1: Application dynamically switches from log1 to log2

D#3: Shadow process applies log1' to shadow database

Appendix E. PPRC with static volumes implementations 151

Remote backupIn a situation where you have to produce a remote backup daily — for example, at the end of the online shift — the very first time you do an initial copy of the volumes pairs. After this initial backup, the pairs are suspended (see Figure E-3 and Figure E-4).

Figure E-3 PPRC-SYNC with static volumes for remote backup — establish initial copy

Figure E-4 PPRC-SYNC with static volumes — suspend pairs after initial copy

PPRCestablish initial copy

B

L

X

C

M

Y

ESS 2:MirroredData

Production Servers



PPRC Split

B

L

X

C

M

Y

ESS 2:MirroredData

Production Servers



Once SUSPEND is complete, applications update production data. PPRC tracks incremental changes while SUSPENDed.

SUSPEND


WIth the volumes in suspended state, the application updates, during the subsequent online shift, will not be transmitted to the secondary volumes. During this time, PPRC is keeping track of the changed tracks. And — last but not least — the application write performance during the online shift is not affected by any synchronous transmission overhead. Please refer to Figure E-5.

Figure E-5 PPRC keeps track of incremental modifications while volumes suspended

The suspended state can be maintained during the whole online shift. Then, at the end of the online shift, the application is quiesced — so no updates occur on the primary volumes — and a PPRC-SYNC re-synchronization is done, sending only the incremental updates (see Figure E-6).

B

L

X

C

M

Y

ESS 2:MirroredData

Production Servers



PPRC maintains bitmap of incremental changes

No write performance impact when PPRC is SUSPEND'ed

Appendix E. PPRC with static volumes implementations 153

Figure E-6 PPRC-SYNC with static volumes — incremental copy in subsequent backups

Then, when the volumes become fully synchronized in duplex state — this day’s off-site backup is done — then the pairs are suspended again. And so, one cycle is completed, and volumes are ready for next day subsequent cycle:

1. During the online shift the volumes will remain suspended — PPRC will keep track of updates; updates will not be propagated to the secondary volumes; application not impacted.

2. End of online shift — quiesce application so volumes become static.

3. Re-synchronize the pairs — PPRC sends only incremental changes.

4. Then suspend again — daily backup is done.

5. Immediately restart the application write activity — volumes are suspended.

6. Leave the pairs suspended — ready to repeat subsequent cycle.

With this process you can send your daily backups off-site — at very long distances, beyond the synchronous 103 km supported distance.

SUSPENDand then

B

L

X

C

M

Y

ESS 2:MirroredData

Production Servers



Applications quieseced, for example, end of day, prior to start of batch run

PPRCre synchronization

incrementalincrementalcopycopy


Related publications

The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this redbook.

IBM RedbooksFor information on ordering these publications, see “How to get IBM Redbooks” on page 155.

� IBM TotalStorage Solutions for Disaster Recovery, SG24-6547

� Implementing ESS Copy Services on S/390, SG24-5680

� Implementing ESS Copy Services on UNIX and Windows NT/2000, SG24-5757

Other resourcesThese publications are also relevant as further information sources:

� IBM TotalStorage Enterprise Storge Server Web Interface User’s Guide, SC26-7346

� IBM TotalStorage Enterprise Storage Server Copy Services Command-Line Interface Reference, SC26-7434

� z/OS DFSMS Advanced Copy Services, SC35-0428

Referenced Web sitesThese Web sites are also relevant as further information sources:

� PPRC Extended Distance support information


How to get IBM RedbooksYou can order hardcopy Redbooks, as well as view, download, or search for Redbooks at the following Web site:

ibm.com/redbooks

You can also download additional materials (code samples or diskette/CD-ROM images) from that site.

IBM Redbooks collectionsRedbooks are also available on CD-ROMs. Click the CD-ROMs button on the Redbooks Web site for information about all the CD-ROMs offered, as well as updates and formats.


http://www.redbooks.ibm.com/




Index

Aapplets when using WUI 75application availability

when using static volumes 16with PPRC-SYNC 10with PPRC-XD 22

application recoverywith PPRC-XD 58, 63

automation 4, 58, 64, 69

Bbandwidth management 67, 69batch window reduction 66business objectives

RPO 12, 18, 23, 66RTO, RPO, NRO 3

Ccatch-up 33, 68

and suspend using CLI 78, 85and suspend using WUI 36, 102description 20task creation 102uses 29, 59, 62, 65–66, 128using TSO 33, 135within synchronous range 129

CESTPAIR command 32, 132parameters’ values 34

CESTPATH command 51, 132CGROUP freeze command 54, 137channel extenders 8, 11, 14, 17–18, 22, 44, 71CLI

catch-up and suspend 85consistency group created 92creation of tasks for CLI execution 96establishing XD pair 84, 91FlashCopy task 90freeze 88go-to-SYNC and suspend 85installation 118path definition 83, 91PiT backup sequence of tasks 82re synchronization of pairs 87rsQuery command 39, 84, 86rsQueryComplete command 38, 86

Command Line Interface, see CLIconnectivity 70consistency group 50

created task 82, 92, 117defining with the WUI 51defining with TSO 51

consistency management 69consistent PiT checkpoint 29, 58, 60, 80, 89, 132

© Copyright IBM Corp. 2002

continental distances 70CQUERY command 38, 133CRIT setting 27

Ddark fibre 70data consistency 49

consistency groups 50freeze 53, 60, 79, 137hardware vs transaction 4management 69on PiT copy 29, 58, 60, 80, 89, 132with PPRC-SYNC 10with PPRC-XD 21with static volumes 16

data migration 65, 150using tapes 124, 127

defining paths 44, 77, 81, 91CESTPATH command 51task creation with the WUI 96, 111using the CLI 83using TSO commands 132

disaster recoverydefinition 2solutions 5tiers 4

distancePPRC-XD efficiency 27supported 70with PPRC-XD 8, 128within synchronous range 62

distance factorwith PPRC-SYNC 11with PPRC-XD 22with static volumes 17

duplex pending XD volume state 19, 30duplex volume state 13DWDM 11, 71

EELB 50, 53, 77, 117ESS Copy Services

creation of tasks 75ESS Specialist 74establishing an XD pair 32, 61, 77, 81

task creation with the WUI 98, 111using the CLI 84using the WUI 34using TSO commands 32, 132

Extended Distance PPRCand consistency grouping 53application availability 22bandwidth management 67, 69data consistency 21

157

distance 22, 62, 70, 128exceeding synchronous range 129for application recovery 58implementation considerations 68long distance solution 27management 9, 32operating characteristics 18, 26overview 8–9, 18, 26PiT copy implementation 27, 66, 76, 132positioning 22, 63, 67queries 37, 84, 86, 133state change logic 30support 9, 71supported distances 70, 128uses 18, 22, 64, 76, 124, 128within synchronous range 62, 129write operations 18, 22

extended long busy, see ELB

Ffailing paths 45, 141FlashCopy

onto tertiaries 30, 61, 80, 90, 108rsQuery 90task creation 108TSO example 137withdraw 82

task creation 113freeze 53

CGROUP command 54, 137task creation with the WUI 106to build consistency 60, 79, 137using the CLI 88using the WUI 55

fuzzy copy, dependent writes 21, 27

Ggo-to-SYNC

and suspend 30, 59and suspend using CLI 78, 85and suspend using WUI 36, 102description 20task creation 102uses 29, 59, 62, 65, 128using TSO 33, 135within synchronous range 129

Hhardware requirements 9, 71

IICKDSF support 9, 32, 146icons

duplex pending XD 76IEA494 message 69implementation considerations 68Information Panel 40installation of the CLI 118

intial copy 32

JJava Virtual Machine

applets 75ESS Specialist support 74

Llinks 44long distance

solution using static volumes 14solution with PPRC-XD 27support with PPRC-XD 70

Mmanaging PPRC 9managing PPRC Extended Distance 32

using the CLI 82using TSO commands 132

message IEA494 69mixed environment

XD and SYNC 67more data protected 69

Nnon-synchronous 9, 19, 26–27, 63, 68, 70

write operations 22

Ooff-site backup 66, 152operating characteristics

PPRC-SYNC 10PPRC-XD 18, 26

optionSYNC 27table for WUI 37XD 27

outagesplanned, unplanned 3

Pparameters’ values

CESTPAIR command 34path

definition 44, 51, 77, 81, 83, 91, 132failure 45, 141information 46

pending volume state 13PiT copy

considerations 128consistent checkpoint 58, 80, 89, 137data consistency 21, 29, 58implementation basics 27solution for application recovery 58tasks creation 96tasks for PiT backup 76


using the CLI 82within synchronous range 62

planned outages 3point-in-time, see PiT copypositioning

performance 67PPRC-SYNC 11PPRC-XD 22, 63static volumes 17

PPRCconfiguration 44connectivity 70operation modes 9, 26overview 8volume states 12, 30

PPRC consistency created 55PPRCOPY command 146PPRC-SYNC, see synchronous PPRCPPRC-XD, see Extended Distance PPRC

Qqueries

CQUERY command 38, 133in PPRC-XD 37Information Panel 40rsQuery command 39, 84, 86rsQueryComplete 38, 86volume states 31

quiesce application 29, 60, 62, 78, 87, 104

Rre synchronization 32, 79

task creation with the WUI 104with the CLI 87with TSO command 136

Redbooks Web site 155Contact us xviii

reestablish 32PPRC-XD relationship 30

rsExecuteTask command 86rsQuery command 39, 84, 86

FlashCopy source 90rsQueryComplete command 38, 69, 86RTO, RPO, NRO

business objectives 3

Ssimplex volume state 13, 31solutions

data migration 65for disaster recovery 5off-site backup 66, 152PiT backup 128PiT copy for application recovery 58, 76PPRC-XD with tapes 65, 124transmission of DB logs 66, 151using static volumes 150

split mirroring 66

state change logic 30static volumes

application availability 16data consistency 16data migration solution 150distance 17long distance solution 14positioning 17transmission of DB logs 151with PPRC-SYNC, description 14write operations 17

supportchannel extenders 71ESS models 9, 71ICKDSF 9, 32

supported distances 70, 128suspend PPRC pair 82suspend XD pair

task creation 114suspended volume state 13SYNC option 27synchronous PPRC

application availability 10data consistency 10distance 11operating characteristics 10positioning 11supported distances 70using static volumes 14write operations 10

Ttapes for data migration 65, 124, 127tasks

consistency group created 117creation with WUI 75, 96define paths 96, 111establishing XD pair 98, 111FlashCopy secondary onto tertiary 108for PiT backup 76freeze 106go-to-SYNC and suspend 102re synchronization of pair 104terminating an XD pair 115withdraw FlashCopy pair 113

terminate PPRC pair 82terminating an XD pair

task creation with the WUI 115tertiary copy 90tiers for DR 4transmission of DB logs 66, 151TSO commands

catch-up 33, 135CESTPAIR 32, 132CESTPAIR parameters’ values 34CESTPATH 51, 132CGROUP freeze 54, 137consistency group definition 51CQUERY 38, 133establishing an XD pair 32, 132

Index 159

FCESTABL 137freeze 137go-to-SYNC 33, 135XD managing example 132

Uunplanned outages 3uses

of PPRC Extended Distance 18, 22, 64–65, 76, 124, 128of PPRC-SYNC with static volumes 17, 150of synchronous PPRC 11

VVM and VSE support 146volume states 12

change logic 30CLI query 84, 86duplex pending XD 19, 30duplex pending XD icons 76

WWeb user interface, see WUIwithdraw FlashCopy pair 82

task creation with the WUI 113within synchronous range catch-up 129write operations

non-synchronous 9, 19, 22, 26–27, 63, 68, 70with PPRC-SYNC 10with PPRC-XD 22with static volumes 17

WUIcatch-up 36, 102consistency group definition 51establishing an XD pair 34freeze 106freeze PPRC consistency group 55go-to-SYNC and suspend 36, 102Information Panel 40options table 37PPRCOPY equivalent functions 146

WUI creation of tasks 96catch-up 102consistency group created 117establishing XD pair 98, 111FlashCopy onto tertiary 108for path definition 96, 111freeze 106go-to-SYNC and suspend 102re synchronization of pair 104suspend XD pair 114

XXD option 27XD pair

catch-up task 102establishing 32, 61, 77, 81establishing task 111

establishing with the CLI 91freeze task 106go-to-SYNC and suspend task 102re synchronization task 104suspend task 114termination task 115


(0.2”spine)0.17”<->

0.473”90<->

249 pages

IBM TotalStorage Enterprise Storage Server PPRC Extended Distance

IBM TotalStorage Enterprise Storage Server PPRC Extended Distance

®

SG24-6568-00 ISBN 0738425303

INTERNATIONAL TECHNICALSUPPORTORGANIZATION

BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE

IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment.

For more information:ibm.com/redbooks

IBM TotalStorageEnterprise Storage ServerPPRC Extended Distance

PPRC Extended Distance description and characteristics

Management and operation of PPRC Extended Distance

Uses and implementations of PPRC Extended Distance

This IBM Redbook describes the characteristics of the new PPRC Extended Distance remote copy function of the IBM TotalStorage Enterprise Storage Server.

In this redbook you will find information on how to operate and manage this new remote copy function of the ESS. It also explains how to use and implement PPRC Extended Distance for remote data copy, off-site backup, remote data migration, and disaster recovery solutions.

PRC Extended Distance is supported on the IBM TotalStorage Enterprise Storage Server models F and newer.

Back cover

IBM TotalStorage Enterprise Storage Server

Documents

Transcript of IBM TotalStorage Enterprise Storage Server