Lotus Domino for AS/400: Problem Determination Guide

Lotus Domino for AS/400:Problem Determination Guide

Robert Boutault, Christina Fasth, Klaus Tebbe, Francine Wiener

International Technical Support Organization

SG24-6051-00

www.redbooks.ibm.com

http://www.redbooks.ibm.com/

International Technical Support Organization SG24-6051-00

Lotus Domino for AS/400:Problem Determination Guide

May 2000

© Copyright International Business Machines Corporation 2000. All rights reserved.Note to U.S Government Users - Documentation related to restricted rights - Use, duplication or disclosure is subject to restrictionsset forth in GSA ADP Schedule Contract with IBM Corp.

First Edition (May 2000)

This edition applies to Releases 4.6 (4.6.3 and later) and 5.0 (5.0 and later) of Domino for AS/400 for use withOS/400 Version 4 Release 2 and later.

Comments may be addressed to:IBM Corporation, International Technical Support OrganizationDept. JLU Building 107-23605 Highway 52NRochester, Minnesota 55901-7829

When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in anyway it believes appropriate without incurring any obligation to you.

Before using this information and the product it supports, be sure to read the general information in Appendix D,“Special notices” on page 289.

Take Note!

Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixThe team that wrote this redbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixComments welcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x

Part 1. General topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1

Chapter 1. Introducing Domino for AS/400 problem determination . . . . . . .31.1 Domino for AS/400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

1.1.1 The Domino server family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31.1.2 Domino for AS/400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31.1.3 AS/400 system architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41.1.4 Domino running on the AS/400 system . . . . . . . . . . . . . . . . . . . . . . . .51.1.5 AS/400 system concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6

1.2 Domino for AS/400 technical support . . . . . . . . . . . . . . . . . . . . . . . . . . . .101.2.1 Need for technical support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .111.2.2 Technical documentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .121.2.3 Additional resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .161.2.4 Support structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

1.3 Problem determination process flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . .161.3.1 Problem determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .171.3.2 Problem source identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .171.3.3 Problem fixing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17

Chapter 2. Domino architecture for problem determination . . . . . . . . . . . .192.1 Common structure of a Domino server . . . . . . . . . . . . . . . . . . . . . . . . . . .19

2.1.1 Kernel design of Domino/Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . .192.1.2 Processing the request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .222.1.3 Controlling resource sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .222.1.4 Database file structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .232.1.5 Shared memory management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .252.1.6 Network address resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .252.1.7 Data representation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26

2.2 New internal structure of a Domino R5 server . . . . . . . . . . . . . . . . . . . . . .272.2.1 Thread pooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .272.2.2 Asynchronous I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .282.2.3 Unified buffer manager (UBM). . . . . . . . . . . . . . . . . . . . . . . . . . . . . .292.2.4 On-disk structure and transaction logging . . . . . . . . . . . . . . . . . . . . .30

Chapter 3. Where to find error information in the system . . . . . . . . . . . . .333.1 AS/400 locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33

3.1.1 Job information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .333.1.2 Spooled files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .383.1.3 Dump files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .393.1.4 Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .393.1.5 Directory synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .413.1.6 Semaphores and shared memory . . . . . . . . . . . . . . . . . . . . . . . . . . .423.1.7 Environment variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45

3.2 Domino locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .463.2.1 Domino console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .473.2.2 Domino log database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .563.2.3 Domino Web server log database and files . . . . . . . . . . . . . . . . . . . .56

© Copyright IBM Corp. 2000 iii

3.2.4 NSD diagnostic tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573.3 When to use a source of information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Chapter 4. How to use the information you gather . . . . . . . . . . . . . . . . . . 614.1 Lotus resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

4.1.1 Lotus Notes Knowledge Base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614.1.2 Other technical resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

4.2 IBM tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 654.2.1 Lotus Domino for AS/400 support information Web page . . . . . . . . . 654.2.2 AS/400 Technical Support Web page. . . . . . . . . . . . . . . . . . . . . . . . 66

4.3 Other tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Chapter 5. How problems can be solved . . . . . . . . . . . . . . . . . . . . . . . . . . 695.1 User errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 695.2 Implementing AS/400 software fixes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

5.2.1 What a PTF is . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 705.2.2 Points to remember about PTFs . . . . . . . . . . . . . . . . . . . . . . . . . . . 705.2.3 More information about PTFs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715.2.4 Looking at installed PTFs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 735.2.5 Ordering PTFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

5.3 Loading and applying PTFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 755.4 Implementing the latest code for Domino for AS/400 . . . . . . . . . . . . . . . . 75

5.4.1 Getting the latest code for Domino for AS/400 . . . . . . . . . . . . . . . . . 765.4.2 Getting the latest fixes package . . . . . . . . . . . . . . . . . . . . . . . . . . . . 775.4.3 Hot fixes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

Chapter 6. Domino administration tools . . . . . . . . . . . . . . . . . . . . . . . . . . 816.1 Domino administration commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

6.1.1 FIXUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 816.1.2 UPDALL -R . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 836.1.3 COMPACT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 866.1.4 When to run FIXUP, UPDALL -R, and COMPACT . . . . . . . . . . . . . . 876.1.5 How long FIXUP, UPDALL, or COMPACT take to run . . . . . . . . . . . 886.1.6 Running Domino administration commands inactive server . . . . . . . 886.1.7 Alternative method for running FIXUP and UPDALL. . . . . . . . . . . . . 91

6.2 Transaction logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 946.2.1 The database instance ID (DBIID) . . . . . . . . . . . . . . . . . . . . . . . . . . 946.2.2 Setting up transaction logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 956.2.3 Changing the transaction logging set up . . . . . . . . . . . . . . . . . . . . . 976.2.4 Disabling transaction logging for a specific database . . . . . . . . . . . . 996.2.5 Fixing corrupted databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1006.2.6 Transaction logging: Troubleshooting . . . . . . . . . . . . . . . . . . . . . . 100

6.3 Troubleshooting tools in Domino Administrator . . . . . . . . . . . . . . . . . . . 1026.4 NOTES.INI file and its parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

6.4.1 Debug parameter versus activity log parameter . . . . . . . . . . . . . . . 1036.4.2 Understanding debug parameters in the NOTES.INI file . . . . . . . . 1046.4.3 Activity log parameters in the NOTES.INI file . . . . . . . . . . . . . . . . . 104

6.5 Editing the NOTES.INI file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1076.5.1 Using the Set Configuration command . . . . . . . . . . . . . . . . . . . . . . 1076.5.2 Using the configuration document . . . . . . . . . . . . . . . . . . . . . . . . . 1076.5.3 Using the Domino Administrator client . . . . . . . . . . . . . . . . . . . . . . 1086.5.4 Using AS/400 Operations Navigator. . . . . . . . . . . . . . . . . . . . . . . . 1126.5.5 Editing the NOTES.INI file from a 5250 display . . . . . . . . . . . . . . . 113

6.6 Tools for monitoring the Domino server . . . . . . . . . . . . . . . . . . . . . . . . . 114

iv Lotus Domino for AS/400: Problem Determination Guide

6.6.1 Starting the Event and Collect tasks . . . . . . . . . . . . . . . . . . . . . . . .116

Chapter 7. Additional considerations for problem determination . . . . . .1177.1 Displaying or editing a stream file from a 5250 display . . . . . . . . . . . . . .1177.2 The DOMINO_SERVERS and LOTUS_SERVERS files . . . . . . . . . . . . .118

7.2.1 When the DOMINO_SERVERS file is corrupted . . . . . . . . . . . . . . .1207.3 The Domino Web Server configuration file (httpd.cnf) . . . . . . . . . . . . . . .1207.4 Ending the Domino for AS/400 server . . . . . . . . . . . . . . . . . . . . . . . . . . .1217.5 IPLing the system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .122

7.5.1 The Power Down System (PWRDWNSYS) command . . . . . . . . . . .122

Part 2. Looking at specific problem areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125

Chapter 8. Server crash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1278.1 What is a server crash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1278.2 A beginner’s checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1278.3 An in-depth analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .129

8.3.1 System name, OS/400, and Domino release . . . . . . . . . . . . . . . . . .1308.3.2 Invocation stack trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1308.3.3 Environment variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1318.3.4 Job log of a current job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1318.3.5 List of all active jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1328.3.6 Status of the current job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1338.3.7 Dump of IPC semaphores and shared memory . . . . . . . . . . . . . . . .1358.3.8 The NOTES.INI file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1368.3.9 Dump of console entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1378.3.10 Successfully processed console entries . . . . . . . . . . . . . . . . . . . .137

8.4 Server crash and automatic restart . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1378.5 Troubleshooting examples and scenarios . . . . . . . . . . . . . . . . . . . . . . . .138

Chapter 9. HTTP server troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . .1419.1 Troubleshooting an HTTP crash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .141

9.1.1 Where to look first . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1419.1.2 Troubleshooting a crash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1419.1.3 What data is needed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1429.1.4 Dynamic thread logging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .145

9.2 HTTP hang . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1479.3 New HTTP Web server commands in R5 . . . . . . . . . . . . . . . . . . . . . . . .149

Chapter 10. Threads and semaphores . . . . . . . . . . . . . . . . . . . . . . . . . . . .15310.1 Gathering call stack information on a secondary thread . . . . . . . . . . . .153

10.1.1 Print screens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15310.1.2 Process dumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .155

10.2 Handling semaphore timeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16210.2.1 What a semaphore and a semaphore timeout are . . . . . . . . . . . . .16210.2.2 Example of a semaphore timeout in Domino . . . . . . . . . . . . . . . . .16210.2.3 Reasons for semaphore timeouts . . . . . . . . . . . . . . . . . . . . . . . . .16310.2.4 Determining if a semaphore timeout has occurred . . . . . . . . . . . . .16310.2.5 Troubleshooting semaphore timeouts . . . . . . . . . . . . . . . . . . . . . .164

Chapter 11. Troubleshooting e-mail delivery. . . . . . . . . . . . . . . . . . . . . . .16911.1 Domino and Internet mail flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .169

11.1.1 What MSF does. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16911.1.2 What POP3 is . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .172

v

11.1.3 SMTP MTA tasks and databases . . . . . . . . . . . . . . . . . . . . . . . . . 17411.1.4 SMTP MTA and MSF: R4.6 AS/400 . . . . . . . . . . . . . . . . . . . . . . . 17511.1.5 R5 server Domino SMTP support. . . . . . . . . . . . . . . . . . . . . . . . . 17711.1.6 R5 server integrated with AS/400 SMTP: MSF. . . . . . . . . . . . . . . 178

11.2 E-mail troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18011.2.1 The starting and ending place . . . . . . . . . . . . . . . . . . . . . . . . . . . 18011.2.2 Internet mail flow on an AS/400 system running Domino . . . . . . . 18111.2.3 MAILROUTER parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18711.2.4 HOSTS file. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18711.2.5 DNS server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18711.2.6 The jobs or tasks that need to be verified that they are active . . . 18811.2.7 Where to look for the mail that doesn’t flow through . . . . . . . . . . . 19311.2.8 Accessing the AS/400 directories. . . . . . . . . . . . . . . . . . . . . . . . . 19411.2.9 What happens when jobs or tasks are not active . . . . . . . . . . . . . 19611.2.10 Settings in the NOTES.INI file to obtain error reporting . . . . . . . 20311.2.11 Helpful NOTES.INI parameters . . . . . . . . . . . . . . . . . . . . . . . . . 205

11.3 Tools and traces for troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . 20511.3.1 Console commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20511.3.2 Using Telnet to test SMTP traffic . . . . . . . . . . . . . . . . . . . . . . . . . 20611.3.3 NETSTAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20811.3.4 NSLOOKUP on the AS/400 system . . . . . . . . . . . . . . . . . . . . . . . 20911.3.5 NotesConnect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21011.3.6 Collecting SMTP Flight Recorders . . . . . . . . . . . . . . . . . . . . . . . . 21011.3.7 Collecting V4R4 flight recorders. . . . . . . . . . . . . . . . . . . . . . . . . . 21111.3.8 AnyMail/MSF dump snap-in . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21711.3.9 Dumping Mail Server Framework (MSF) . . . . . . . . . . . . . . . . . . . 21811.3.10 Communications trace of a line . . . . . . . . . . . . . . . . . . . . . . . . . 21811.3.11 Component journaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

11.4 Tools for e-mail monitoring with Domino . . . . . . . . . . . . . . . . . . . . . . . 22211.4.1 Tracking mail messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22211.4.2 Mail probes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

11.5 Copying spooled files to physical files . . . . . . . . . . . . . . . . . . . . . . . . . 22411.5.1 Example 1: MSF dump and communications trace spooled files . 22411.5.2 Example 2: AnyMail/MSF Dump Snap-In spooled files . . . . . . . . . 226

Chapter 12. Database integration problems. . . . . . . . . . . . . . . . . . . . . . . 22912.1 DB2 Call Level Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229

12.1.1 DB2 UDB for AS/400 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22912.1.2 Call Level Interface (CLI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230

12.2 Tools for database integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23012.2.1 @DB functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23112.2.2 LotusScript Data Object (LS:DO) . . . . . . . . . . . . . . . . . . . . . . . . . 23112.2.3 Lotus Domino Connector APIs . . . . . . . . . . . . . . . . . . . . . . . . . . . 23112.2.4 Domino Enterprise Connections Services (DECS) . . . . . . . . . . . . 23112.2.5 Lotus Enterprise Integrator (LEI) . . . . . . . . . . . . . . . . . . . . . . . . . 231

12.3 Options for database integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23212.3.1 Access to DB2 UDB for AS/400 data from a Domino application . 23212.3.2 Data replication between the AS/400 system and Domino . . . . . . 23312.3.3 Access to an AS/400 application from a Domino application . . . . 236

12.4 Database integration and OS/400 DB2 CLI . . . . . . . . . . . . . . . . . . . . . 23612.4.1 V4R2 changes in CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23712.4.2 Determining if a problem is with CLI or Domino . . . . . . . . . . . . . . 23712.4.3 Locating the job and the job log . . . . . . . . . . . . . . . . . . . . . . . . . . 237

vi Lotus Domino for AS/400: Problem Determination Guide

12.5 Additional information on DB2 CLI problem determination. . . . . . . . . . .24012.5.1 Journaling required at OS/400 V4R4 . . . . . . . . . . . . . . . . . . . . . . .24012.5.2 Tracing CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24312.5.3 Prestart Job QSQSRVR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .245

Chapter 13. Performance troubleshooting . . . . . . . . . . . . . . . . . . . . . . . .24713.1 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .248

13.1.1 Domino performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24813.1.2 Performance considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . .251

13.2 Handling performance issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25213.2.1 AS/400 sizing considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . .25213.2.2 Using the statistics information . . . . . . . . . . . . . . . . . . . . . . . . . . .25413.2.3 High activity for CPU, paging, or disk . . . . . . . . . . . . . . . . . . . . . .25613.2.4 Determining network problems . . . . . . . . . . . . . . . . . . . . . . . . . . .259

Chapter 14. Network troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . .26514.1 NETSTAT command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26514.2 Checking network connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .267

14.2.1 Using PING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26714.2.2 Resolving PING command problems . . . . . . . . . . . . . . . . . . . . . . .269

14.3 NSLOOKUP command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26914.3.1 Client error: ‘Not a known TCP/IP host’ . . . . . . . . . . . . . . . . . . . . .271

14.4 Tracing routes with TRACERT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27114.5 Domino Administrator display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27214.6 Connection trace from a Notes client. . . . . . . . . . . . . . . . . . . . . . . . . . .27314.7 NotesConnect diagnostic aid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .274

Part 3. Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .277

Appendix A. Domino job names and corresponding functions . . . . . . . . 279A.1 SDD/NAB synchronization jobs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281A.2 SMTP/MIME MTA program objects and jobs . . . . . . . . . . . . . . . . . . . . . . . . 282

Appendix B. Domino problem determination tools. . . . . . . . . . . . . . . . . . . 285

Appendix C. Debugging settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287

Appendix D. Special notices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289

Appendix E. Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293E.1 IBM Redbooks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293E.2 IBM Redbooks collections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293E.3 Other resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294E.4 Referenced Web sites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294

How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .297IBM Redbooks fax order form. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298

Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .299

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .305

IBM Redbooks review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .309

vii

viii Lotus Domino for AS/400: Problem Determination Guide

Preface

Since it was released in 1998, Lotus Domino for AS/400 has been installed bynumerous organizations all over the world. Many new releases have been madeavailable, including the major Release 5.0 of the product. Lotus support has beenvery active in supporting customers, and a lot of information has been gatheredon the subject of problem determination. This redbook intends to share some ofthat information. It provides the Domino for AS/400 administrator the tools andmethods to work on Domino for AS/400 problem determination.

In the information technology world, what is true today is often no longer truetomorrow. Release 5.0.2 can’t be compared to Release 4.6b. The AS/400environment is also changing constantly. For this redbook, we gatheredinformation that can be useful when working on problem determination withprevious releases, current releases, and hopefully some other newer releases ofDomino for AS/400.

The team that wrote this redbook

This redbook was produced by a team of specialists from around the worldworking at the International Technical Support Organization Rochester Center.

Robert Boutault is an Advisory AS/400 Specialist at the International TechnicalSupport Organization, Rochester Center. He writes extensively and teaches IBMclasses worldwide on Domino for AS/400. Before joining the ITSO in 1998, heworked in IBM France as a Domino for AS/400 specialist in the e-businessEnterprise Notes and Web Servers team, and as a system engineer in NetworkComputing Solutions and AS/400 since 1988. He graduated fromEAP—European School of Management.

Christina Fasth is an AS/400 System Support Specialist in IBM Sweden, whereshe provides operational and software support to customers. Before joining IBM,she worked 20 years as a customer using the AS/400 and S/3X systems andSystem 3. Her areas of expertise include Domino for AS/400 and Client Access.

Klaus Tebbe is an AS/400 System Support Specialist in IBM Germany. He has10 years of experience in system support and operational support for AS/400 andSystem/390. He has worked with Lotus providing Domino for AS/400 softwaresupport in Europe since it became available on the AS/400 system.

Francine Wiener works as a certified pre-sales Domino for AS/400 IT specialistfor the IBM Server Group in Belgium and Luxembourg. She has more than 20years experience in office and groupware solutions on several platforms and wasa member of the International MidRange and Office Competence Center(MIDOCC). She also participated to the AS/400 pre-announcement residencies inRochester in 1988.

Thanks to the following people for their invaluable contributions to this project:

Terry AckmanCharles CarrollBranden Kolb

© Copyright IBM Corp. 2000 ix

Laurie MillerTerry O’BrienTony PerkinsSteve SparrowIBM Programming Laboratory, Rochester

Stacy FribergMichael GilleyJennifer GinderAmy HoerleWalter ScanlanJohn WeiserIBM/Lotus Domino for AS/400 Support, Rochester

Jennifer BloomLotus North Reading

Ralf StuessLotus Germany

Justine MiddletonInternational Technical Support Organization, Rochester Center

Mike EbbersInternational Technical Support Organization, Poughkeepsie Center

Wilfried BlankertzIBM Germany

Felix ZalcmannIBM Austria

Comments welcome

Your comments are important to us!

We want our Redbooks to be as helpful as possible. Please send us yourcomments about this or other Redbooks in one of the following ways:

• Fax the evaluation form found in “IBM Redbooks review” on page 309 to thefax number shown on the form.

• Use the online evaluation form found at http://www.redbooks.ibm.com/

• Send your comments in an Internet note to [email protected]

x Lotus Domino for AS/400: Problem Determination Guide

http://www.redbooks.ibm.com/contacts.html


Part 1. General topics

This first part of this redbook covers the following topics:

• An introduction to Domino for AS/400 problem determination• An inside look at the Domino for AS/400 architecture for problem

determination• Where to find error information in the system• How to use the information you gather• How to solve Domino for AS/400 problems• Domino administration tools• Additional considerations for problem determination

© Copyright IBM Corp. 2000 1

2 Lotus Domino for AS/400: Problem Determination Guide

Chapter 1. Introducing Domino for AS/400 problem determination

This chapter has three main parts:

• A brief overview of Lotus Domino, Lotus Domino for AS/400 and the AS/400architecture, and how Domino for AS/400 runs on the AS/400 system

• An introduction to Domino for AS/400 technical support

• A description of the problem determination process flow

1.1 Domino for AS/400

The Domino server family is an integrated messaging and Web applicationsoftware platform for growing companies that need to improve customerresponsiveness and streamline business processes.

The only solution built on an open, unified architecture, Domino is trusted by theworld's leading companies to deliver secure communication, collaboration, ande-business applications. Domino R5 servers set a new standard for rich Internetmessaging, ease of administration, integration with backend systems, andreliability.

1.1.1 The Domino server familyThe Domino server family allows companies to quickly and easily start with whatyou need today, whether that is messaging or e-business applications, andextend your Domino infrastructure investment whenever you're ready. TheDomino server family is comprised of three core servers:

• Domino Mail Server: This combines full support for the latest Internet mailstandards with Domino's industry-leading messaging capabilities, all in onemanageable and reliable infrastructure.

• Domino Application Server: This server is an open, secure platformoptimized to deliver collaborative Web applications that integrate yourenterprise systems with rapidly changing business processes.

• Domino Enterprise Server: This server delivers all the functionality of theDomino Mail and Application Servers. It is reinforced with partitioning andclustering for the high availability and reliability required by mission-criticalapplications.

In March 2000, Lotus announced that it has more than 55 million users of LotusNotes.

InformationWeek, in its look at the pace of technology in the 1990s, named LotusNotes one of the most important advances of the decade.

1.1.2 Domino for AS/400Domino for AS/400 was ported from the UNIX version of the Domino product. Itbecame available to customers in February 1998.

As a server, the AS/400 system provides reliability and scalability to enhanceexisting business applications and reach new levels of collaboration andcoordination.


Domino for AS/400 supports multiple (partitioned) servers on the same hardwareto give you the flexibility either to combine servers or to keep your existing serveridentities while consolidating your hardware. Because each partitioned Dominoserver runs in its own AS/400 subsystem, you can use standard OS/400 workmanagement capabilities to allocate system resources and manage theperformance characteristics of each server.

The AS/400 system has a reputation for reliability. For AS/400 owners,performing an IPL (rebooting) is a rare, and usually a scheduled, event.Consolidation of any type reduces system administration, including networkmanagement and system backups.

The AS/400 system is also well known for ease of management. Systemadministration functions are integrated into the AS/400 system and share acommon interface and command language. Different administrative softwarepackages (backup and recovery, network management, and so on) do not need tobe learned. There is no need to ensure that the administrative packages worktogether. The AS/400 system consistently rates among the best (lowest cost) intotal cost-of-ownership studies.

With Domino for AS/400, you can tackle many business problems that requireorganizing unstructured data or managing the flow of information. At the sametime, you can link this work flow and unstructured information to the wealth ofbusiness data in your DB2 UDB for AS/400 databases to enhance the value ofwhat you’ve already built.

1.1.3 AS/400 system architectureThe AS/400 system is designed and built as a total system. This means thatfacilities, such as security, a relational database, and networking capabilities, arefully integrated into the operating system and the hardware microcode. The usercommunicates with all of these functions through a single control language (CL)or by using the system menus and prompts.

The following list provides some important system concepts:

• Layered machine architecture: This architecture, together with the high-levelmachine interface, insulates user applications from hardware characteristics.It enables them to move to new hardware technology at any time withoutchanging their application programs.

• Object-based design: Everything that can be stored or retrieved on thesystem is known as an object. Objects exist to make user interfaces andapplications independent of the internal structure of the machine.

• Single-level storage: Main storage and disk storage appear contiguous. Anobject is saved or restored on the system through a device-independentaddressing mechanism. This means that extra main storage or disk storagecan be added to the system and used without affecting the applicationprograms. A user or a programmer is not concerned with the location of theprogram or files. If they want to use it, they simply name it.

• Hierarchy of microprocessors: In addition to the main system processor, theAS/400 system has a large number of microprocessors. Every microprocessoris called an input/output processor (IOP). This means that requests for writingto or reading data from any I/O device can be delegated to a specializedprocessor while the main system processor executes other applications.


OS/400 is much more than just an operating system for the AS/400 system. It is asingle entity that fully integrates all of the software components needed for mostcommercial computing environments, including the components shown in Figure1:

• Security• Relational database (DB2 for AS/400 UDB)• Network capabilities• Integrated database• Security and authorization• Single-level store• Process management• I/O system• e-business computing• Client/Server computing

Figure 1. AS/400 environment

For more information, refer to the AS/400 System Handbook, GA19-5486. Itintroduces the AS/400 system architecture and provides an overview to allhardware and software capabilities.

1.1.4 Domino running on the AS/400 systemAn AS/400 system running Domino for AS/400 server may have the simplifiedstructure show in Figure 2 on page 6.

Chapter 1. Introducing Domino for AS/400 problem determination 5

Figure 2. Domino running on the AS/400 system

Figure 2 shows the following basic characteristics of a Domino server running onan AS/400 system:

• The server runs in a subsystem (as each partitioned server on an AS/400system).

• The Domino databases are stored in the integrated file system (IFS), whichoffers the same kind of file structure as a UNIX or a PC server. The integratedfile system is described in more detail in 1.1.5.4, “Integrated file system” onpage 8.

• The server is protected by AS/400 security, which means the directory inwhich the databases and the files are stored is protected.

• TCP/IP protocol is used for communications between Notes clients and theDomino server, and between the Domino server and other Domino servers(AS/400 server, Windows NT, UNIX, and so on).

The installation and configuration of Release 5 of the Domino for AS/400 server isdescribed in the “Installing and setting up Domino for AS/400” chapter in LotusDomino for AS/400 R5: Implementation, SG24-5592.

The installation and configuration of Release 4.6 of the Domino for AS/400 serveris described in the “Installing and setting up Domino for AS/400” chapter in LotusDomino for AS/400: Installation, Customization, Administration, SG24-5181.

1.1.5 AS/400 system conceptsThis section provides a brief overview of AS/400 system concepts. It includes anexplanation of AS/400 Operations Navigator, AS/400 control languagecommands, the AS/400 Integrated File System, and security.


1.1.5.1 AS/400 Operations NavigatorAS/400 Operations Navigator provides a Window-like graphical interface forconfiguring, monitoring, and managing the OS/400 environment.

An example of an AS/400 Operations Navigator window is shown in Figure 3.

Figure 3. The AS/400 Operations Navigator interface

In this example, you can see Operations Navigator provides an explorer-like viewof three AS/400 connections (systems), with system AS20 expanded (by clickingthe + (plus) sign) to show a wide range of functions useful for systemadministrators, operators, and end users alike.

In the Servers sub-section of the Network section, there is a Domino subsection,which allows you to manage Domino servers, as illustrated in Figure 3. Thissubsection is available if you install the plug-in for Domino, in AS/400 OperationsNavigator, on your workstation.

AS/400 Operations Navigator includes system navigation, configuration, andplanning capabilities. It incorporates many functions available through an OS/400text-based command interface. These commands (with some restrictions) may beentered from a 5250 keyboard/display device, or within a program. OperationsNavigator also provides an interface to AS/400 tasks that can only be performedusing AS/400 Operations Navigator.

While still incorporating the traditional functions and concepts of the AS/400system, AS/400 Operations Navigator provides a familiar user interface for thoseexperienced in using Windows 95/98 and Windows NT. Operations Navigatorexploits much of the functionality common in the Windows environment (such asdrag and drop, help windows, pull-down menus, and configuration wizards).Therefore, you can leverage users’ Windows skills to interact with the AS/400system. This also minimizes the need for users to learn the AS/400 commandinterface.


AS/400 Operations Navigator provides the ability to view and manage one ormultiple AS/400 systems in a network. Management Central greatly expands thecapabilities of AS/400 Operations Navigator when managing multiple AS/400systems.

More information on AS/400 Operations Navigator is available in ManagingAS/400 V4R4 with Operations Navigator, SG24-5646.

1.1.5.2 Control language commandsControl language (CL) is used by system operators to perform functions, such asmonitoring the system, performing backups, or manipulating objects.

Control language is much more than just a set of operator commands. It is aninterface between the system user and the AS/400 system. This includes allmenus, high-level languages, program product, and system utilities that use CLto perform various system tasks. These tasks include creating files, startingspooling files, changing operational priorities, and initiating work.

Designed for ease-of-use, the consistency and structure of AS/400 CL commandnames make them easy to learn. Control language command names consist ofverb and subject abbreviations.

Domino for AS/400 administrators should become familiar with the basic AS/400CL commands. To learn more about AS/400 CL commands, see the booksAS/400e Series CL Programming Version 4, SC41-5721, and AS/400e Series CLReference, SC41-5722.

This information can also be found at the following AS/400 Web site:http://www.as400.ibm.com/infocenter

1.1.5.3 Object-based architectureThe AS/400 system uses an object-based architecture. There are many types ofobjects and each object has certain unique characteristics. Some examples ofobjects include files, programs, message queues, device descriptions, and userprofiles. OS/400 treats each object type uniquely, even though the controllanguage (CL) commands are consistently simple and easy to use.

The AS/400 operating system (OS/400) and the objects it controls have provenimmune to PC viruses. However, the AS/400 server can be a carrier for a PCvirus introduced from the Internet or e-mail. Because of the structure of theAS/400 operating system, it is unlikely that the operating system or objectscontained within it would be exposed to the virus.

1.1.5.4 Integrated file systemTypically, objects are accessed using a file system. Under the integrated filesystem, there are multiple ways to store and manage objects. The integrated filesystem allows the use of multiple logical file systems on a single AS/400 system.Each logical file system has its own structure and rules, and a common set ofuser and application programming interfaces. The AS/400 logical file systems ofmost interest to Domino users are the root files system and library file system.

Containing a hierarchical directory tree structure, the root file system allows forthe storage of stream files. These files potentially contain long strings of data. PCfiles are an example of stream files. The data directory for the Domino server on


the AS/400 system is a subdirectory in the root file system. For example,/LOTUS/NOTES/DATA or /DOMINO/DOMSVR3 are subdirectories in the root filesystem. Domino databases are stored in the root file system in the same formatas on other Domino platforms. Working on behalf of the clients (Notes clients,Web browsers, and so on), the Domino server is the only “user” that is authorizedto access the IFS Domino databases and files. Access to the Domino databasesis limited to authorized client workstations through the server.

The AS/400 library file system is a directory of object names, types, and storageaddresses that point to the object. DB2 UDB for AS/400 tables (physical andlogical files) and AS/400 programs are the primary objects supported by theAS/400 library file system. Domino for AS/400 provides many ways of accessingDB2 UDB for AS/400 data. It also provides many ways for AS/400 programs toaccess Domino databases. Both Domino for AS/400 and AS/400 programs aremanaged by the integrated file system.

1.1.5.5 SecurityAS/400 security features protect the integrated file system. Security features,such as AS/400 user profiles and group profiles, control access to the system.Special authorization is given to access-specific resources, such as libraries,programs, and files. These AS/400 security concepts are similar to Dominosecurity concepts.

The basic authority levels on the AS/400 system include *READ, *CHANGE,*ALL, *EXCLUDE, and *USE. There are additional special levels of authority forobject management and more advanced administrative tasks.

The AS/400 system and Domino have security concepts that complement eachother. The AS/400 security system protects Domino databases from any accessoutside of Domino. Domino security controls access within Domino for AS/400.

If you plan to use a Domino application to access DB2 UDB for AS/400 data, youmust control access at the AS/400 level and the Domino for AS/400 level.

Although end users may access all sorts of applications, they want the right datafor the application they are using at one time. Users should not be concernedabout where the data is stored. Nor should they have to open a secondapplication to find or update the data they need while using an application.

For Domino applications, this means that the Domino application contains theright data replicated from the AS/400 system (which is important to mobile users),or it can access DB2 UDB for AS/400 in real time. Therefore, you must relate thedifferent applications to the different sources of data.

We expect you to already know how to perform the following tasks:

• Access DB2 UDB for AS/400 databases, IFS files, OS/400 functions, oranother application from an AS/400 application

• Access a Domino database or a different Domino application from yourDomino application, if you are a Domino developer


A simple way to think about integration is to answer to the question: Which is theclient environment? See Table 1.

Table 1. Which is the client environment

A graphical representation is shown in Figure 4.

Figure 4. Integration with enterprise applications

There are many ways to answer these needs. You can choose to develop thesolution by yourself or use a ready made module that saves you a lot of time andeffort.

Refer to Lotus Domino for AS/400: Integration with Enterprise Applications,SG24-5345, for more details about the available integration options andimplementation examples.

1.2 Domino for AS/400 technical support

The need for access to technical support may start when you install a Dominoserver. As explained later in this chapter, each installation is unique. From thebeginning, using the right documentation helps for using the product the way itshould be used. It also helps to avoid known problems. Software support can berequired at any time to provide help in a problem situation.

From To

AS/400 applicationPC or other platform application

Domino for AS/400 application, Domino forAS/400 database

Notes ClientWeb BrowserJava applets and applicationsPOP3, IMAP, LDAP clientsDomino for AS/400 serverOther platform Domino server

DB2 for AS/400 databasesIFS filesOS/400 functions or dataAS/400 application


1.2.1 Need for technical supportEvery customer implementation of a Domino for AS/400 server is unique forseveral reasons, including:

• Every customer need is unique:

– How many Domino servers have been configured and are running on theAS/400 system?

– What are the Domino services used on the server, for example e-mail,calendar, and HTTP server functions?

– If using partitioned servers, are the servers configured to run differentservices?

– What are the Domino applications used on each server?

• Applications created from Lotus Domino templates?

• Homemade applications?

• Applications bought from a third-party software provider?

• Domino solutions, such as Lotus Enterprise Integrator, Domino.Doc,Learning Space, and so on?

• Are those applications integrated with DB2 UDB for AS/400 data andAS/400 applications?

– How many users are using each service or application?

– Does the Domino for AS/400 server integrate a Domino domain where itwill communicate with other Domino servers, some of which may beimplemented on another operating system platform?

– Is there a need to exchange e-mail with the Internet?

• Different Domino for AS/400 releases can be used:

– For Release 4.6, the following intermediate release levels are available:

• Release 4.6b• Release 4.6.2• Release 4.6.3• Release 4.6.4• Release 4.6.5• Release 4.6.6

– For Release 5.0, the following intermediate release levels are available:

• Release 5.0• Release 5.0.1• Release 5.0.2a

• Different Domino for AS/400 language releases can be used:

– International English– Brazilian Portuguese– Chinese (simple Chinese and traditional Chinese)– Danish– Dutch– Finnish– French– German– Italian


– Japanese– Korean– Norwegian– Spanish– Swedish– US English

• Different OS/400 versions can be used:

– The OS/400 version can be Version 4 Release 2 (V4R2) or higher.

– Installed OS/400 program temporary fixes (PTFs) can be different from onesystem to another, depending on the PTF implementation policy and on theother software used on the AS/400 system.

• Installing Domino may require an OS/400 secondary language.

• Other applications can coexist with Domino for AS/400 on the same AS/400system. Some of them may need OS/400 or OS/400 products PTFS, oroperational constraints may result.

This list of reasons is not exhaustive. This just gives a flavor of the diversity ofDomino for AS/400 server implementations. It also gives an idea of whereproblem determination starts.

When a problem occurs, it can be the result of one or several factors:

• A user error, for example a programming error in a Domino application, orusing a function without understanding exactly how and when it should beused, or stopping a Domino server in a wrong way

• A problem inside the Domino for AS/400 software, which needs to be fixed bya correction provided by Lotus

• A problem inside the OS/400, which needs to be fixed by a correction providedby IBM

In any problem situation, it is important to understand the situation. Educationand technical documentation are available to help you understand whether theproduct has been well implemented.

Lotus support is there to give customers assistance on problem determination.Depending on the situation, problem solving can consist of helping to properlyimplement the product. It can also involve providing the fix for a software problem(Domino, OS/400, or other Lotus or IBM product).

1.2.2 Technical documentationTo implement and master a Domino for AS/400 server and its numerousfunctions, a Domino for AS/400 administrator needs to read and understand:

• Lotus Domino cross-platform documentation• Lotus Domino for AS/400 documentation• AS/400 documentation

1.2.2.1 Domino documentationDomino administration is documented in Domino 5 Administration Help.


The Domino 5 Administration Help (help\help5_admin.nsf) database containsdetailed Domino 5 System Administration information. It can be downloadedat: http://notes.net/notesua.nsf

The Domino 5 Administration Help database is also available in hardcopy asthe R5 Domino Administration Doc Pack. It can be purchased through theLotus Store at: http://www.lotus.com/store

1.2.2.2 Domino for AS/400 documentationFor each release of Domino for AS/400, two documents are provided:

• Domino for AS/400 Help

This as400hlp.nsf database contains all AS/400-specific information. It is alsodelivered with the software as the printed book Installing and ManagingDomino for AS/400.

• Domino for AS/400 Release Notes

This readas4.nsf database contains last-minute information specific to theAS/400 platform. A printed version comes with the Domino for AS/400CD-ROM. An updated version may be available on the Web at:http://notes.net/doc

For example, Release Notes documentation for Domino for AS/400 Release5.0.1 contains the following chapters:

– What's New introduces you to Domino for AS/400, tells you about the newfeatures and enhancements in this release, and points you to furtherinformation.

– Things You Need to Know describes requirements, restrictions, and otherinformation that you need to know before installing this release.

– Troubleshooting describes limitations and issues associated with thisrelease of Domino for AS/400.

– Documentation Updates describes corrections and updates to the Dominofor AS/400 documentation.

– Interoperability describes any known restrictions or potentialincompatibilities between R5 and R4 of Domino for AS/400.

– History of Changes contains historical information about enhancementsmade in previous releases of R5 that carry forward into this release.

We recommend that you read the “What's new?” and “Things You Need to Know”sections of the Release Notes.

1.2.2.3 AS/400 documentationAS/400 documentation can be accessed from the IBM AS/400 Web page at:http://www.as400.ibm.com

Click Library and then AS/400 books and manuals. You can navigate in theOnline Library to find online books for different OS/400 versions or releases. Youshould look for the documentation that corresponds to the version/releaseinstalled on the AS/400 system where you run Domino for AS/400 (V4R2 or later).


If the AS/400 platform is for you, you can find the AS/400 Information Center onthe same site. This tool has been available since OS/400 V4R3. It can beaccessed from the same Web page: http://www.as400.ibm.com

Click Library->AS/400 books and manuals->AS/400 Information Center.Select a version and release (for example V4R4), and one of the availablelanguages.

The IBM AS/400 Information Center is the AS/400 professional's source fortechnical information (Figure 5). You can use this site to find information about:

• AS/400 backup, recovery, and availability• Client Access Express, the IBM solution for PC-to-AS/400 connectivity• AS/400 clustering• AS/400 database and file systems• Domino for AS/400 server• Internet and secure networks• Java on the AS/400 system• Logical partitions on the AS/400 system• AS/400 networking• Operations Navigator, the graphical interface for AS/400• AS/400 programming• AS/400 system administration and maintenance• TCP/IP• AS/400 troubleshooting• Other AS/400 information

A CD-ROM containing AS/400 books and manuals can also be purchased. Forexample, a V4R4 AS/400 Softcopy Library CD-ROM is available.

Note


Figure 5. AS/400 Information Center

From the AS/400 Information Center, there are several links to other places onthe Internet you can go for more AS/400 information:

• IBM Redbooks can be accessed at: http://www.redbooks.ibm.com

• The AS/400 Technical Studio, which can help you find online workshops,wizards, other tools, and information on a smaller set of topics, can be foundat: http://www.as400.ibm.com/tstudio

• To find AS/400 news, product information, support services, industryinformation, and more, visit the AS/400 home page at:http://www.as400.ibm.com

• To find information about product fixes, new technology, technical supportservices, and more, visit the AS/400 Worldwide Technical Support home pageat: http://as400service.ibm.com

The AS/400 Information Center is also available on CD-ROM, which contains:

• A subset of AS/400 technical documentation

• Links to the Internet version of the AS/400 Information Center and othertechnical information

• PDF files of Information Center topics

A V4R4 AS/400 Information Center CD-ROM and AS/400 Softcopy LibraryCD-ROM kit are available.


1.2.3 Additional resourcesYou can find more information at the following Web sites:

• The IBM Domino for AS/400 Web page at: http://www.as400.ibm.com/domino• Lotus Web page on Domino for AS/400 at: http://www.lotus.com/dominoas400• Lotus Domino and Notes User Assistance Web site at: http://notes.net/doc

1.2.4 Support structureDomino for AS/400 support is provided by a Lotus support contract. Lotus supportcoordinates with the IBM AS/400 Support Center when appropriate to performproblem source identification and problem resolution.

Typical support contracts are available in conjunction with a Lotus PassportAdvantage agreement. The purchase of a Lotus support contract is highlyrecommended for a successful Domino implementation on any platform.

We recommend that you have both AS/400 support and Lotus support for themost successful Domino implementation on the AS/400 system. For more detailsabout support options, refer to the Domino for AS/400 Buying/Selling Guide at:http://www.as400.ibm.com/domino

At the site, click How to Buy and then Buying and Selling Guide.

Packaged services for planning and installation, as well as migration workshops,custom deployment, and application development services are also availablefrom IBM/Lotus. For more details, see the Domino for AS/400 Buying/SellingGuide at: http://www.as400.ibm.com/domino

At the site, click How to Buy and then Buying and Selling Guide.

A wealth of business partner applications and services for Domino for AS/400 isavailable and growing. Sources of information about available offerings include:

• IBM Global Software Solutions Guide:http://www.ibm.com/software/solutions/isv

• AS/400 PartnerWorld for Developers online Domino solutions catalog:http://www.as400.ibm.com/developer/ebiz/index.html

At this site, click Domino Solutions.

• Lotus/IBM Server Solutions Directory (hardcopy solutions catalog published atthe time of LotuSphere), Lotus Part Number AB1695.

• Lotus business partner catalog (not platform specific): http://www.lotus.com

Click Partners and then Business Partners.

1.3 Problem determination process flow

The following sections describe the classical phases you go through to solve anyproblem, including:

• Problem determination• Problem source identification• Problem fixing


1.3.1 Problem determinationFirst, you discover that something is not going the way it should. This is theproblem determination phase. In this phase, you gather information about theexternal and internal aspects of the problem.

It may be first discovered by the Domino user, for example, no access to theserver, or mail is not delivered. Or it may manifest itself to Domino administratorsand AS/400 administrators or system programmers through messages or othertechnical indications.

1.3.2 Problem source identificationIn the second phase, you study the information you have collected to isolate thesystem components that are indicated in the failure. This is the problem sourceidentification phase. Some problems are easy to limit to a single component. Butsome situations may be more complex, involving relationships between variouscomponents.

At that point, you may contact the Lotus Support Center to receive assistance.You may try, with their help, to recreate the problem, run commands, collect tracedata, perform tests, or take whatever steps are suitable to pinpoint what is notworking.

1.3.3 Problem fixingThe third phase involves fixing the problem, if possible, or at least circumventingit. If the root cause of the problem is well defined and understood, the solution tothe problem is certainly not far away.

Some problems will point to the core code of either the OS/400 operating systemor the Domino server. Only the developers maintaining those systems canprovide a modification that will solve such problems. Others problems will point touser-written or third-party applications or operational procedures.

This methodology is system-independent, when the problems that need to befixed are linked to the logic of the Domino server itself. However, some of thetools and techniques used may differ according to the operating system on whichyou run a Domino server. Therefore, it is important to understand who the mainpeople are that are involved and what tools they use to manage their system andto identify problems.


Chapter 2. Domino architecture for problem determination

Lotus Domino is the leading environment for the development of groupware andmail applications. In addition, Domino provides several Internet integrationfunctions, such as Web application infrastructure and Internet messaging.

Before looking at problem determination tools and techniques, this chapterdiscusses architecture information about the Domino server to help you withproblem determination.

2.1 Common structure of a Domino server

The current release of Lotus Domino is Release 5. Figure 6 shows the functionsthat Domino R5 provides.

Figure 6. Functions of Domino R5

The main component of Domino is Notes services. It is a client/server systembased on its own remote procedure call mechanism, called Notes RemoteProcedure Call (NRPC). Lotus Domino servers and Notes clients run on manyplatforms, from workstations to mainframes, but the basic structure of the Notesservices function is the same. Any platform differences are transparent to theuser and the application programmer. Notes appears the same across platforms.

This chapter describes the general internal structure of Domino, ignoringrelease-specific details. There are several changes in the internal design for R5.We explain those changes in 2.2, “New internal structure of a Domino R5 server”on page 27.

2.1.1 Kernel design of Domino/NotesThe core modules of the Notes software are shared between the client and servercode. The kernel is known as the Notes “C” API, called Notes Object Services

Java VM & Adapter

LDAP

IMAP4 POP3

SMTP MTA

MailClient

NotesClient

Notes Services

Mail

Security

Replication

Object Store

Directory

WebClient

Dynamic HTTP

SSL / X.509

HTML FILES

CGI / PERL

Java (.class)

NNTP

NewsClient

NewsServer

iChatChat

CORBA /IIOP

Domino Entreprise ConnectionServices

ERP

RDBMS

Transactional


(NOS). Some NOS manage Domino databases and files. Others are “operatingsystem”-type services (memory management, time and date services, andnetwork services) that interface to the equivalent services in the host operatingsystem. They hide the particular differences between operating systems fromtheir callers, making it easier to develop portable client and server applications.

This approach is necessary in any architecture that must span existing and futureoperating system choices. Insulation of the groupware application developer fromplatform details is the best way to create a truly open system, where a choice ofsoftware is not constrained by a particular hardware or operating system vendor.Application developers (at the API level or at the Notes user interface) gain threekey isolation layers:

• From the operating system• From network transport• From the physical location of the object store (through a transparent form of

remote procedure call)

Figure 7. Structure of the Notes kernel

All these core modules are available to higher layers of Notes, including APIprograms, in the client and in the server environments, on all platforms supportedby Domino/Notes. Figure 7 shows the Notes kernel, the major constituents of theNotes workstation and server, and the integration options available to APIdevelopers.

Important modules in the Notes kernel are:

• OS: The operating system isolation layer provides platform-independentaccess to memory, shared resources, semaphores, environment information,and so on. Highly optimized implementations of these services for eachindividual operating system are beneath the isolation layer.

NTI (partially exposed to API)

OS/390OS/400

NT

SCO

Mac

Solaris

AIX HP/UX

W indows

SECsecurity

FTfulltext indx

NIFindexing

NSFstorage

ComputeKernel Functions

"RPC" (internal)

OSisolation

IPX/SPX NetBIOS TCP/IP A/Talk X.PC etc

addin server tasks

Chronos

Indexer

MAIL

ReplicatorNotes/FX

Domino Server

Notes Workstation

DESK

VIEW EDIT

Import/Export

NEM

SERVER

yourownUI

API Application

Hook

Only protocol supported byDomino on the AS/400 system


• SEC: The Security module provides access to user information, certificates,and encryption keys, based on the BSAFE security package from RSASecurity Inc.

• NSF: The Notes Storage Facility is a subsystem that manages the Dominodatabase, allowing its users to create, open and delete databases; create,open and delete documents; and store and retrieve information. Part of thisfunction includes On-Disk Structure (ODS) management, which ensuresportability of the .NSF file format across platforms and on the network. All thepublished NSF interfaces are independent of the database location; a remoteprocedure call system transparently redirects requests to the local disk or tothe appropriate server. This RPC layer is also used by other kernel modules,such as NIF (the indexer, which is used to retrieve a pre-built index from theappropriate server).

• NIF: The Notes Index Facility manages indexes of Notes documents intoViews. Views define the selection of particular documents and columnscontaining information from those documents (or calculated values based onthe documents). Columns can be sorted or categorized (where similar valuesappear under one heading), and they can use the hierarchy of main andresponse documents in a database. These collections of documents areindexed in a B-tree structure, for presentation to the user as a Notes view. NIFis responsible for maintaining and using the indexes, and adding andremoving information incrementally as documents are modified.

• FT: The Full Text Index facility provides content-based retrieval and weightingof documents with Boolean logic searches through the full text of anydocument.

• NTI: The Network Transport Layer provides a single interface to drivers formany networking protocols. It has the ability to initiate and receive phone callsor create LAN-protocol sessions for communication over ports defined by“connection” documents in the Name and Address Book.

• COMPUTE: This module is responsible for performing calculations, using theNotes formula language. Notes API programs can ask COMPUTE to createand evaluate formulae and also implement custom @ functions that are calledby the COMPUTE module.

• NAME: The user directory service, while not strictly a part of the kernel code,has some direct APIs that give access to the Lotus Notes Name and AddressBook. This is a special Domino database that contains user names and e-mailaddresses. User identification uses an X.500-based naming model andconforms to X.509 certificate and authentication standards.

The Domino/Notes server consists of one core server program, which managesthe server's other processes, threads, and users' connections with the server.The server functionality is largely implemented with “add-in” modules. Some area necessary part of any server system, and others are optional or developed bythird parties. Important add-ins include the database replicator (which schedulesand connects to other servers or workstations to replicate databases), the indexer(which keeps indexes up-to-date for immediate access by workstations), the mailrouter (which directs mail between mailboxes and between servers), and Chronos(which schedules “agents” to perform background tasks in Domino databases atthe application designer's request).

Chapter 2. Domino architecture for problem determination 21

2.1.2 Processing the requestEach request from the Notes client is processed by the thread owned by theserver process. A thread is a way of sharing the CPU within an application. Athread is a piece of code that owns a stack, registers, and its priority. It shareseverything else (code, variables, and so on) with all the other threads in theprocess. Figure 8 shows the relation between the user sessions and the threadsrunning on the server processes.

Note: Figure 8 only applies to releases of Domino before R5. The new processstructure in Domino R5 is shown in Figure 12 on page 28.

Figure 8. Server processes and user sessions (before Domino R5)

Before R5, there was a restriction on the number of threads per process. Forexample, R4 was restricted to 255 threads per server process on the AS/400system. When the number of concurrent requests exceeded this limit, anotherserver process was executed and created the next session. Starting with Release5, a new concept of thread pooling extended the limit of sessions per serverprocess, reducing the number of server processes. Thread pooling is discussedin 2.2.1, “Thread pooling” on page 27.

2.1.3 Controlling resource sharingOn a Domino server, each thread in the server process shares system resources,such as Domino databases and stream files. For concurrent processing, theDomino server uses semaphores to serialize access to the resources. A generalusage of the semaphore is shown in Figure 9.

serverprocess

threadNotes Client

thread

thread

....

.....

"show tasks debug" --> display Process ID and Thread ID

The number ofprocessesdepends on thenumberof sessions.


Figure 9. Usage of semaphores

A semaphore is a software switch or flag that is used to synchronize the execution ofvarious tasks to ensure that one process has completed before another begins. Athread that requires the resource locks an appropriate semaphore to prevent othertasks from using that resource until it is finished. If another thread tries to use thesame resource, it has to wait for the owning thread to finish and unlock thesemaphore. In Figure 9, this operation may happen in the following way:

1. Thread A locks the semaphore “0” to access the resources.

2. Thread B tries to lock the same semaphore but cannot. Then, Thread B waitsfor that semaphore to become available.

3. Thread A finishes its use of the resource and frees the associated semaphore.

4. Thread B tries again and succeeds this time.

In the previous flow, we mention only one semaphore. But in the actualprocessing environment, there are many semaphores used for controlling accessto shared resources. If the thread tries to lock multiple semaphores at one time, itmay cause a deadlock situation. Looking again at Figure 9, suppose that ThreadB had already locked semaphore “1” (pointed to by A) and entered a wait state. IfThread A tries to lock the semaphore “1” before releasing the previoussemaphore “0” (pointed to by B), it will also wait and both threads will not run anymore. If this happens on a Domino server, it will cause a server error.

On the AS/400 system, Domino semaphores are implemented with mutexes andcondition variables. Mutexes provide synchronization and access to conditionvariables. Condition variables provide the waiting semantics.

2.1.4 Database file structureDomino databases are also known as “Domino applications” because theycontain more than just data documents. They are complete, self-containedapplications having code, schema, indexes, security, and data all bundled intoone file. They are the foundation of the Notes architecture and can reside eitheron clients or servers. They have unique structures designed to supportsimultaneous online access by multiple users. They also supportre-synchronization of the contents of two copies (replicas) of the same database.

Kernel

Thread A Thread BSemaphore

Semaphore

1 2

34

AB

XO

X

0

1


This makes it easy for an individual to “take an application offline” for awhile andthen merge offline edits back into a common, shared database.

A Domino database is a container for documents, form designs, view definitions,and all the other classes of notes. The Notes flexible structures and features aretailored to the need to transport and identify them, such as unique identifiers,last-modified timestamp, summary information, and so on. The containerdatabase also reflects its function, such as the need to quickly retrieveappropriate information, the need for extreme reliability, and the need to supportreplication. Figure 10 shows the structure of this database.

Figure 10. Format of a Domino database file

The Domino database (a file on disk, usually with extension .NSF) begins withheader information and an allocation map. The important parts of the headerinclude:

• Database ID: Uniquely identifies this file

• Replica ID: Uniquely identifies all replica copies of this file on all servers

• Creation timestamp

• Last Modification timestamp: Identifies the last time that any of thedatabase's contents were modified

• Title and a category: For easy identification

• Design Class: Indicates that the design elements should be inherited from acentral template database and maintained in synchronization with thattemplate

The replicator task and others use this information to efficiently decide whether adatabase should be replicated and which database replicas exist on anyparticular server. The database contents can be scanned extremely quickly for

Access Control List

Collections andother non-replicating objects(reconstructed on each server)

Allocation map

Form note

Header

View definition note

Shared field definition note

Document

Document

Icon

Macro definition note

... other notes ...


the notes that have been added or modified since a certain time. This again isoptimized for the replicator.

The rest of the database contains notes and other objects, summary buffersstored for rapid access, and non-summary items in another area of the file. Thereis another internal abstraction. Notes themselves are a specialized class ofobjects. Other object classes include file attachments, OLE packages, andpacked lists of documents (unread lists, for example). Every object is identified bya Record Relocation Vector (RRV), which is a file-position pointer. A NOTEID issimply the note's RRV.

Some objects, such as the collections that constitute an index of the database,are never replicated (since they must be built for each local copy of thedatabase).

2.1.5 Shared memory managementLotus uses shared memory segments to allocate portions of the segmentdesignated for Notes use to pools and semaphores. This segment is referred toas the memory manager, which manages all shared memory segments allocatedto Notes.

After the server starts, and the initial segment for the memory manager isallocated, the server needs some memory space with which it can work. At thistime, Domino creates its first working area memory segment. Space within thissegment starts getting taken up by NSF buffering space to contain information forthe Domino Directory (formerly called the Name and Address Book), log files, andmail.box (these are files that are always open when a Notes or Domino server isrunning).

Next, the Domino server receives a request to open a database. This requires theuse of a semaphore and takes some space in the segment as well. Now an indexhas to be read, and Domino needs space in which to put it, so the serverallocates space for the NIF pool to store the index information. The semaphoresare regarded as “traffic cops” for the shared memory segments. They control theallocated space and the sharing of semaphores. Therefore, this scenario occursover and over. When it needs more space, Domino creates a new segment of thespecified size (whether it is set using the algorithm or the spool variable). Sharedmemory segments are taken from the AS/400 single-level storage.

2.1.6 Network address resolutionIn a Domino environment, the name resolution service is basically provided bythe Notes Name Services (NNS) used with a specific remote procedure callmechanism, called Notes Remote Procedure Call (NRPC).

All Domino servers and Notes users have a hierarchical name based on theX.500 naming standard. This naming scheme uses a tree structure that reflectsthe structure of the company using Notes. For example, the following exampleshows the hierarchical name of Joe Smith who works in the accountingdepartment of the XYZ organization:

Joe Smith/Accounting/XYZ


This name is broken down as follows:

Common Name (CN) = Joe SmithOrganizational Unit (OU) = AccountingOrganization (O) = XYZ

For the server, the server name is placed in the Common Name field. To accessthe server, the client looks up the Domino directory to get the server document,which includes several network details about the server.

When using TCP/IP as a network, it is important to manage the relationshipbetween the server common name and the TCP host name or IP address. In theTCP/IP environment, the TCP host name is managed by the Domain NameServer or stored in the local hosts file. The host name is translated to an IPaddress to make a TCP/IP connection. Using TCP/IP and the Domino nativename services, a Domino server or Notes client processes the name-to-addressresolving, as shown in Figure 11.

Figure 11. Domino server/host name resolve process using TCP/IP

2.1.7 Data representationIn a Domino environment, there is a special code set used for representing thedata in the database. It is the Lotus Multi-Byte Character Set (LMBCS). LMBCS is

Initiating system asking fortarget systemsaddress

(Get Host by Name)

Have we accessedthe system before

Is a connection docpresent

Is the targetin Hosts file

Is the targetin DNS

Host name in theNet address field

Retrieve NotesName ServiceCache entry

Retrieve theHost name entry

IP address in theNet address field

Pass serverscommon name

entry as being theHost name

Retrieve thenumeric IP

address entry

Does the targetrespond

Error: Not a known TCP/IPHost

Pass value to TCP/IP stack

Error: Server notresponding

Connection established

Yes

Yes

Yes

No

No

No

No


based on the US ASCII code set, with some modifications. Table 2 shows thecode assignment of LMBCS.

Table 2. Code assignment of LMBCS

2.2 New internal structure of a Domino R5 server

Domino R5 is the latest generation of Lotus Internet products. The Domino R5server builds on its already strong foundation for messaging and Webapplications. To improve functionality of availability, reliability, and performance,the Domino R5 server introduces several new structures. How they implementthese enhancements depends on the server platform. Domino for AS/400 runs onan OS/400 UNIX System Services environment. This section describes the newdesign based on the implementation in a UNIX environment.

2.2.1 Thread poolingAs described in 2.1.2, “Processing the request” on page 22, Domino R4 serversuse one thread per connection. When a client comes in with a request, therequest is received and a thread of execution is created. The thread of executionthen handles all requests made against that connection. This works well for 100to 500 users, but with thousands of users, there are thousands of threadsexecuting on a system, trying to get to the same shared resources at the sametime. A lot of time and resources are spent managing contention between thesethreads.

Domino R5 uses a thread pool model, in which each server thread can supportmany connections. Figure 12 on page 28 shows the thread pool model and howthe server process uses it.

First byte Secondbyte

Third byte Representation

0 None None Terminate

1 - 23 Not 0 Not 0(if required)

Pre-defined multibyte

24 - 31 Not 0 None Application defined character

32 - 127 None None US ASCII

128 - 255 None None Code from products optimization group


Figure 12. Thread pool and server process

When a user connects to the server, a client session is created with a uniquesocket descriptor, and a virtual thread ID is assigned. When a client performs I/O,a physical thread is associated with the virtual thread and does the associatedwork on behalf of the client’s session. After the work is completed, the physicalthread is returned to the pool of threads. It is no longer associated with the virtualthread and is available to service another request. For this reusable threadmodel, the server can keep fewer threads than the maximum number ofconcurrent user connections.

By having fewer threads, the contention for resources is reduced significantly.Also, there is a reduction in the amount of memory consumed per connection.The thread pool model is a much faster performer on the same system. There is agreater reduction in the amount of resources consumed, because you don't havea threads storage issue. There are more CPU cycles available because youaren't managing contention.

2.2.2 Asynchronous I/OIn the thread pool model, it is necessary that the operating system supports bothasynchronous I/O and a fast event notification method. I/O event notificationsays, in effect, “I know when this read or write operation is completed, and I cannow go do the next task.” The way it works is that all connections have anasynchronous read posted and each of the threads is waiting on an event fromthe asynchronous notification queue. When a client on the other end of aconnection makes a request, the operating system creates an event that isplaced on the asynchronous notification queue. The request is then handled bythe next available thread. After the request is satisfied, the whole process beginsagain.

Notes Client

NetworkStack

Serverprocess

.....The # of socketsis also the# of sessions

Each thread is attachedto several sessions

thread

thread

thread

thread

thread.....

thread

socket

socket

socket

socket

Only oneserver process


On all UNIX platforms, the POSIX asynchronous I/O is used for all network readsand writes. This allows Domino to post reads and writes, and then come back andtake control with the thread. While this works well on all UNIX platforms, theproblem is to find out when these I/Os completed (the notification to the O/S thatthe pending async I/O had completed).

The normal method of determining that a POSIX I/O has completed is to examinethe per I/O status structure. The problem with this is that if you have 10,000connections or users, it would take a long time to look at each of the 10,000 ormore containers to find out (for every I/O operation) which one had activity.

This doesn’t make sense. Domino R5 general architecture uses the signalcontext, passed in from a blocking call, to determine which I/O has completed.The server issues the sigio for asynchronous operation and then waits for thesignal using a sigwait.

Signals are not used on the AS/400 system. Instead, Domino for AS/400 uses anI/O Completion port where multiple worker threads can wait for the completedasynchronous I/O request, either startRecv() or startSend().

On the AS/400 system, Domino for AS/400 QMU 5.0.1.02 introducesasynchronous I/O completion ports (IOCP).

The following AS/400 PTFs are required for enabling this enhancement:

• V4R3:

– 5769999 MF22639– 5769SS1 SF57214– 5769SS1 SF57215– 5769SS1 SF57216

• V4R4:

– 5769999 MF23076– 5769SS1 SF58438

The new server design enhancement allows Domino for AS/400 to use fewerthreads to support the users in the SERVER job. Instead of one thread per user, asmall group of threads handles the work for all the users. Tests have been donewith up to 10,000 users in a single partition, which required only 200 threads tobe active.

2.2.3 Unified buffer manager (UBM)Unified buffer manager (UBM) is a shared memory cache for most disk residentdata in Domino databases. It is also a memory cache for temporary files likeexternal sorting temporary files or containers. Cached data is sharedsystem-wide. UBM has fair replacement policies and high concurrency in alloperations.

In R4, there are many types of caches for each database, but they do not havefunctionality such as global sharing, global fair replacement, fair competition formemory among database components, uniform “control” over I/O, orasynchronous clearing and pre-fetching.


UBM combines elements of many buffer managers, for example the clockreplacement algorithm similar to DB2. It also has some unique requirements:

• Variable-size buffers• Content-specific I/O and management methods• Page “groups”• Retrofit into a code base that was not designed to use a buffer manager

A buffer manager is a cache for database and temp file data. It saves doing I/Oeach time data is modified or accessed. A buffer manager presumes a paginatedstructure of data. Pages have embedded metadata that improves integrity,enables recovery, and increases performance. It is important that you distinguisha database buffer manager from the file system buffer cache.

A buffer manager manages a pool of memory called the buffer pool (NSF BufferPool). A buffer manager maintains a dynamic mapping of database “pages” ontomemory “buffers”. A buffer is the memory area used to cache the page when it is“buffered”. A page is a contiguous section of a database, including the datastored in the page and any persistent page metadata stored in the page.

Buffered pages can be shared by any running process and thread. Multiplemodification can be absorbed in a buffered page before it is written to disk.Multiple accesses (reads) can be done against a buffered page once it is readfrom disk. Reads and writes can be mixed for a single buffered page.

2.2.4 On-disk structure and transaction loggingIn Domino R5, there is a new on-disk structure (ODS) for the database. This newformat optimizes I/O and increases reliability. It has better utilization of disk I/O byusing page-oriented handling of data. Writing to disk takes place less frequentlybut more efficiently, which means the I/O is dramatically reduced. The maximumdatabase size limit has been increased to more than 64 GB (this was themaximum tested, not an architectural limit) from 4 GB in R4. You must be runningOS/400 V4R4 or later to enable this support.

When migrating the database files to the new ODS format, you can select theoption of transaction logging. Transaction logging is an industry standardtechnique for ensuring reliable data storage. In the event of failure, the serverrestart time is much faster, and no fix up is required. Significantly improveduptime is possible with R5 logging. Domino can be restarted in a matter ofminutes even with many very large databases. A simplified view of the loggingprocess is shown in Figure 13.


Figure 13. Transaction logging

Let us say that a user writes a document to a database. When the new documentis saved, Domino writes it to a memory buffer 1. When the memory buffer isneeded for other data, the contents are posted to the transaction log 2. At sometime later, determined by the system, the logged event is written to the actualDomino database 3.

When you enable transaction logging, Domino creates a unique databaseinstance ID (DBIID) for every database on that server. This enables Domino to

1

2

3

Ideally, the logs should be on a mirrored disk, since a failure of the devicecontaining the logs can mean losing a lot of transactions which have not beenwritten out to the databases. On the AS/400 system, this can be done using anauxiliary storage pool (ASP).

Important

An ASP is a software definition of a group of disk units on your AS/400 system.This means that an ASP does not necessarily correspond to the physicalarrangement of disks. Conceptually, each ASP on your system is a separatepool of disk units for single-level storage. The system spreads data across thedisk units within an ASP. If a disk failure occurs, you need to recover only thedata in the ASP that contained the failed unit. There are two types of ASPs:

• System auxiliary storage pool• User auxiliary storage pools

For more information, refer to the “Working with Auxiliary Storage Pools”chapter in OS/400 Backup and Recovery, SC41-5304.

Auxiliary storage pool (ASP)


record information about transactions made to each database. The server thencreates the transaction log files and a control file. Now, every time a database isupdated, the server logs the transaction to the sequential log file, and a separatetask actually updates the database file. The transaction is marked complete whenthe log operation is completed. The server, therefore, does not have to wait untilthe transaction is actually written to the database.

Transaction logging not only improves database reliability and integrity but, forsome platforms, it also improves performance (no formal test has been publishedfor the AS/400 system). Also, backups of servers can be more easily managed.

More information on transaction logging can be found in 6.2, “Transactionlogging” on page 94.


Chapter 3. Where to find error information in the system

This chapter shows the sources of information that Domino for AS/400 provideswhen an error occurs. This information is helpful in the problem determinationphase (described in 1.3.1, “Problem determination” on page 17).

The following information is described in this chapter:

• AS/400 locations:

– Job information– spooled files– Dump files– Messages– Directory synchronization– Semaphores– Environment variables

• Domino locations

– Domino console– Domino log database– Domino Web server log database and files– NSD diagnostic tool

Refer to Table 3 on page 58 at the end of this chapter, which helps you identifywhen a certain source of information should be used.

3.1 AS/400 locations

From an AS/400 point of view, we must look at the following locations:

• Job information (active jobs, job logs, spooled files)• Dump files• Messages• Semaphores• Other

– Call stack information on a secondary thread– Collecting a VLIC trace on an active job

3.1.1 Job informationTo look for jobs running in the Domino server subsystem, use the AS/400 Workwith Active Jobs (WRKACTJOB) command, or the Work with Domino Servers(WRKDOMSVR) command. Then, enter option 9 (Work server jobs) and press Enter.

The Work with Active Jobs display is shown in Figure 14 on page 34.


Figure 14. Work with Active Jobs display

A complete listing of all jobs created by the Domino for AS/400 product can befound in Appendix A, “Domino job names and corresponding functions” on page279.

To toggle between the different views in the screen, press F11. This displays theelapsed data, thread data, and again the status view of the Work with Active Jobsdisplay.

3.1.1.1 Displaying a job logTo access a single job log, enter a 5 (Work with) next to the requested job. Thenselect option 10 (Display job log, if active or on job queue), and press F10(Display detailed messages). Look for error messages within the job log. Anexample is shown in Figure 15.

Work with Active Jobs AS2502/17/00 15:17:49

CPU %: 3.1 Elapsed time: 00:00:43 Active jobs: 257

Type options, press Enter.2=Change 3=Hold 4=End 5=Work with 6=Release 7=Display message8=Work with spooled files 13=Disconnect ...

Opt Subsystem/Job User Type CPU % Function StatusDOMINO01 QSYS SBS .0 DEQWADMINP QNOTES BCI .1 SELWAMGR QNOTES BCI .0 SELWAMGR QNOTES BCI .0 CNDWBILLING QNOTES BCI .0 SELWCALCONN QNOTES BCI .0 CNDWDECS QNOTES BCI .0 SELWEVENT QNOTES BCI .2 SELWHTTP QNOTES BCI .0 SELW

More...Parameters or command===>F3=Exit F5=Refresh F7=Find F10=Restart statisticsF11=Display elapsed data F12=Cancel F23=More options F24=More keys

A job log is a temporary object that exists while the corresponding job is active.It may be written to a spooled file when the job ends, depending on what isspecified in the job description used by the job.

Note


Figure 15. Display All Messages display

If you want to save all current job information for later use, or if you need to sendit to a support person, you can write it into a spooled file. The contents of thespooled file can then be printed or copied to a file using the Copy Spooled File(CPYSPLF) command. Refer to 11.5, “Copying spooled files to physical files” onpage 224, for an example.

Use the Display Job Log (DSPJOBLOG) command. Then, examine the jobinformation shown on the Display Job Log screen that appears. First, you need todetermine the jobs name, user, and number.

For the agent manager job shown in Figure 14, the command would be:

DSPJOBLOG JOB(147506/QNOTES/AMGR) OUTPUT (*PRINT)

You can access the spooled file by using the WRKSPLF QNOTES command. For moreinformation about the Work with Spooled Files (WRKSPLF) command, refer to3.1.2, “Spooled files” on page 38.

Display All MessagesSystem: AS25

Job . . : AMGR User . . : QNOTES Number . . . : 147506

Job attributes not set from locale.Job 147506/QNOTES/AMGR started on 02/17/00 at 14:36:18 in subsystemDOMINO01 in QUSRNOTES. Job entered system on 02/17/00 at 14:36:18.

Public write authority on "/DIR5345/Java".Job 143726/QUSER/QSQSRVR used for SQL server mode processing.Job 143986/QUSER/QSQSRVR used for SQL server mode processing.

BottomPress Enter to continue.

F3=Exit F5=Refresh F12=Cancel F17=Top F18=Bottom

There is an easier way to print the job log of an active job. This method can beproposed by a service representative who has to guide you on the phone. Theprocedure consists of the following steps:

1. In the WRKACTJOB display, enter option 5.

2. Do not press Enter. Press F4 instead.

3. In the prompt, enter *PRINT (parameter Output) and *JOBLOG (parameterOption).

This way, you don’t need to write down the job name and number.

Note

Chapter 3. Where to find error information in the system 35

3.1.1.2 Displaying the job call stackOn the Work with Active Jobs display, press F23 (More options) to access option10 displayed under Type options, press Enter. Enter option 10 (Display call stack)to display the initial thread’s call stack. This shows the programs and proceduresthat are running within the initial thread of the job, which in fact represent the job.See Figure 16 for an example.

Figure 16. Display Call Stack display

Within this display, you can switch between different views of information bypressing F11. The information is displayed in the following order:

• The number and name of the activation group within which the program orprocedure is running.

• The ILE module, ILE program, and whether a control boundary is active for aparticular program or procedure.

• The high-level language statement identifier. This field is blank if the programor procedure has no debugging tables.

Keep in mind that this is the call stack information for the job at the time theWRKACTJOB command was issued.

3.1.1.3 Displaying job threadsIn computer programming, a thread is placeholder information associated with asingle use of a program that can handle multiple concurrent users. From theprogram's point-of-view, a thread is the information needed to serve oneindividual user or a particular service request. If multiple users are using theprogram or concurrent requests from other programs occur, a thread is createdand maintained for each of them. The thread allows a program to know whichuser is being served as the program alternately gets re-entered on behalf ofdifferent users. One way thread information is kept is by storing it in a specialdata area and putting the address of that data area in a register. The operating

Display Call StackSystem: AS25

Job: AMGR User: QNOTES Number: 147506

Thread: 00000061

ProgramRqs orLvl Procedure Library Statement Instruction

_CXX_PEP QSYS 0000000139main QSYS 0000000127

< SpawnChild QSYS 0000000372< d__FPviPPc QSYS 0000000501_C_pep QNOTESmain QNOTES 0000000001notes_main QNOTES 0000000014

< lNotesMain QNOTES 0000000563NotesMain QNOTES 0000000004AddInMain QNOTES 0000001021

< cutiveMain QNOTES 0000000768More...

F3=Exit F10=Update stack F11=Display activation group F12=CancelF17=Top F18=Bottom F22=Display entire name


system always saves the contents of the register when the program is interruptedand restores it when it gives the program control again.

A mutex (mutual exclusion object) is a program object that is created so thatmultiple program threads can take turns sharing the same resource, such as theaccess to a file. Typically, when a program is started, it creates a mutex for agiven resource at the beginning by requesting it from the system and the systemreturns a unique name or ID for it. After that, any thread needing the resourcemust use the mutex to lock the resource from other threads while it is using theresource. If the mutex is already locked, a thread needing the resource istypically queued by the system and then given control when the mutex becomesunlocked (once more, the mutex is locked during the new thread's use of theresource).

On the Work with Active Jobs display, first press F23 (More options) to accessoption 12. Enter option 12 (Work with threads) to work with the threads for the job(Figure 17).

Figure 17. Work with Threads display

For each thread, different types of information can be displayed, depending onthe option chosen:

• Option 5 (Display attributes) displays attributes such as Thread, User, Currentlibrary, and Thread Type.

• Option 10 (Display call stack) displays the call stack for the job’s initial thread(see 3.1.1.2, “Displaying the job call stack” on page 36), which is normally thefirst thread in the display. The additional threads being shown are secondarythreads of the initial thread of this job.

• Option 11 (Work with thread locks) shows all external object locks held by thedisplayed thread, including both held locks and locks being waited for.

• Option 14 (Work with thread mutexes) shows all mutexes associated with aspecific thread, including mutexes that are held and the mutex or which the

Work with ThreadsSystem: AS25

Job: AMGR User: QNOTES Number: 147506

Type options, press Enter.3=Hold 4=End 5=Display attributes 6=Release 10=Display call stack11=Work with thread locks 14=Work with thread mutexes

Total Aux RunOpt Thread Status CPU I/O Priority

00000061 CNDW 2.224 555 2600000065 JVAW .000 0 2000000064 JVAW .035 1 20

BottomF3=Exit F5=Refresh F9=Command line F12=Cancel F17=Top F18=Bottom


thread may be waiting. When displaying mutexes for a job, only mutexesassociated with the initial thread are displayed. The Work with threads orDisplay threads option should be used to obtain mutex information aboutsecondary threads within a job.

3.1.2 Spooled filesIf a job on the AS/400 system has ended because of an error, the job log is savedin a spooled file based on the user profile. For the Domino servers, all spooledfiles are found under user profile QNOTES.

To see the job logs of all the Domino server jobs that have ended abnormally,type WRKSPLF QNOTES on an AS/400 command line, and press Enter. This isbecause all Domino server tasks have a job log created by user QNOTES. Anexample of the output from the Work with Spooled Files command is shown inFigure 18.

Figure 18. Work with All Spooled Files of user QNOTES

Notice that, in Figure 18, the User Data column is the name of the Domino task.As a general rule in problem determination, look at the job logs that are morethan one page. When you press F11, you see the date and time when the job logwas spooled. This may help you identify the failing job, if there are many job logson your system.

To see the contents of a spooled file, enter 5 in the options field of the file to beselected, and press Enter.

Note that it is best to look at the spooled files while in the Assistance mode ofIntermediate. It provides the maximum available information to be displayed by thatAS/400 command.

To access the Intermediate Assistance mode, enter the WRKSPLF command. Whenit displays the list of spooled files, press PF21. This displays an operationalassistance panel, which lets you set your assistance level to Intermediate. If you

Work with All Spooled Files

Type options, press Enter.1=Send 2=Change 3=Hold 4=Delete 5=Display 6=Release 7=Messages8=Attributes 9=Work with printing status

Device or Total CurOpt File User Queue User Data Sts Pages Page Copy

QPJOBLOG QNOTES QEZJOBLOG ROUTER RDY 2 1QPJOBLOG QNOTES QEZJOBLOG SCHED RDY 2 1QPJOBLOG QNOTES QEZJOBLOG BILLING RDY 2 1QPJOBLOG QNOTES QEZJOBLOG EVENT RDY 2 1QPJOBLOG QNOTES QEZJOBLOG DECS RDY 3 1QPJOBLOG QNOTES QEZJOBLOG AMGR RDY 32 1QPJOBLOG QNOTES QEZJOBLOG ADMINP RDY 2 1QPJOBLOG QNOTES QEZJOBLOG UPDATE RDY 2 1QPJOBLOG QNOTES QEZJOBLOG QRWTSRVR RDY 3 1

More...Parameters for options 1, 2, 3 or command===>F3=Exit F10=View 4 F11=View 2 F12=Cancel F22=Printers F24=More keys


do not set the proper assistance level, the Work with Spooled Files (WRKSPLF)command will not display the job logs that are spooled. Figure 18 shows anexample of the output from the WRKSPLF command with the assistance modeset to intermediate. This level should be used at any time you are searching forerror information in the system.

3.1.3 Dump filesA dump of a job is needed in different situations:

• When a job looks hung, for example the HTTP task• When a job loops

To generate a dump for further analysis. Complete the following process:

1. Record the Job, User and Job number by typing the WRKACTJOB command, andpress Enter. Press F11 two times and write down the information.

2. Type the Start Service Job (STRSRVJOB) command on the command line, andpress F4. Enter the Job name, User, and Number, and press Enter.

3. Enter the Dump Job (DMPJOB) command, and press Enter.

For example, the command to dump only one job thread information if theselected thread ID to include is 00000001 would be:

DMPJOB PGM(*NONE) JOBARA(*NONE) JOBTHD(*YES) SLTTHD(00000001)

For more information on the DMPJOB command, see OS/400 CL ReferenceV4R4, SC41-5725.

4. Type WRKSPLF and press Enter. Look for a spooled file called QPSRVDMP. Thisis the dump generated by DMPJOB.

5. Do not forget to enter the End Service Job (ENDSRVJOB) command and pressEnter to end the Service Job.

3.1.4 MessagesMessages can provide very useful information when working on a problem. Youcan look for messages in the following places:

• The system operator message queue• Domino tasks job logs• The AS/400 system history log

3.1.4.1 QSYSOPR message queueMessages are issued to the AS/400 QSYSOPR (system operator) messagequeue. You can check for messages by typing DSPMSG QSYSOPR on an AS/400command line and pressing Enter. Figure 19 on page 40 shows an example of theQSYSOPR message queue.


Figure 19. Display Messages for QSYSOPR

Within the message queue, look for any messages with the message formatLNTxxxx, which is the format Domino for AS/400 is using.

3.1.4.2 Messages in a job logMessages are also issued to the job logs. Therefore, look at the job log that isreporting the error. Look for the job logs within the subsystem in which Domino isrunning. Or, if you are receiving errors while submitting Domino for AS/400commands, such as STRDOMSVR or WRKDOMCSL, you have two choices:

• Place the cursor on the error message at the bottom of your 5250 emulationand press F1. This shows you the Additional Message Information display withyour actual error message and message identification. If you press F10(Display messages within the job log), you see the Display all Messagesdisplay, which shows you the complete job log of your current job.

• On any AS/400 command line, type the Display Job Log (DSPJOBLOG) command,and press Enter. Then, press F10 (Display detailed messages) to show allinformation. This also shows you the job log of your current job. Here you cansearch for any previous messages within your job.

To see more information within the job log, you may have to press PgDn andPgUp. An example of the Display All messages display is shown in Figure 15 onpage 35.

Display MessagesSystem: AS25

Queue . . . . . : QSYSOPR Program . . . . : *DSPMSGLibrary . . . : QSYS Library . . . :

Severity . . . : 90 Delivery . . . : *HOLD

Type reply (if required), press Enter.Cleanup of OfficeVision/400 calendar items successfully completed.Cleanup of system journals and system logs started.Cleanup of system journals and system logs successfully completed.Cleanup has completed.Job 141815/QSYS/CRTPFRDTA completed normally on 02/08/00 at 00:00:02.Job 141894/DHQB/ANZDFTPWD5 submitted for job schedule entry ANZDFTPWD5number 001485.

Job 141894/DHQB/ANZDFTPWD5 completed normally on 02/08/00 at 01:00:00.Job 141945/DHQB/ANZDFTPWD1 submitted for job schedule entry ANZDFTPWD1number 001481.

Job 141945/DHQB/ANZDFTPWD1 completed normally on 02/08/00 at 06:00:00.Line TRNLINE status information, token-ring active monitor reassigned,line running.

BottomF3=Exit F11=Remove a message F12=CancelF13=Remove all F16=Remove all except unanswered F24=More keys

For errors on STRDOMSVR end ENDDOMSVR failures, always check theQNNINSTS job log.

Note


3.1.4.3 Messages in QHSTEnter the Display Log (DSPLOG) command and press Enter. The Display Log(DSPLOG) command shows the system history log (QHST). The history logcontains information about the operation of the system and system status.

The display contains the messages sent to the log, the date and time themessage was sent, and the name of the job that sent it.

3.1.4.4 Displaying a single messageFor specific information about messages, put the cursor on the message youwant information about and press the Help key (F1). An additional messageinformation display is shown (Figure 20).

Figure 20. Additional Message Information display

The Additional Message Information display gives you more detailed informationabout a message. Among the information shown is the date and time sent and themessage text along with a cause and recovery if applicable.

To display even more information about this message, select F9 (Displaymessage details). This display provides the name, library, module, procedure andstatement of the program that sent the message and to which the message wassent.

The message file location for Domino for AS/400 error messages isQNOTES/QLTSNSVMSG. To display a certain message description, enter thefollowing command:

DSPMSGD RANGE(LNTxxxx) MSGF(QNOTES/QLTSNSVMSG)

3.1.5 Directory synchronizationDomino for AS/400 directory synchronization provides the capability toautomatically synchronize user information in a Domino Directory with the user

Additional Message Information

Message ID . . . . . . : LNT0910 Severity . . . . . . . : 30Message type . . . . . : DiagnosticDate sent . . . . . . : 02/09/00 Time sent . . . . . . : 17:14:13

Message . . . . : Error occurred, unable to continue.Cause . . . . . : An error has occurred and processing can not continue.Recovery . . . : See previous messages in the job log.Technical description . . . . . . . . : Previous errors have occurred andprocessing can not continue.


F3=Exit F6=Print F9=Display message details F12=CancelF21=Select assistance level


information in the AS/400 system distribution directory. By taking advantage ofdirectory synchronization, you can avoid manually updating information for thesame user in two places. More details on that function can be found in the“Directory synchronization” chapter of Lotus Domino for AS/400 R5:Implementation, SG24-5592.

Information about the directory synchronization function can be found in thefollowing locations:

• QNNDIMSG message queue. You can enter the DSPMSG QNNDIMSG command.For an example of a displayed message queue, see Figure 19 on page 40.

• SDD/Domino Directory (System Distribution Directory/Domino Directory)messages use the format LNTxxxx. However, the message file isQNOTESINT/QLTSNINMSG. You an also use the WRKMSGQ QNNDIMSG command.

• The job logs of all directory synchronization jobs. These jobs are namedQNND* and are running in subsystem QSYSWRK. To display them, enter theWRKACTJOB command and press F4 (Prompt). Then, press F10 (Additionalparameters). You get the Work with Active Jobs (WRKACTJOB) display. TypeQNND* for the Job name parameter and press Enter.

You can also type the WRKACTJOB JOB(QNND*) command. You get the sameresults as above. Figure 21 shows the display that appears.

• The logging database for the Domino server is NNDILOG.NSF. Use a Notesclient to view it. Click File->Database->Open, and select your server. Thenselect the database Directory Synchronization Log, and click Open.

Figure 21. WRKACTJOB JOB(QNND*) results

3.1.6 Semaphores and shared memoryIn programming, especially in UNIX systems, semaphores are a technique forcoordinating or synchronizing activities in which multiple processes compete forthe same operating system resources. A semaphore is a value in a designated




Opt Subsystem/Job User Type CPU % Function StatusQNNDIMSG QNOTES BCH .0 PGM-QNNDIFMG MSGWQNNDIRQS QNOTES BCH .0 PGM-QNNDIRQS DEQWQNNDI2NA QNOTES BCH .0 PGM-QNNDI2NA DEQWQNNDI2SD QNOTES BCH .0 PGM-QNNDI2SD DEQW

BottomParameters or command===>F3=Exit F5=Refresh F7=Find F10=Restart statisticsF11=Display elapsed data F12=Cancel F23=More options F24=More keys


place in operating system (or kernel) storage that each process can check andthen change.

Depending on the value that is found, the process can use the resource or findthat it is already in use and must wait for some period before trying again.Semaphores can be binary (0 or 1) or can have additional values. Typically, aprocess using semaphores checks the value. Then, if it is using the resource, theprocess changes the value to reflect this so that subsequent semaphore userswill know they need to wait.

3.1.6.1 Displaying through a 5250 interfaceTo display information about semaphores and shared memory, enter thecommand:

CALL PGM(QP0ZIPCS) PARM('-Emsa')

The generated report contains information about shared memory andsemaphores for the entire AS/400 system.

To display this spooled file, enter the WRKSPLF command on the AS/400 commandline. Search for a spooled file with the name IPCS (Inter Process CommunicationStatus) and your own user ID. Enter option 5 (Display) to display the spooled filecontents.

Here are some tips for reading the IPC status report:

• An M in the IPC T column identifies shared memory.

• An S identifies a semaphore.

• Both semaphore and shared memory can be shared by several jobs.

You can see this when several entries in the JOBNUM field can be found.

• A damaged semaphore or shared memory has a D in the mode field as shownin the spooled file sample in Figure 22 on page 44. We actually entered the Dto show you the position, as we did not have a real example of a damagedsemaphore.

• The owner of the entries is displayed in the owner field.

The CALL PGM(QP0ZIPCS) PARM('-Emsa') command needs to be entered carefully,because the parameter is case-sensitive.

Note


Figure 22. Dump of shared memory and semaphores in an IPC Status report

3.1.6.2 Displaying through AS/400 Operations NavigatorIn AS/400 Operations Navigator, click Application Development->InterprocessCommunication->Semaphore Sets. Right-click Semaphore Number, and thenclick Properties to find the last job that modified the semaphore value. Anexample can be found in Figure 23. In this example, it is the AMGR job with jobnumber 147505.

Figure 23. Semaphores displayed through AS/400 Operations Navigator

AS/400 Operations Navigator can also display Shared Memory segments. InAS/400 Operations Navigator, click Application Development. Then, select

Display Spooled FileFile . . . . . : IPCS Page/Line 1/6Control . . . . . Columns 1 - 130Find . . . . . .*...+....1....+....2....+....3....+....4....+....5....+....6....+....7....+....8....+....9....+....0....+....1....+....2....+....3IPC STATUS AS OF Thu Feb 17 15:46:36 2000T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME

AJOBNUM NUMATTSHARED MEMORY:M 3 0X0100854B --RW-RW-RW- QTMHHTTP *NONE QTMHHTTP *NONE 1 1488 241 31 07:53:56 00:00:00 07:53:56

143792 1M 4 0X0200854B --RW-RW-RW- QTMHHTTP *NONE QTMHHTTP *NONE 1 1488 241 34 07:53:58 00:00:00 07:53:58

143793 1M 300 0XF8000800 --RW-RW---- QNOTES *NONE QNOTES *NONE 14 6288224 64 1554 15:37:03 15:37:04 14:35:48

147522 1147511 1147510 1147509 1

147508 1T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME

SEMNUM SEMVAL LJOBNUM WAITZ WAITP WAITVALSEMAPHORES:S 1054 0000000000 --RW-RW-RW- QNOTES *NONE QNOTES *NONE 1 15:38:55 14:35:49

0 1 147499S 1055 0XF6010501 D-RW-RW-RW- QNOTES *NONE QNOTES *NONE 3 15:39:38 14:35:49

0 1 1475051 9985 P0015542 0 147499

F3=Exit F12=Cancel F19=Left F20=Right F24=More keys


Interprocess Communication->Shared Memory, and right-click Memorysegment identifier. Click Properties->Job Attached to get the list of thecurrently attached jobs. You can see an example in Figure 24. Click General tosee the general information for this shared memory segment.

Figure 24. Display of shared memory with OPSNAV

3.1.7 Environment variablesEnvironment variables are used to choose specific settings for the Domino serverenvironment in OS/400.

The Work with Environment Variables (WRKENVVAR) command shows you a listof current environment variables and their names. From here, you can selectoptions to add, change, remove, display the details of, or print the environmentvariables. You must have *JOBCTL special authority to use this command to add,change, or remove system-level environment variables.

For example, to add the NOTES_AS400_STSDEBUG variable to dump allenvironment variables at server start to the QNNINSTS job log, perform thefollowing tasks:

1. On an AS/400 command line, enter WRKENVVAR, and press F4.

2. Select *JOB or *SYS:

• *JOB to set the variable only within your current job• *SYS to set the variables at each server start


3. Enter 1 for ADD. Use NOTES_AS400_STSDEBUG as the variable name.

4. Change the default value *Null to 1 and press Enter.

You have the choice between *JOB environment variables (*JOB) and systemenvironment variables (*SYS) since OS/400 V4R4. The job environment variableincludes the system environment variables.

Some of the more frequently used variables are:

• FaultRecovery, AS400_DOMSVR_RESTART: See 8.4, “Server crash andautomatic restart” on page 137.

• PATH: This variable indicates the data directory that should be used for theexecution of subsequent commands that do not specify a path in thecommand itself.

• NOTES_AS400_CONSOLE_ENTRIES: The console entries for a Domino forAS/400 console are written in a stream file stored in your data directory. Thereyou find a file named SUBSYSTEMNAME.NOTESHST (whereSUBSYSTEMNAME is substituted by the name of the subsystem in which theserver is running). This file contains the console entries for the Dominopartition. The WRKDOMCSL and DSPDOMCSL commands are interfaces thatlet you work with this file. The SUBSYSTEMNAME.NOTESHST file is clearedat the start of a server run. There is a limit to the number of console entriesthat can be stored in this file. The default limit is 2000 entries, and themaximum is 100,000 entries. When the limit is reached, the console starts towrap. The WRKDOMCSL/DSPDOMCSL support handles this. If for anyreason you want to allow more entries in the file, add the environment variableNOTES_AS400_CONSOLE_ENTRIES. If the variable is set to a value greaterthan the maximum, which is 100,000, it is set to the maximum.

• NOTES_AS400_STSDEBUG: This environment variable can be used todebug a server problem. If the variable is set to 1, the following elements aredumped to the QNNINSTS job log:

– All environment variable settings– The link to the program object used on the spawn request– The restart values– The current library list– If a restart occurs, additional information about the stream file copied,

where it was copied to, the path used, and if the copy request wassuccessful

3.2 Domino locations

From a Domino for AS/400 point of view, we have to look at the followinglocations:

• Domino console• Domino log database• Domino Web server log database and files• NSD files


3.2.1 Domino consoleThere are four ways to interact with the console of a Domino for AS/400 server.

• First, you can use the remote console support that is available from the Notesclient.

• Second, you can use the Display Domino Console (DSPDOMCSL) command.This allows you to view the console entries but does not allow you to enterconsole commands. Any number of people on the system can be running theDSPDOMCSL command at the same time.

• The third way to interact with the console is through the Work with DominoConsole (WRKDOMCSL) command. This permits you to view the consoleentries and enter the familiar Domino console commands. There is a limit ofonly one person using the WRKDOMCSL command for a given partition at onetime. This means is that, if you have two Domino partitions on your system (forexample, DOMSVRA and DOMSVRB), a person can be in WRKDOMCSL forDOMSVRA and another person can be in WRKDOMCSL for DOMSVRB at thesame time. However, two people cannot be in WRKDOMCSL for DOMSVRAat the same time.

• The fourth way is to use the Submit Domino Command (SBMDOMCMD)command. It allows you to submit a Domino server console command to aDomino server without accessing the Domino console. You can use thiscommand if someone else has acquired the console using the Work withDomino Console (WRKDOMCSL) command or you want to launch commandsfrom a batch program.

The implementation of the console support for Domino on the AS/400 system isunique from all other platforms. The console entries are actually written to astream file in the Domino partition's data directory. If you look in the datadirectory, you will find a file named DOMINOxx.NOTESHST (where DOMINOxx isthe partition's subsystem name). This file contains the console entries for theDomino partition. The WRKDOMCSL and DSPDOMCSL commands are theinterfaces that let you work with this file.

This implementation has a couple of benefits. One is that, if you useDSPDOMCSL or WRKDOMCSL while the server is not running, you will see theconsole entries for the last running of the server. The DOMINOxx.NOTESHST fileis cleared when a server starts running, so the last running is preserved in the fileuntil the server is started again. This enables you to view that file, even if thedomino server is not running.

It can happen that some debugging statements are not logged into the log.nsf. Todump all information from the Domino console, either active or stopped server,type the WRKDOMCSL or DSPDOMCSL command, and press Enter. Then, press first F5to refresh the console, then press F6, and press Enter. A spooled file is createdfor your user. To display the spooled file, enter the WRKSPLF command. Search atthe bottom for a file called QSYSPRT, and enter 5 in the options field.

One limitation with the AS/400 implementation of the Domino console support is thatthere is a limit to the number of console entries allowed in the file. The default limit is2,000 entries, with a maximum of 100,000 entries. When the limit is reached, theconsole file starts to wrap. The WRKDOMCSL and DSPDOMCSL support handlesthis wrapping, but there are times when 2,000 entries is not enough. Therefore, theenvironment variable, NOTES_AS400_CONSOLE_ENTRIES, is used to increase


the amount of entries in the console file. This variable must be set before starting theserver. If the value of the variable is greater that the maximum, it is set to themaximum value.

The following process is only valid for R4.6x. When the server is automaticallyrestarted, the console is copied to a temporary file. The name of the file can be foundin the job log of the QNNINSTS job. If you put the cursor on the line that begins withDomino server has ended,... and press PF 1, the second-level message appears.This message tells you where the console file has been copied. As shown in Figure25, the console file has been copied to the/TMP/DOMINO08.NOTESHST.19980309093448874000000 file.

You can use the Display File (DSPF) or Display Stream File (DSPSTMF) commandto view this file. For more information on these commands, refer to 7.1,“Displaying or editing a stream file from a 5250 display” on page 117.

The files in the /TMP directory are not erased automatically. You need toperiodically display the /TMP directory and delete the old NOTESHST files, sinceR5, the NOTESHST file is deleted at each start of the server.

Figure 25. Where the NOTESHST file can be found

3.2.1.1 Domino server console commandsThis section lists some of the server commands. They must be typed at theserver console.

broadcast “msg” [“user”] Broadcasts a message to users of this server

dbcache Database cache management commands

DISABLE Disables use of the database cacheFLUSH Clears out the database cacheSHOW Shows the contents of the database cache

Additional Message Information

Message ID . . . . . . : LNT0929 Severity . . . . . . . : 20Message type . . . . . : DiagnosticDate sent . . . . . . : 03/09/98 Time sent . . . . . . : 09:34:49Message . . . . : The Domino server has ended, probably because of anerror. Trying to restart the Domino server.

Cause . . . . . : The Domino server has probably encountered anunexpected error. When this happens, the Domino server tries to endany related Domino jobs and clean up any system resources that havebeen used.

Recovery . . . : Recovery may not be needed. This job will try torestart the Domino server.If the Domino server fails to start, the console log/TMP/DOMINO08.NOTESHST.19980309093448874000000 and the current errorreturned by the restart request will be needed to diagnose anyproblems.

Technical description . . . . . . . . : The completioncode returned by the main Domino server job indicates that theDomino server jobs were ended

More...Press Enter to continue.

F3=Exit F6=Print F9=Display message details F12=CancelF21=Select assistance level


drop [“username”] [ALL] Drops one or more sessions

exit [password] Exits the server

help Displays help information

load pgmname Loads the program

pull servername Replicates one-way (pull)

push servername Replicates one-way (push)

quit [password] Quits or exits the server

replicate servername Replicates a two-way request

restart Restarts information:

SERVER [password] Restarts the serverPORT portname Disables or enables transactions on a port

route servername Routes mail to the server

set Sets server options:

CONFIGURATION “variable=value” Configuration variableSECURE [current-password] [new-password]Secure console passwordSTAT [Facility] [Statname] Reset statistics

show Shows server information:

ALLPORTS Shows configuration for all portsCLUSTER Clusters informationCONFIGURATION variable Configuration variableDATABASE filename Shows database informationDBS Shows open database informationDIRECTORY Directory informationDISKSPACE filesystem Available disk spaceMEMORY Memory informationPORT portname Port specific informationSCHEDULE Next schedule [Server/Program/Location] [Appl]SERVER Server informationSTATISTIC variable Statistic variableTASKS Server tasksUSERS Users with open sessions

start Starts the specified service

PORT portname Enables transactions on a port

stop Stops the specified service

PORT portname Disables transactions on a port

tell taskname command-string Sends command string to a task

The following pages show the syntax, a description, and examples of some ofthe major Domino server commands when typed on the server console.Detailed information about these commands is described in the onlineDomino 5 Administration Help guide (help5_admin.nsf) in the “Domino servercommands” chapter.

Important


broadcastSyntax: broadcast message usernames

Description: This command sends a message to specified users or to all users ofthis server. Use this command to warn users when a server is brought down formaintenance. The message you enter appears in the user's status bar.

Example:

> broadcast “this is a test note” “Michel Dupont”

dropSyntax: drop username

Description: This command loses one or more server sessions. To visuallyconfirm which sessions are dropped, you must enter the Log_Sessions=1 settingin the server's NOTES.INI file.

Examples:

> drop “Mary” Closes the current session running under the username Mary

> drop “Peter” “Michael” Closes the sessions running under the user namesPeter and Michael

> drop All Closes all server sessions

loadSyntax: load program name

Description: This command loads and starts a specified server task or programon the server. You can start a server add-in program or one that takes acommand line for additional data, such as a backup program. The program yourun must be on the server's search path. Use the load command to run a programuntil it completes or, if the program runs continually, until you stop the server.Where applicable, you can include arguments that determine how the programruns.

Example:

> load fixup Loads and runs the fixup server task

quitSyntax: quit

Description: This command stops the server. This command is identical to the exitserver command. However, the quit server command differs from the tell servercommand, which you use to stop a particular server task without stopping theserver.

If you stop a server while it's replicating databases or routing mail, these tasksresume at the next scheduled interval after you restart the server. Replication ormail routing continues until the databases are fully replicated and until thecomplete mail message is transferred or returned to the sender.


Before you use the quit server command to stop the server, use the broadcastserver command to warn users to finish their current tasks before you stop theserver.

replicateSyntax: replicate servername [databasename]

Description: This command forces replication between two servers: the serverwhere you enter this command and the server you specify. Use the server's fullhierarchical name. If the server name is more than one word, enclose the entirename in quotes.

To force replication of a particular database that the servers have in common,specify the database name after the server name. The initiating server (whereyou're currently working) first pulls changes from the other server, and then givesthe other server the opportunity to pull changes from it. You can use thiscommand to distribute changes quickly or to troubleshoot a replication orcommunication problem.

If the server is already replicating when you issue the command, Domino queuesthe command until the current replication ends. To check the status of theReplicator, enter the show tasks or sh ta command on the console.

The server displays one of the following messages:

• If the server isn't replicating, the word Idle appears next to the replicatorprogram.

• If the server is replicating, a status line, such as Replicating ORDERS.NSF from

MARKETING\ORDERS.NSF, appears.

To optimize resources, Domino replicates only what is necessary. For example, ifthe servers recently replicated, and no changes have since been made to anydatabases on either server, the servers will not replicate when you enter aReplicate command. Also, the replication is two-way only if databases on bothservers have changed since the last replication. If databases on only one of theservers changed, the replication is one-way.

To force replication in only one direction, use the Pull or Push server commands.

Examples:

> replicate domsvr3\Forum Test Initiates replication between yourserver and the domsvr3\Forum Test

server

> replicate domsvr3\Forum Test NAMES.NSF Initiates replication of NAMES.NSFbetween your server and thedomsvr3\Forum Test server

set configurationSyntax: set configuration setting

Description: This command adds or changes a setting in the NOTES.INI file.


Example:

> set config SERVERTASKS=Router,Replica,Update,Amgr,AdminP,CalConn,Sched,Event,Stats,HTTP,DECS

show portSyntax: show port portname

Description: This command displays traffic and error statistics.

Example: See Figure 26.

Figure 26. show port tcpip command

show statSyntax: show stat statisticname

Description: Used without the optional statisticname argument, this commanddisplays a list of server statistics for disk space, memory, mail, replication, andnetwork activity. To display a single statistic, enter the name of the statistic as theoptional argument. To display only a subset of statistics, add a group of statisticsas an optional argument by using an asterisk (*) as a wildcard.

An example is shown in Figure 27.

Figure 27. show stat command examples

> show port tcpip

> show port tcpip

TCP/IP Port DriverTransport Provider: TCP

Notes Session Local Address Foreign Address153F0001 10.8.62.33:1352 *:*

> show stat server.version.notesServer.Version.Notes = Release 5.0.1 (Intl)

> show stat server.version.osServer.Version.OS = OS400 V4R4M0

> show stat server.users.peak.timeServer.Users.Peak.Time = 02/23/2000 11:21:26 CET

> show stat database.n*Database.NAMELookupCacheCacheSize = 7,744Database.NAMELookupCacheHashSize = 450,898Database.NAMELookupCacheHits = 2,605Database.NAMELookupCacheLookups = 3,781Database.NAMELookupCacheMisses = 1,176Database.NAMELookupCacheNoHitHits = 0Database.NAMELookupCachePool.Peak = 1,048,576Database.NAMELookupCachePool.Used = 471,296Database.NAMELookupCacheResets = 0Database.NAMELookupMisses = 988Database.NIFPool.Peak = 1,048,576Database.NIFPool.Used = 281,472Database.NSFPool.Peak = 1,048,576Database.NSFPool.Used = 243,808

>


Reading server statistics outputIf we cannot analyze and understand our servers, generating statistics has novalue. It is with statistics that we can understand the behavior of our servers andapplications. Then we can decide how well we can optimize them. The followinglist describes some of these statistics.

• Database.BufferPool.Maximum: The maximum allowed NSF buffer pool forthe Domino server. There are two ways to set this variable,NSF_Buffer_Pool_Size and NSF_Buffer_Pool_Size_MB. If usingNSF_Buffer_Pool_Size, you need to specify the correct number of zeroes,such as 250000000. You can find it much easier to useNSF_Buffer_Pool_Size_MB, as the same setting would be specified at 250.The recommended setting varies depending on which release of Domino youare running on the AS/400 system and the number of partitioned Dominoservers that are being used.

Consider these recommendations:

– For Release 4.6.x, set NSF_Buffer_Pool_Size_MB to a value of 75% of thepool size that the Domino subsystem is running in, which is usually pool 2(QBASE), or a maximum of 507 MB.

– In Release 5.0.x, set NSF_Buffer_Pool_Size_MB to a value of 25% of thepool size that the Domino subsystem is running in, or a maximum of 300MB.

The Domino server needs to be stopped and restarted for the change to takeeffect.

• Database.BufferPool.Peak: This value provides an accumulated figure thatthe pool has grown since the server startup.

Choosing the biggest possible value for Database.BufferPool.Maximum isnot always the best choice. Different values should be tested, and the realbuffer pool usage should be checked.

Use the SHOW STAT command on the Domino console to obtain the followinginformation:

• Buffer pool maximum• Buffer pool used• Buffer pool peak

If the peak is less than 95% of the maximum, the index is performingoptimally within the buffer. Use the reported tasks to capture statistics perserver over a period of time, and graph them perhaps to identify trends andpeaks.

Note

Do not be alarmed if your statistics show that the values of Peak and Usedexceeded the Maximum value by a small percentage. This is due to the waythe server reads some data structures into the pool.

Note


• Database.BufferPool.PerCentReadsInBuffer: The cumulative percentage oftime that the buffer pool was used to service requests. The higher this number,the better the results.

• Database.BufferPool.Reads: The cumulative number of times the buffer hasbeen read.

• Database.BufferPool.Used: The amount of memory allocated within thebuffer pool. This value may be misleading, because it does not reflect theamount of the buffer pool that is currently being utilized, but rather the amountthat has been used. Prior to Domino 5, this value is recycled only if it reachesits maximum or when the server gets recycled. In Domino 5, the server taskwill manage it efficiently.

• Database.BufferPool.Writes: The cumulative number of times that data hasbeen written to the buffer.

• Server.Users: Specifies the number of active Notes Sessions. This includesall users and all server tasks connecting to the server (for example, theReplicator).

• Server.Users.1MinPeak: This specifies the number of users who have, in thepast minute, performed a Notes transaction (some network activity caused, forexample, when opening a database or document on the server). This valuedoes not specify the number of users or transactions in the past minute. TheDomino server checks this data regularly and updates the value only if thenumber is greater than the previous value.

• Server.Users.1MinPeakTime: This value shows the date or time at which themaximum number of users have done a network operation in the past minute.Note that “Server.Users.1MinPeak” specifies the number of users during thispast minute. Users who have not done any transaction so far, but are waitingfor a transaction, are also counted in this list.

• Server.Users.5MinPeak and Server.Users.5MinPeakTime: These show thenumber of users and the time, respectively, at which the server had the mostusers performing a transaction during the past five minutes.

• Server.Users.Peak and Server.Users.PeakTime: These show the maximumnumber of users and the time for users who had an active connection to theserver.

Show tasksThis command displays the name of the server, the path of the Domino programdirectory, and the status of the active server tasks. An example is presented inFigure 28.


Figure 28. show tasks example

tellThis command issues a command to a server program or task. The command isespecially useful for stopping a server task without stopping the server. This isthe most powerful server command, with a lot of possibilities.

For example, > tell http quit stops only the HTTP task. All other tasks on theserver continue to run.

Some tell commands are common to all server tasks, such as the tell task quit

command. Other tell commands are unique to a particular task. These tasks haveunique tell commands and are available for the following services:

• Administration Process• Agent Manager• Cluster Replicator• NNTP

Lotus Domino (r) Server (5.01.02 for OS/400) 03/01/2000 03:26:18 PM

Server name: DOMSVR3/Forum TestServer directory: /domino/DOMSVR3Partition: DOMSVR3Elapsed time: 06:08:26Transactions/minute: Last minute: 0; Last hour: 0; Peak: 47Peak # of sessions: 3 at 03/01/2000 11:21:26 AMTransactions: 190Availability Index: 100 (state: AVAILABLE)Message Tracking: Not EnabledShared mail: Not EnabledNumber of Mailboxes: 1Pending mail: 0 Dead mail: 0Waiting Tasks: 0Transactional Logging: Not Enabled

Task Description

Database Server Perform console commandsDatabase Server Listen for connect requests on TCPIPDatabase Server Load Monitor is idleDatabase Server Idle taskDatabase Server Idle taskDatabase Server Perform Database Cache maintenanceDatabase Server Idle taskDatabase Server Idle taskDatabase Server Idle taskDatabase Server Idle taskDatabase Server Idle taskHTTP Web Server Listening on port(s) 8080DECS Server 5 Active RealTime ActivitiesEvent Monitor IdleCalendar Connector IdleSchedule Manager IdleAdmin Process IdleAgent Manager Executive '1': IdleAgent Manager IdleStats IdleIndexer IdleRouter Idle


• LDAP• Router• Schedule Manager• Statistic Collector• Web Navigator• Web Server

Note: For detailed information about the different tell commands, see the onlineDomino 5 Administration Help guide (help5_admin.nsf) in the “Domino servercommands” chapter.

3.2.2 Domino log databaseEvery Domino for AS/400 server has a log file (LOG.NSF) that reports all serveractivity and provides detailed information about databases and users on theserver. The log file is created automatically when you start a server for the firsttime.

The following information corresponds to Domino R5. To view the log file(LOG.NSF) using Domino Administrator:

1. From the Domino Administration, click the Server - Analysis tab.2. Select the server that stores the log file you want to view.3. Click Notes Log.4. Click the desired view.5. Open the desired document.

You can also use the Web Administrator to open the log (LOG.NSF).

3.2.3 Domino Web server log database and filesThe Domino Web server log database (DOMLOG.NSF) is used to log all DominoWeb server activity and tracks information about each HTTP request such as:

• Date and time the request was made

• User's IP address (or the DNS address if DNS lookup is enabled in the Serverdocument)

• User's name (if the user supplied a name and password to access the server)

• Status code the server returns to the browser to indicate its success or failurein generating the request

You can log Domino Web server information in the Domino Web server log(DOMLOG.NSF), in text files, or in both the Domino Web server log and text files.

The Domino Web server log database and files allow you to know which HTTPrequest was the last to be triggered. It also helps gather diagnostics for EVERYCRASH to determine:

• What the server was doing when it crashed• What URL the server was processing at the time of the crash

More information on Domino Web server log database and files is available in9.1.3.2, “Domino Web server log database DOMLOG.NSF” on page 142, and9.1.3.3, “Domino Web server log text files” on page 143.


3.2.4 NSD diagnostic toolThe Notes Server Diagnostic (NSD) is a Lotus tool designed to gather informationabout a failing Domino server. This tool has been ported to run on OS/400 for usewith Domino for AS/400. This is the main tool for studying Domino servercrashes.

The console log is reproduced at the end of the NSD file (from Release 4.6.4.01and later releases, including 5.0.1 and later releases).

3.2.4.1 NSD activation on the AS/400 systemTo check if the NSD tool is active on your Domino for AS/400 server, look in yourNOTES.INI file for the CleanupScriptPath=CALL QNOTES/NSD parameter. If thisparameter is set, an NSD file is generated in your Domino server data directory.

3.2.4.2 Listing NSD filesTo get a list of the actual existing NSD files in your Domino server data directory,you can either use the Work with Object Links (WRKLNK) command or AS/400Operations Navigator, or map your AS/400 system as a network drive.

Change your current directory to your Domino server data directory and look forfiles starting with nsd*. The complete name would be, for example,nsd_20000211_07.48.39.nsd. In this example, the name of the file contains20000211, which is the date of the dump, and 07.48.39, which is the time of thedump.

In the WRKLNK display, the file names for the NSD files are truncated. You canselect option 8 to display the full path and name of the NSD file. This informationis usually copied to be pasted in the file name parameter of the Display File(DSPF) command or Edit File (EDTF) command. For more information on thesecommands, refer to 7.1, “Displaying or editing a stream file from a 5250 display”on page 117.

Because the original naming format provides long file names, you may want torename the file prior to working with it. Giving it a short, but meaningful, namemakes it much easier to work with the files.

There are no RIP files generated by Domino for AS/400 (since there are NSDfiles). RIP files are the equivalent of NSD files on Windows NT implementationof Domino.

Note

There is another way to quickly find the NSD files for a Domino server.

In the WRKDOMSVR list, enter option 12, and press F4, so you don’t have tokey the full path of the Domino server’s data directory. Then replace the *

(asterisk) at the end of the path by *.NSD. Press Enter. You will see only theNSD files contained in the directory.

Tip


Renaming an NSD file, or copying it to another directory, can also help to keep itfor future reference.

3.2.4.3 Reading an NSD fileYou can read an NSD file in two ways:

• Use the Display File (DSPF) command or the Display Stream File (DSPSTMF)command. For more information on these commands, refer to 7.1, “Displayingor editing a stream file from a 5250 display” on page 117.

• Display the file by using any ASCII editor on you workstation. Therefore, youhave to download the file via FTP to your workstation by starting an FTPsession on your workstation and type, in order:

1. ftp MyAS400

2. userid

3. password

4. ascii

5. cd /mynotes/mydata (data directory of your Domino server)6. get nsdname

7. quit

Notice that using the binary mode here transforms the AS/400 stream file (STMFfile type) to a plain ASCII file. Now you can open the NSD file with a standardASCII editor on your workstation.

3.3 When to use a source of information

As we have seen in this chapter, there are many places where information can belooked for when having a problem. Table 3 lists all the available sources ofinformation and describes when they can be helpful for problem determination.

Table 3. When a source of information is useful

Source of information When What can be found

Active jobs, job logs, job logspooled file

Server or task no longerrunning normally

Error messages (LNTxxxx,CPIxxxx, MCHxxxx...)

Dump files Job appears hung or islooping

Failing program, instruction,and error condition

Messages Server starting or endingabnormally, failing tasks

Specific types of error

Semaphores and sharedmemory

System resource notavailable

Damaged semaphores orshared memory

Call stack information on asecondary thread

Job appears hung or islooping

What procedure is the causeof the hang or loop

Collecting a VLIC Trace onan Active Job

Crash where the failingprocedure is below the MI

Information related todevelopment

Console Abnormal situation anddebugging statements set

Recent activities/problems/debugging information

LOG.NSF Abnormal situation Activities history

Web server log databaseand files

HTTP crash Recent accesses, internalserver errors, CGI problems


NSD file Server crash Current job information, callstack, console status,NOTES.INI settings used attime of crash

Automatic server restartsettings

Server crash Automatic recovery on or off,number of attempts torestart

Source of information When What can be found


Chapter 4. How to use the information you gather

How can you evaluate a problem? Which tools can you use to search for similarproblems? There are many technical databases and information repositoriesavailable from Lotus, IBM, and other sources. Such information sources arehelpful at the beginning of the problem source identification (described in 1.3.2,“Problem source identification” on page 17).

4.1 Lotus resources

Lotus provides a lot of technical information free of charge on the Web.Knowledge Base is the first tool that should be used to look for the answer to aproblem, specially if the problem appears to be a software bug.

This section introduces you to Knowledge Base and other resources that areavailable from Lotus.

4.1.1 Lotus Notes Knowledge BaseKnowledge Base is the same Lotus Notes database used by Notes Support toanswer support questions about Domino. Knowledge Base is available on:

• CD-ROM

• The Web at: http://www.lotus.com

More details on accessing Knowledge Base on the Web are given later in thissection.

Originally designed to assist Lotus Support personnel in supporting the Lotusproducts, Knowledge Base has evolved into a system for managing the deliveryand retrieval of all technical information available from Lotus.

In addition to the latest technical notes, also known as Technotes, KnowledgeBase contains several types of information and tools, including:

• Troubleshooting scripts

• White papers

• Release notes written by members of the Development and Technical Supportstaff

• Product news

• System requirements

• Product press releases

• Executable programs

• Tools

• Product updates

• Other support databases

• A complete catalog of educational videos, books, and other supplementalmaterials


Finding information in Knowledge Base is easy. You can navigate using themenus and views or use the full-text search feature to search either the entiredatabase or any level within the database's hierarchy.

A Software Problem Report (SPR) is a written record of a single anomaly foundduring testing or using a build or release of a Lotus software product.

As shown in Figure 29, three views about SPRs are available in the KnowledgeBase:

• Software Problem Reports: By Category• Software Problem Reports: By SPR Number• Software Problem Reports: By SPR Status

SPRs can fall into several different categories, and not all of these reports end upbeing categorized as bugs. For example, after investigation it may be determinedthat the software is functioning as designed or that the issue is the result offeature-usage misunderstanding. In these cases, the Knowledge Basedocuments are updated to reflect this. If the SPR is determined to be a softwarebug, the Knowledge Base documents contain any known workarounds and areupdated with SPR status changes and any fix information as it becomesavailable.

Figure 29. Lotus Knowledge Base

Figure 30 shows a Technote, displayed in Lotus Notes. The status of the SPR isshown.

The status of an SPR can be:

• New• Open• Pending• Fixed/Resolved• Closed• Transferred


Most SPRs go through the following states, from creation through resolution:

1. When an SPR is first entered into the database, the SPR is new, and it'sassigned to a Lotus development engineer.

2. The Lotus development engineer determines whether the SPR is a valid bug.If the bug is valid, the Lotus development engineer clicks the Validate buttonand responds to the validation prompts. The SPR system then changes theSPR's state to Open and assigns it to a developer or writer. If the bug is notvalid or is a duplicate, the Lotus development engineer clicks the Close buttonand responds to the close prompts. The SPR system then changes the SPRstate to Closed. No further action is needed for a closed SPR.

3. A developer or writer works on an open SPR, and, optimally, fixes it. Thedeveloper or writer then clicks the Dev Action button and responds to theaction prompts. The SPR system changes the SPR's state to Pending andassigns it to a Lotus development engineer for verification.

4. The Lotus development engineer verifies the developer's or writer's action.The Lotus development engineer then clicks the Verify button and responds tothe prompts, indicating whether the bug is fixed and whether the engineeragrees with the developer or writer's action.

If the Lotus development engineer agrees that the SPR is fixed or with thedeveloper/writer's action, the SPR system changes the SPR state to Resolved.No further action is needed for a resolved SPR.

If the Lotus development engineer disagrees, the SPR system resets the SPRstate to Open and again assigns the SPR to the developer or writer for furtherwork.

Figure 30. Technote displayed in Lotus Notes

The Knowledge Base is available for customers and business partners on theWeb. This access is free. To access it from a Web browser, follow these steps:

Chapter 4. How to use the information you gather 63

1. Connect to http://www.lotus.com

2. Click Tech Support.3. Click Technical Resources.4. Click Technical Search and Knowledge Base Library.5. Click Notes/Domino Knowledge Base.6. Click Server.7. Choose a subtopic.

The last two steps can be replaced by clicking Search and entering searcharguments.

On the Web, the documents appear as Customer Support Technotes and includethe document number, the publication date, the problem, and the solution. Figure31 shows an example.

Figure 31. Technote accessed on the Web

4.1.2 Other technical resourcesOther Lotus technical resources are also available on the Web. This sectionintroduces you to the Technical Resource page.

The Technical Resources page, which is on the way to Lotus Knowledge Base,also contains links to other valuable sources of information. Such resourcesinclude the Lotus product documentation, support downloads, developerresources, and IBM Redbooks.

To access the Technical Resources page from a Web browser, follow these steps:


1. Connect to: http://www.lotus.com2. Click Tech Support.3. Click Technical Resources.

4.2 IBM tools

The Web is a place that the AS/400 division is using actively to publish technicalinformation about the AS/400 system and AS/400 numerous solutions. Since thefirst days after its announcement, a lot of marketing and technical informationabout Domino for AS/400 has been made available at: http://www.as400.ibm.com

Direct access to that information is located at: http://www.as400.ibm.com/domino

The AS/400 technical support Web page is another place where usefulinformation can help in problem determination.

4.2.1 Lotus Domino for AS/400 support information Web pageThe Lotus Domino for AS/400 support information Web page can be accessed at:http://www.as400.ibm.com/domino

From the IBM Domino for AS/400 Web page, click Support to access the pagedisplayed in Figure 32.

Figure 32. Domino for AS/400 support information Web page


This is the place where information about Quarterly Maintenance Updates(QMUs), Quarterly Maintenance Releases (QMRs), and OS/400 PTFs isavailable.

4.2.2 AS/400 Technical Support Web pageThe page displayed in Figure 33 can be accessed at: http://as400service.ibm.com

Figure 33. AS/400 Technical Support Web page

This site is the main entry point for AS/400-related technical questions on theWeb. You can find information sources about troubleshooting tips, such as:

Quarterly Maintenance Updates (QMUs) fix specific issues that occur betweenreleases of Domino for AS/400.

Domino Quarterly Maintenance Releases (QMRs) are intermediate releasesthat are published several times a year.

For IBM software support, a PTF is a program temporary fix. This is a programpackage allowing to apply corrections to an operating system or an application.

More information on QMUs, QMRs, and PTFs can be found in Chapter 5, “Howproblems can be solved” on page 69.

Note


• Support line Knowledge Base: This database contains documents createdby representatives in the AS/400 Support Center. Documents go through anextensive review process to ensure technical accuracy. The database is usedinternally by the AS/400 Support Center to search for customerproblems/solutions.

Documents classified as Public Use or Registered are replicated to theas400service.ibm.com Web site. The registered documents are onlyaccessible by Monthly Support Line customers that have registered with thesite.

• Authorized Problem Analysis Reports (APARs) database: APAR stands forAuthorized Program Analysis Report. APARs are requests for correction of adefect in a current release of an IBM-supplied program. An InformationalAPAR provides information not linked to a particular problem, such asconsiderations about a product or function or specific technical informationthat was not previously documented in a manual.

The IBM AS/400 Authorized Problem Analysis Reports database can beaccessed from the Web page shown in Figure 33(http://as400service.ibm.com). On the left pane, click Tech Info & Databases,and then Software Problems - APARS. A full text search is provided.

• Program temporary fixes (PTFs) database: A PTF is a program packagethat allows you to apply corrections to applications and operating systemfunctions installed on the AS/400 system. A PTF corrects a problem reportedby an APAR.

PTFs come with cover letters that state the exact procedures andcircumstances necessary for application.

The IBM AS/400 PTF Cover Letters database can be accessed from the Webpage shown in Figure 33 (http://as400service.ibm.com). On the left pane, clickTech Info & Databases, and then PTF Cover Letters. A full text search isprovided.

For more information on AS/400 PTFs, refer to 5.2, “Implementing AS/400software fixes” on page 70.

4.3 Other tools

Several AS/400 magazines host technical forums about Domino and Domino forAS/400 on the Web. Most of the topics concern the practical implementation of afunction. They also include discussions about problems.

When browsing the AS/400 Technical Support Web page, displayed in Figure 33,which can also be accessed at: http://as400service.rochester.ibm.com

Click Forums and Discussions. The next page, displayed in Figure 34 on page68, offers you access to several non-IBM discussion forums and discussiongroups.


Figure 34. Access to non-IBM forums and discussion groups


Chapter 5. How problems can be solved

As explained in 1.3.3, “Problem fixing” on page 17, different kinds of problemscan have different kinds of solutions. In this chapter, we give an overview of thepossible types of solutions.

A solution for a problem can be:

• Correcting a user error

• Installing an OS/400 PTF

• Implementing the latest Domino for AS/400 code Quarterly MaintenanceRelease (QMR)

• Installing a Domino for AS/400 Quarterly Maintenance Update (QMU)

• Obtaining a hot fix from Lotus

Before describing how Lotus provides fixes for Domino, it should be clear how thePTF process on AS/400 works. A “PTF” refers to two different concepts:

• A process within IBM Support to distribute and track software fixes for theAS/400 system and other IBM servers. Lotus does not participate with thisprocess (although IBM started recently to include Domino fixes within PTFscumulative tapes).

• A function in OS/400 to implement and log fixes to any kind of software,including OS/400, IBM Licensed Program Products (LPPs), potentially anycustomer or Independent Software Vendor-written software, and finallyDomino. QMUs and hot fixes are PTFs.

5.1 User errors

When looking for the reason of a problem affecting a Domino server, user errorsmust not be forgotten as a possible cause. They can happen in several places, forexample in a Domino for AS/400 server administration process or in applications(for example in an agent written in LotusScript or Java).

In the Lotus support technology, “Not A Bug” is a substate in the SPR database(others may be “Reproduced”, “Fixed”, and so on), which is used whendetermining that the problem is a user error.

Examples of user errors are:

• Stopping a Domino for AS/400 using the End Subsystem (ENDSBS) CLcommand to stop the subsystem in which the Domino for AS/400 server isrunning instead of using the End Domino Server (ENDDOMSVR) CLcommand.

This results in having Domino databases damaged. A fixup is necessarybefore restarting the server.

• An agent has been written that uses the Java toolkit for OS/400. Theapplication attempts to convert a string to an integer.

• An agent calls an external program and passes an invalid number ofparameters.


• The customer had five servers configured, all for AS/400 SMTP support or*MSF on the same system, and all were running at the same time.

• A Domino server directory was moved (or even deleted), or the ownership ofsome files changed from QNOTES to another AS/400 user profile, andQNOTES can no longer access the Domino server environment.

5.2 Implementing AS/400 software fixes

This section describes what a PTF is and gives information on how AS/400 PTFscan be implemented to help solve a Domino for AS/400 problem.

5.2.1 What a PTF isFor IBM software support, a PTF stands for program temporary fix. It is a programpackage that allows you to apply corrections to an operating system or anapplication.

IBM periodically creates PTFs to correct problems or potential problems foundwithin a particular IBM licensed program. PTFs are designed to replace one ormore objects in the licensed program. Generally, PTFs are incorporated in afuture release of the program.

A PTF corrects a problem reported by an Authorized Problem Analysis Report(APAR), which is a request for correction of a defect in a current release of anIBM-supplied program. More simply, a PTF is a “fix” for the AS/400 system.Customers receive this fix via electronic means from IBM.

Lotus does not provide OS/400, TCP/IP, and Java PTFs. They are provided byIBM software support.

Note: DB2 UDB for AS/400 is included in OS/400. PTFs for DB2 UDB for AS/400are OS/400 PTFs.

PTFs come with cover letters that state the exact procedures and circumstancesnecessary for application. You can refer to Basic System Operation,Administration, and Problem Handling, SC41-5206, for additional information onPTFs application.

5.2.2 Points to remember about PTFsKeep the following information in mind when installing PTFs:

• A PTF can be permanently or temporarily applied.

• Once permanently applied, PTFs can never be “backed off” elegantly.

The only ways to come back to the previous state are:

– To restore the program environment that was saved before the PTF waspermanently applied.

– To uninstall and reinstall the software release and all fixes validated for theprevious installation.

• When the OS/400 is upgraded, temporary PTFs are permanently applied.


This is also true for QMUs (Domino for AS/400 fixes), which have a PTFnumber on the AS/400. QMUs are described in 5.4.2, “Getting the latest fixespackage” on page 77.

5.2.3 More information about PTFsA single PTF is a programming solution that is created to correct a problem orpotential problem within a particular IBM licensed program. On occasion, PTFscontain new code enhancements within an IBM licensed program.

Some PTFs may need other PTFs to be loaded and applied. There are twodifferent types of dependencies, prerequisites, and corequisites.

5.2.3.1 Cumulative PTF packagesCumulative PTF packages are the primary method of performing preventivemaintenance on the OS/400 operating system and IBM licensed programs.Cumulative PTF packages contain recommended PTFs, and they are updated ona periodic basis. Cumulative PTF packages should be installed a minimum ofevery three to four months if the system environment is complex, the latesthardware or software releases are used, or major changes to the systemenvironment (new hardware or software) are planned.

CUMs are labeled according to the year, Julian date, and AS/400 operatingsystem software version. In the following designation, CUM C9230440, note thefollowing points:

• 9 represents the Year (99).• 230 represents the Julian Date minus the year.• 440 represents OS/400 V4R4M0.

5.2.3.2 Group PTFsA group PTF is a single PTF that can be ordered and provides a logical set ofPTFs affecting a specific function (for example database, Java, HIPERs, Y2K,and so on). When new PTFs affecting the function become available, group PTFsare dynamically updated. You can find a list of the group PTFs available forordering by linking to the Web site at: http://www.as400service.ibm.com

The AS/400 Group PTFs page is shown in Figure 35 on page 72.

Before QMU 5.0.1.02, AS/400 cumulative packages do not contain Domino forAS/400 QMUs.

Note

Chapter 5. How problems can be solved 71

Figure 35. AS/400 Group PTFs page

The page shown in Figure 35 can be accessed at:http://as400service.rochester.ibm.com/s_dir/sline003.NSF/

GroupPTFs?OpenView&view=GroupPTFs

5.2.3.3 Service packsA service pack is a collection of code fixes for AS/400 Client Access products,contained in a single AS/400 PTF that can be ordered. You can find service packinformation at the Web site: http://www.as400.ibm.com/clientaccess

This site may help you fix a problem affecting the Domino plug-in for AS/400Operations Navigator, a function provided in Client Access Express and ClientAccess.

Note: A service pack applies to the client portion of Client Access and is,therefore, not an AS/400 fix, although it is packed as a PTF.

5.2.3.4 Preventive Service Planning (PSP)Between releases of cumulative PTF packages, corrective single PTFs may becritical to your operations. PSPs contain information on all known hiper problemsthat may affect your operation and general installation information. Each PSPlisting contains a chart that lists all of the PSPs currently available.

You can access AS/400 Preventive Service Planning information at:http://as400service.ibm.com

At the site, click Tech Info & Databases->Prev Serv Planning - PSP.


5.2.4 Looking at installed PTFsTwo environments allow you to look at installed PTFs:

• The 5250 display interface• AS/400 Operations Navigator

5.2.4.1 Looking at installed PTFs from a 5250 displayTo determine the status of the PTFs for products that are installed and supportedon your system from a 5250 display, follow these steps:

1. Enter the Display Program Temporary Fix (DSPPTF) CL command.

2. Press F4 (Prompt).

3. Specify the licensed program number (for example: 5769SS1).

If you want to see the status for all the products on the system, specify *ALL forboth parameters, and press Enter until all the licensed programs aredisplayed.

The Display Program Temporary Fix (DSPPTF) display shows the status of allPTFs installed on the AS/400 system.

To determine the installed cumulative package, follow these steps:

1. Type DSPPTF 5769SS1

2. Look for the first PTF number that starts with TC9xxxx. The xxxx represent thefirst four numbers of the cumulative package identifier. For example, PTFTC99230 would represent cumulative package C9230440.

5.2.4.2 Looking at installed PTFs through AS/400 Operations NavigatorInstalled PTFs for an AS/400 software can be easily displayed through AS/400Operations Navigator. Figure 36 on page 74 shows the PTFs installed for Dominofor AS/400, which has the product number 5769-LNT.

To access the list for a software product, click the name of the system. Then clickConfiguration and Service->Fixes Inventory and then the product number. Thelist of PTFs appears in the right pane.

You may find PTFs that have been superseded. A superseded PTF is one thathas never been installed on the system, but shows up as superseded becausethe changes from this PTF have been included in a new PTF that has beeninstalled on the system. If the PTF had been temporarily applied and you installthe new PTF, it automatically and permanently applies the old PTF so the codefrom the new PTF is used, and the new PTF is now in a temporarily appliedstatus.

Note


Figure 36. Domino for AS/400 Release 5.0.1 PTFs displayed through AS/400 Operations Navigator

5.2.5 Ordering PTFsTo order individual PTFs, use the SNDPTFORD PTFID(nnnnnnn) command, wherennnnnnn is the number of the individual PTF you want to order. You can order upto 20 PTFs at a time.

When you receive PTFs electronically on your service support communicationsline, they are placed in a save file in the QGPL library. In most cases, the save filename is the PTF identifier prefixed with a Q. For example, the save file for PTFidentifier SFnnnnn is QSFnnnnn

A PTF may depend on other PTFs. There are two different types ofdependencies, prerequisites and corequisites. The prerequisite and corequisitePTFs are sent along with the PTFs order to make sure you have everything youneed to load and apply the PTF.

Cumulative PTF packages can be ordered electronically. To order a cumulativePTF package electronically, perform the following steps:

1. Identify your licensed program release level.

2. Type GO LICPGM on any command line, and press the Enter key.

3. Enter option 10 (Display installed licensed programs) on the Work withLicensed Programs display.

4. Press the F11 key.

The current version, release, and modification level is shown in the InstalledRelease column, where V is the version, R is the release, and M is themodification level.

5. Type SNDPTFORD SF99VRM, where VRM is the version, release, and modificationlevel you found in step 1.


For example, for Version 4 Release 4 Modification 0, type:

SNDPTFORD SF99440

6. Verify the shipping information on the Verify Contact Information display. If anyof the information is incorrect, change it, and press the Enter key.

7. Select option 1 (Send service request now) on the Select Reporting Optiondisplay.

This places the PTF order.

AS/400 Internet PTF download (iPTF)This facility allows you to select, order, and download AS/400 PTFs over theInternet.

5.3 Loading and applying PTFs

Loading a PTF copies it from a tape, CD-ROM, or save file onto the AS/400 system.This is done through the Load Program Temporary Fix (LODPTF) command. Afterbeing loaded, the PTF is known to the system, but no code change has happened.

Applying the PTFs actually changes the code in the software product. This is donethrough the Apply Program Temporary Fix (APYPTF) command.

To load and apply PTFs and cumulative packages, refer to Basic SystemOperation, Administration, and Problem Handling, SC41-5206, for instructions onapplying PTFs or cumulative packages. Read the “What are PTFs and Why Do INeed Them?” chapter, which includes these subtopics:

• AS/400 program maintenance strategy• Ordering PTFs and PTF information• Installing PTFs• Advanced PTF topics

5.4 Implementing the latest code for Domino for AS/400

A fix is the solution for a single problem. Every three months, Lotus gathers fixesspecific for a release in a package called a Quarterly Maintenance Update(QMU). A new release is called a Quarterly Maintenance Release (QMR).

Lotus support asks customers to first start implementing the latest code available,which means installing the latest QMR, and then the latest QMU for Domino forAS/400.

There are rare cases where manual action is required (as described in thecover letter) to activate the PTF after applying it. One example is the PTF forV4R2 and V4R3, which provides the Edit File (EDTF) command and theDisplay Stream File (DSPTSMF) command.

After applying the PTF, you have to run the Restore Object (RSTOBJ) commandbefore you can use any of the commands.

Note


If the fix is not available in a QMU or a QMR, there can be three situations:

• A hot fix has been validated and can be provided immediately to the customer.• A fix has been created, but needs further testing.• The fix needs to be created and escalation to the laboratory is required.

5.4.1 Getting the latest code for Domino for AS/400The latest code for Domino can be a complete release, such as Domino Releases4.6 and 5.0. Or it can be an intermediate release that is published as a QMR.

For Release 5.0, examples of QMRs are:

• Domino for AS/400 5.0.2a• Domino for AS/400 5.0.1

For Release 4.6, examples of QMRs are:

• Domino for AS/400 4.6.6b• Domino for AS/400 4.6.5• Domino for AS/400 4.6.4• Domino for AS/400 4.6.3• Domino for AS/400 4.6.2

5.4.1.1 Information on a QMRInformation on a new Domino release can be found in customer support technicalpapers available at: http://support.lotus.com/lshome.nsf

At the site, click Technical Notes->Notes/Domino, and then search for “Dominoand QMR”.

Figure 37 shows an example of release a notes document, titled Lotus Notes andLotus Domino Release 4.6.6 Quarterly Maintenance Release (QMR), found in thesearch results.

Domino for AS/400 4.6.7 is planned to be the last QMR for Release 4.6.

The current OS/400 release (V4R4) is the last release that will support Dominofor AS/400 R4.6.x. Any future releases of OS/400 will not support Domino forAS/400 4.6.x. To use Domino for AS/400 on an OS/400 version past V4R4,Release 4.6 users need to move to Release 5.0. When the next release ofOS/400 is available, Lotus will still support Domino for AS/400 R4.6.x on thecurrently supported releases. However, Lotus will not support Domino forAS/400 R4.6.x on OS/400 releases after V4R4.

Notes


Figure 37. Lotus Notes and Lotus Domino Release 4.6.6 QMR release notes

AS/400-specific functions new with a Domino for AS/400 QMR and PTFs requiredfor that QMR are described at: http://www.as400.ibm.com/domino/DomSupport.htm

5.4.1.2 Obtaining a QMRA Lotus support contract is required to access QMR. Lotus Domino for AS/400QMR are available for customers from their local authorized Lotus resellers. If thecustomer has a paid software subscription (maintenance), the Domino for AS/400CD is mailed.

5.4.1.3 Installing a QMRUnless specially stated, a QMR can be installed over the existing code. However,we strongly recommend that you save the installed release and each server’senvironment.

The installation of a Domino for AS/400 new release may require the customer tofirst install and apply one or more PTFs on their AS/400 system.

See the Web site at: http://www.as400.ibm.com/domino/DomSupport.htm

5.4.2 Getting the latest fixes packageA QMU contains fixes for Domino for AS/400 grouped in a single package. Theobjective of a QMU is to fix specific issues related to a new release or a QMR,before a new QMR or release is made available.

On the AS/400 system, a QMU is installed as a set of one or more PTFs. Moreinformation about PTFs is provided in 5.2, “Implementing AS/400 software fixes”on page 70.


QMUs may be downloaded from the Web or obtained as AS/400 PTFs (startingwith QMU 5.0.1.02).

5.4.2.1 Getting a QMU from the Domino for AS/400 Support Web pageTo download QMUs from the Domino for AS/400 Support Web site, follow thesesteps:

1. Access Lotus Domino for AS/400: Support Information at:http://www.as400.ibm.com/domino/DomSupport.htm

2. Click Domino for AS/400 QMUs.

3. Choose the appropriate upgrade.

4. Follow the instructions provided in the QMU installation notice.

Usually, a QMU is downloaded as a binary file that you will copy to the AS/400 viaFTP or another file transfer method. It is an AS/400 save file (type *FILE andattribute SAVF), and it is not a self-extracting executable, even though it has a“.exe” extension. Do not attempt to run this file on a PC, or you will receive theerror message: Program too big to fit into memory.

The corresponding PTF will be loaded from the save file and applied on product5769-LNT (product number for Lotus Domino for AS/400 on the AS/400 system).

5.4.2.2 Obtaining a QMU as a PTFStarting with QMU 5.0.1.02, Domino QMUs for AS/400 are also distributed asAS/400 PTFs. They can be obtained through the IBM AS/400 PTF distributionprocess.

The IBM AS/400 PTF distribution process allows PTFs to be ordered individuallyor obtained as part of PTF Cumulative Packages.

Example of a QMU delivered as a PTFV4R3 Cumulative Package C9341430 contains the PTF SF58802, which is LotusQMU 5.0.1.02. The following paragraphs are excerpt from the instructions relatedto installing the PTF:

If you have Version 5.0.1 of Domino for AS/400 installed and have any interimfixes (also known as hot fixes) for Domino on your system, you must removethese fixes from your AS/400 prior to installing the cumulative PTF package.Failure to do so will result in permanent application of the existing hot fixes forDomino.

You may use the OS/400 CL command DSPPTF LICPGM(5769LNT) to checkwhether any hot fixes are currently installed on your system for Domino. If nofixes are applied, the message will read No PTF activity exists for product

5769LNT.

If any hot fixes exist on the system, you must use the Remove ProgramTemporary Fix (RMVPTF) command to remove them from the Domino systembefore applying the cumulative PTF. The Domino servers cannot be activeduring this removal.

5.4.2.3 Upgrading OS/400 and QMUsSpecial attention is necessary when upgrading OS/400, for example from OS/400V4R3 to V4R4. When upgrading OS/400, any PTF is applied permanently. Thisapplies also to any Domino QMU or hot fix, as it is implemented as a PTF on the


AS/400 system. Applying a PTF permanently means it cannot be removed in caseit turn out to be erroneous.

Once the PTF is installed permanently, it can only be removed by reinstalling theproduct and available fixes (PTFs). In the situation where Domino for AS/400 isconcerned, the product has to be removed and reinstalled, and the latest QMUwould have to be installed again.

5.4.3 Hot fixesA hot fix can be described as a fix that has been created for a specific release buthas not been included in a QMU (for the same release) or a QMR (more recentrelease than the release implemented).

Hot fixes are test programs provided by Lotus development to customers for themto test a specific fix. A hot fix is similar to what Lotus calls a “test DLL” in theWindows environment.

Hot fixes are both customer-specific and problem-specific. They are notdistributed to other customers without development and Lotus Supportinvolvement. Upon verification that the hot fix works in the customer environment,the correction contained in a hot fix is submitted to the next appropriate QuarterlyMaintenance Update or Quarterly Maintenance Release.

It is important to note that hot fixes are not fully regression tested and are “hot” offof the developer's machine. Testing these hot fixes is usually limited to testingdone by the developer and perhaps limited testing by the Quality Engineeringteam or a person from Lotus Support. Hot fixes are not intended for productionuse and should not be distributed throughout an organization.

When a hot fix is required for a customer installation, the hot fix is directly sent tothe customer.

By nature, the hot fix life finishes when the fix is included in a QMU or a QMR.These are the methods to determine the hot fix “end of life”:

• To know if a hot fix is integrated in a QMU, look for the SPR numberassociated with the hot fix in the QMU description.

• To know if a hot fix is integrated in a QMR, look for the SPR numberassociated with the hot fix in the QMR description, in the release notes fixeslist.

Lotus Support recommends that you have the latest Domino for AS/400 codelevel (latest QMR and QMU), and test the environment before upgrading theOS/400.

Any fix (or hot fix) that has not been completely tested or validated should beremoved before upgrading OS/400.

Important


5.4.3.1 Special considerations about hot fixesLotus support recommends that you always remove any installed hot fix beforeinstalling a new QMU or QMR.

If the QMU is applied over a hot fix, and the QMU supersedes the hot fix, the hotfix will be permanently applied. This should cause no functional problem.However, the hot fix cannot be removed without a product remove and reinstall. Ifthere is a problem with the hot fix, the product needs to be reinstalled to removethe hot fix. If there is a problem with the QMU fix, the QMU needs to be removed,but the hot fix remains permanently applied.

If the QMU is applied over a hot fix, and the QMU does not supersede the hot fix,the hot fix is not permanently applied. However, if the hot fix involved an updateto a libnotes service program, removing the hot fix may reapply a saved versionof the library libnotes that would move the QMU down-level. The user wouldneed to reapply the QMU PTF to correct the down-level situation.

A problem does not necessarily occur to every customer that applies a QMU overa hot fix. It would appear to be only those that have needed to remove a hot fix.

Cumulative PTF and hot fixesTo avoid the permanent application of a hot fix, remove it prior to installing acumulative PTF.

Determine if any of the PTFs are hot fixes, as explained earlier in this section,and use the Remove Program Temporary Fix (RMVPTF) command to remove themfrom the system before applying the cumulative PTF package. Prior QMU PTFsmay remain installed. The Domino servers must be stopped during the removal ofhot fix PTFs.

Upgrading OS/400 and hot fixesThe considerations described in 5.4.2.3, “Upgrading OS/400 and QMUs” on page78, are also true for hot fixes.

On the AS/400 system, all PTFs are created first as test fixes. The namingconvention for the test fix begins with SA, and not SF. This is the currentconvention for Domino for AS/400 hot fixes. It helps you to recognize a hot fixwhen displaying all Domino for AS/400 PTFs. However, some Domino forAS/400 hot fixes have been delivered in the past with a name starting with SF.

You have to look at the details of the PTF to determine if it is a hot fix or a QMU.Display the PTF symptom string to determine if the applied fix is a hot fix or aQMU. QMU PTFs contain such text as “DOMINO QMU v.r.m” for the symptomstring, where hot fix PTFs describe a particular problem.

Note


Chapter 6. Domino administration tools

Domino administration commands, such as FIXUP, UPDALL, or COMPACT, canbe used to solve problems with corrupted databases.

Domino Release 5 supports transaction logging and recovery. With this featureenabled, the server captures database changes and writes them to thetransaction log. Then, if a system or media failure occurs, you can use thetransaction log and a third-party backup utility to recover your databases.

Domino provides several tools to help you troubleshoot problems. Most of thetools are available through the Domino Administrator.

Some debug parameters can be implemented in the NOTES.INI file and help youto better understand a problem. Caution is necessary when using debugparameters.

6.1 Domino administration commands

Depending on the problem encountered, the following Domino administrationcommands can be used to solve problems with corrupted databases:

• FIXUP• UPDALL• COMPACT

However, they cannot be considered as a complete solution for all problems.

6.1.1 FIXUPThe FIXUP command reads every item in a database (data notes, design notes,view indexes, etc.) and verifies that it is consistent with what is expected for thattype of item. If the item is found to be inconsistent, it is either repaired (this doesnot happen often) or purged from the database.

Views are also verified if they are found to be inconsistent or they are marked forrebuild. Note that FIXUP does not rebuild a damaged view. It only marks it forrebuild. Only UPDALL -R rebuilds the view. If you have a view problem, you needto use UPDALL -R, not FIXUP.

FIXUP tests only the structure of the documents, not the data in the fields. If thecontent of the field is corrupt, the FIXUP runs without error.

Manually loading FIXUP from the server console or administration panel rebuildsany corrupted views or folders. Corrupted views and folders are not fixed whenFIXUP runs automatically at server startup or when users open corrupteddatabases.

FIXUP cannot run on open databases, and databases cannot be opened whenFIXUP is running on them. However, when FIXUP encounters an open database(for example, LOG.NSF, MAIL.BOX, or NAMES.NSF), FIXUP reports this to thelog file.


6.1.1.1 FIXUP exampleThe following example shows an abnormal server shutdown. If a server shutsdown abnormally while data is written to a database, that database can becomecorrupted, and the Domino for AS/400 server can’t be restarted. Sometimes theNSD file may show which database is the culprit.

To repair a damaged database, use the following command:

Load FIXUP Database.NSF

FIXUP will not run on a database that is open (in use).

6.1.1.2 FIXUP optionsTable 4 was copied from the Domino Release 5 Administrator Help database. Itdescribes the options you can use with FIXUP. The first column lists the optionsas they appear when you run FIXUP using the FIXUP tool or the Task - Start toolin the Domino Administrator. The second column lists the equivalentcommand-line options that you use when you run FIXUP using a consolecommand or using a Program document.

Table 4. FIXUP options

FIXUP options inFIXUP tool andTask - Start tool

Commandline

equivalent

Description

FIXUP all databasesFIXUP only thisdatabase or folder

database path “FIXUP only this database or folder” runs FIXUPonly on a specified database or all databases ina specified folder. To run FIXUP on a databasein the Domino data folder, enter the file name, forexample SALES.NSF. To run FIXUP on adatabase or databases in folder within the datafolder, enter the path relative to the data folder.For example, to run FIXUP on all databases inthe DATA\SALES folder, specify SALES“FIXUP all databases”, or no command linedatabase path runs FIXUP on all databases onthe server.Note: To specify databases or folders to run onusing the FIXUP tool, select the databases orfolders.

Report all processeddatabases to log file

-L Reports to the log file every database FIXUPopens and checks for corruption. Without thisargument, FIXUP logs only actual problemsencountered.

Scan only since lastFIXUP

-i When you run FIXUP on a specific database,FIXUP checks only documents modified sinceFIXUP last ran. Without this option, FIXUPchecks all documents.

Scan all documents -F When you run FIXUP on all databases, FIXUPchecks all documents in the databases. Withoutthis option, FIXUP checks only documentsmodified since it last ran.Note: To specify this option using the FIXUPtool, deselect Scan only since last FIXUP.


6.1.2 UPDALL -RThe UPDATE and UPDALL tasks keep view indexes and full-text indexesup-to-date. This sections explains each in more detail.

6.1.2.1 UPDATEUPATE is loaded at server startup by default and runs continually, checking itswork queue for views and folders that require updating. When a view or folderchange is recorded in the queue, UPDATE waits approximately 15 minutes beforeupdating all view indexes in the database so that the update can include anyother database changes made during the 15-minute period. After updating viewindexes in a database, it updates all databases that have full-text search indexesset for immediate or hourly updates.

When UPDATE encounters a corrupted view index or full-text index, it rebuildsthe view index or full-text index in an attempt to correct the problem. This meansit deletes the view index or full-text index and rebuilds it.

6.1.2.2 UPDALLUPDALL is similar to UPDATE, but it doesn't run continually or work from aqueue. Instead, you run UPDALL as needed. Like UPDATE, UPDALL rebuilds allcorrupted view indexes and full-text search indexes that it encounters.

Perform quick FIXUP -Q Causes FIXUP to check documents morequickly, but less thoroughly. Without this option,FIXUP checks documents thoroughly.

Exclude views (faster) -V Prevents FIXUP from running on views. Thisoption reduces the time it takes FIXUP to run.Use if view corruption isn't a problem.

Don't purge corrupteddocuments

-N Prevents FIXUP from purging corrupteddocuments. The next time FIXUP runs a useropens the database, FIXUP must check thedatabase again. Use this option to salvage datain documents if the corruption is minor or if thereare no replicas of the database.

Optimize user unreadlists

-U Reverts ID tables in a database to the previousrelease format. Don't select this option unlessCustomer Support recommends doing so.

FIXUP transactionlogged databases

-J Causes FIXUP to run on Release 5 databasesthat are enabled for transaction logging. Withoutthis option, FIXUP generally doesn't run onlogged databases.If you are using a backup utility certified forDomino Release 5, it's important that youschedule a full backup of the database as soonas possible after FIXUP finishes.

FIXUP options inFIXUP tool andTask - Start tool

Commandline

equivalent

Description

Chapter 6. Domino administration tools 83

You can specify options when you run UPDALL, but without them, UPDALLupdates any view indexes or full-text search indexes on the server that needsupdating.

The Notes log is one of the best troubleshooting tools at your disposal. Obtain theLog file from the customer to see if there are any view corruptions. For example,“View Index is Corrupt” is a common error that appears in the log.

UPDALL -R, which rebuilds full-text indexes and all used views, can solvecorruption problems with a specific database. However, this command consumesa lot of resources and should not be used without experience.

6.1.2.3 UPDALL -R exampleThe following message is an example of an error that appears when sending amail message. You attempt to send a message and receive the error messageUser Name Not Found in Name and Address Book.

When you check the Domino directory (or Notes Address Book), you find that, infact, the user name is listed. Based on this, you can suspect that the view usedby the Router ($Users) is out of date and, therefore, determine that UPDALL -Rneeds to be run on the $Users view.

You can run UPDALL -R on that view alone, rather than running it on the entireDomino directory (or Notes Address Book), by using the -T parameter.

To run UPDALL -R on a single view, use the command:

UPDALL DATABASE NAME -T "viewname" -R

For example, to run UPDALL -R on the view $Users of the Domino directory(database NAMES.NSF), enter:

Load UPDALL names.nsf -T "($Users)" -R

UPDALL optionsDifferent sets of options are available for the Domino administration commandUPDALL. Each set is shown in the following tables.

To save disk space, UPDALL also purges deletion stubs from databases anddiscards view indexes for views that have been unused for 45 days, unless thedatabase designer has specified different criteria for discarding view indexes.

Use the NOTES.INI setting Default_Index_Lifetime_Days to change whenUPDALL discards unused view indexes.

Note


Table 5 describes the basic options.

Table 5. UPDALL: Basic options

Table 6 describes the update options.

Table 6. UPDALL: Update options

Table 7 describes the rebuild options.

Table 7. UPDALL: Rebuild options

Option in Task -Start tool

Command-lineoption

Description

Index all databasesIndex only thisdatabase or folder

database path “Only this database” updates only the specifieddatabase. To update a database in the Dominodata folder, enter the file name, for exampleSALES.NSF. To update databases in a folderwithin the data folder, specify the database pathrelative to the data folder, for exampleDOC\README.NSF.“Index all databases” (or no database path)updates all databases on the server.

Update this view only database -Tviewtitle

Updates a specific view in a database. Use, forexample, with -R to solve corruption problems.


Command-lineoption

Description

Update: All builtviews

-V Updates built views and does not update full-textindexes.

Update: Full textindexes

-F Updates full-text indexes and does not updateviews.

Update: Full textindexes: Only thosewith frequency setto: Immediate

-H Updates full-text indexes assigned “Immediate”as an update frequency.

Update: Full textindexes: Only thosewith frequency setto: Immediate orHourly

-M Updates full-text indexes assigned “Immediate”or “Hourly” as an update frequency.

Update: Full textindexes: Only thosewith frequency setto: Immediate orHourly or Daily

-L Updates full-text indexes assigned “Immediate”,“Hourly”, or “Daily” as an update frequency.


Command-lineoption

Description

Rebuild: Full-textindexes only

-X Rebuilds full-text indexes and does not rebuildviews. Use to rebuild full-text indexes that arecorrupted.

Rebuild: Full-textindexes andadditionally: All usedviews

-R Rebuilds full-text indexes and all used views.Using this option is resource-intensive, so use itas a last resort to solve corruption problems witha specific database.


Table 8 describes the search site options.

Table 8. UPDALL: Search site option

6.1.3 COMPACTWhen documents and attachments are deleted from a database, Domino tries toreuse the unused space, rather than immediately reducing the file size.Sometimes, Domino won't be able to reuse the space or, due to fragmentation, itcan't reuse the space effectively until you compact the database. The deletionstabs are still present and only using the COMPACT command can remove them.

There are three styles of compacting used in Release 5:

• In-place compacting with space recovery• In-place compacting with space recovery and reduction in file size• Copy-style compacting

For more information, see the document “Compacting databases” in the DominoRelease 5 Administration Help database.

This utility rarely fixes a problem inside a database because, in order forCOMPACT to work, it must be able to read these items. If it cannot read the item,it will fail to compact the database and generate an error message.

By default, compacting converts Release 4 databases to the Release 5 file formator on-disk structure (ODS). To prevent this automatic conversion, use the -Roption with the load COMPACT command. To prevent a database that you'recreating from using the Release 5 file format and to prevent COMPACT fromconverting the database to the Release 5 file format, give the database theextension .NS4. Because there are many advantages to using Release 5 formatdatabases, we recommend that you convert the Release 4 file format databasesto the Release 5 file format.

6.1.3.1 COMPACT examplesThe following examples demonstrate situations where COMPACT can be used tofix a database problem.

• Example 1: Used Space Below 80%: A database is performing very slowly.Check the usage of the database by selecting File->Database->Properties,

Rebuild: Full-textindexes andadditionally: Allunused views

database -C Rebuilds unused views and a full-text index in adatabase. Requires you to specify a database.


Command-lineoption

Description

Update databaseconfigurations:Incremental

-A Incrementally updates search-site databaseconfigurations for Release 4 search-sitedatabases.

Update databaseconfigurations: Full

-B Does a full update of search-site databaseconfigurations for Release 4 search-sitedatabases.


Command-lineoption

Description


and clicking the Info tab. Click the %Used button, which shows thepercentage of used space on the database.

If this value is below 80%, you should compact the database using thefollowing command:

Load COMPACT Database.nsf

• Example 2: UNK Table is Full: Obtain the Log and determine whether thereare errors such as “UNK Table is Full” (where UNK is Unique Key Table).

COMPACT throws away the UNK table and rebuilds it. Running this taskwould solve the problem.

• Example 3: Users Removed from Address Book: A company has just soldoff a major division, and the administrator has just removed one tenth of theemployees from the Domino directory. After work hours or on a weekend,perform the following steps:

1. Replicate the changes made to the NAB to the other servers, using forcedreplication.

2. In all replica copies of the NAB (on all servers), change the value in theRemove Documents Not Modified in the Last n Days field to a lower value,such as 10. By default, this setting is 90 days.

3. Stop the replication task, and shut down the server.

4. Run COMPACT from the operating system.

5. Change the Space Saver setting back to the original value, and thenre-enable replication between the other servers.

6.1.4 When to run FIXUP, UPDALL -R, and COMPACTFIXUP, UPDALL -R, and COMPACT are tools that are often used unnecessarily.In the course of troubleshooting an issue, if you determine there is a problemwithin a databases, for example corruption or a faulty index, you can use theappropriate tool to try to correct the problem. These utilities are not universalfixes for all ailments.

The use of these tools can be time consuming, cause downtime of a Notesserver, can be resource intensive, and can have a tremendous impact on aproduction environment. Using these tools unnecessarily or incorrectly can leavea production server unavailable for a lengthy period. Therefore, you should usethem only when you are certain they offer value.

To determine the appropriate time to use FIXUP, UPDALL -R, or COMPACT, youshould first determine why the issue has occurred or why it is recurring. Running

While a server is active, the Domino directory names.nsf database is in use.One of the Domino R5 enhancements is the support for the compacting ofdatabases even when they are in use. After creating a new server, we foundthat although the on-disk structure (ODS) of the NAMES.NSF file is in the R5format, the first compact of names.nsf must be done with the server inactiveusing the instructions in this section. After this, names.nsf can be compactedwhen the server is active.

Note


one or more of these tools may fix the immediate problem without addressing theroot cause. Problems with Domino databases could be the result of:

• Inappropriate server shutdown• Power surges or power outages• Memory or CPU subsystem problems• Operating system or memory leaks of some program in memory• Protocol transmission failures• Bugs in the product

Databases containing corrupted views, folders, or documents are the mostcommon cause of server crashes. Database corruption is most frequently causedby improper shutdown of the operating system because of system reboot, systemcrash, power failure, or use of improper shutdown procedures or improperdatabase access by an API program.

6.1.5 How long FIXUP, UPDALL, or COMPACT take to runThere is no specific formula to calculate the run time of these tasks. It dependson the number of updates that have been made to the database, on the numberand size of views, and whether a database has been customized by the user.

It is not unusual for a process to take four to five hours on a large database,depending on what you are doing to the database and how bad its condition is.

6.1.6 Running Domino administration commands inactive serverHow do you run the FIXUP, UPDALL, or COMPACT task on the AS/400 systemwithout the Domino server running?

FIXUP, UPDALL, or COMPACT are executable programs that may be run on aDomino server when the Domino server is not running.

However, to do this on the AS/400 system, certain other commands must also berun to ensure that the environment is correct. Additionally, the FIXUP, UPDALL,and COMPACT tools must be run under the user profile QNOTES.

By design, QNOTES “owns” all Domino for AS/400 jobs. Also by design, theQNOTES user profile may not (and should not) be enabled for signon by anindividual. In fact, the initial program to run in the user profile (*USRPRF) objectis *SIGNOFF.

Since you cannot sign on with the user profile QNOTES, you may submit thenecessary commands in one of two ways:

• Submit a batch job with the AS/400 SBMJOB command.

• Create a Control Language (CL) program with the necessary commands andthen call the program.

Domino for the AS/400 system is integrated with the AS/400 security system.Therefore, there are some unique requirements for running a Domino taskoutside the server (that is, not through the load command in the Domino console).

Whether you are submitting the batch job or creating a CL program as statedabove, you need to perform the following tasks:


1. Find and use the job description (*JOBD) that is used for the server inquestion. You will find this job description in library QUSRNOTES. It has thesame name as the subsystem in which the server runs. For example, if theserver called “Domsvr5” runs in subsystem DOMINO02, the job description(*JOBD) is also called DOMINO02.

Run the WRKOBJ command and press F4. Enter *ALL for the Object, QUSRNOTES forthe library, and *JOBD for the object type. Then, press Enter.

This displays all the job descriptions in the QUSRNOTES library. Find the jobdescription for the subsystem in which your Domino server runs (you onlyneed to find the name at this point).

2. You must set two environment variables: Path and NotesPartition (the secondvariable is needed only with Release 4.6).

“Path” must include:/QIBM/UserData/LOTUS/NOTES:/QIBM/ProdData/LOTUS/NOTES:/(theDomino server data directory path here, excluding the parentheses)'

An example of your path would be /lotus/domino/domsvr5. This should specifythe path to the Domino server data directory for this server.

To add an environment variable, type ADDENVVAR, and then press F4. Thevariable is PATH. The initial value would be:

'/QIBM/UserData/LOTUS/NOTES:/QIBM/PRODDATA/LOTUS/NOTES:/(your path here excluding the parentheses)'

You must use uppercase and lowercase as shown above. Additionally, youmay need to use the CHGENVVAR command if there is already a valuespecified here. The difference here is that one is to add and the other is tochange the variable depending on what you already specified.

“NotesPartition” (required only with Domino for AS/400 Release 4.6) mustinclude the partition number of Domino as found in the DOMINO_SERVERSfile (or LOTUS_SERVERS file, if you are using Domino for AS/400 Release5.0.2 or a later release).

Use the Display File (DSPF) command, if you are using OS/400 V4R4, or theDisplay Stream File (DSPSTMF) command, if you are using OS/400 V4R2 orV4R3, to display the file. Both commands are described in 7.1, “Displaying orediting a stream file from a 5250 display” on page 117.

Type ADDENVVAR, and press F4. The variable is NOTESPARTITION, tab. Theinitial value is the two digit partition number, return.

A message appears at the bottom of the screen (5250 green screen) tellingyou that the variable was set or changed.

3. Run the job under the QNOTES user profile and under the Domino server jobdescription.

4. Make sure the current IFS directory is the appropriate data directory.

As mentioned previously, you must set up the correct environment variables. If itis always the same, you can set it up in the initial program of the user invoking thecommand. The initial user program is specified in the user profile. It behaves likea logon script. If you want to specify it only for a particular job, you should write aCL program that looks similar to the example shown in Figure 38 on page 90.


Figure 38. CL program to submit FIXUP

Lines 2 through 5 in Figure 38 set the PATH environment variable. Notice thatthere is a list specified here. The elements of the list are separated by colons.Actually, we indicate these three directories:

• /QIBM/PRODDATA/LOTUS/NOTES• /QIBM/USERDATA/LOTUS/NOTES• /LOTUS/DOMINO/DOMSVR5 (the Domino server data directory)

Line 6 sets the PARTITION environment variable to 02 (the server’s partition). Asmentioned before, this variable is required only in Release 4.6.

Line 7 runs the CHGCURDIR command to set the current directory to the datadirectory.

Lines 8 through 12 run the FIXUP command using the Submit Job (SBMJOB)command under the QNOTES user profile with the job description (DOMINO02)for the Domino server. The Copy environment variables (CPYENVVAR)parameter must be set to *YES.

If you need to specify any arguments to run with FIXUP, COMPACT, or UPDALL,follow the example given, but add the -L argument just after the word PARM, forexample:

PARM('-L’ ‘/lotus/domino/domsvr5/Names.nsf')

Columns . . . : 1 71 Edit CHRISF/QCLSRCSEU==> DOMCMDSFMT ** ...+... 1 ...+... 2 ...+... 3 ...+... 4 ...+... 5 ...+... 6 ...+... 7

*************** Beginning of data *************************************0001.00 PGM0002.00 ADDENVVAR ENVVAR(PATH) +0003.00 VALUE('/QIBM/PRODDATA/LOTUS/NOTES: +0004.00 /QIBM/USERDATA/LOTUS/NOTES:/lotus/domino/domsvr5') +0005.00 REPLACE(*YES)0006.00 ADDENVVAR ENVVAR(NOTESPARTITION) VALUE('02') REPLACE(*YES)0007.00 CHGCURDIR DIR('/lotus/domino/domsvr5')0008.00 SBMJOB CMD(CALL PGM(QNOTES/FIXUP) +0009.00 PARM('/lotus/domino/domsvr5/names.nsf')) USER(QNOTES) +0010.00 CURLIB(*USRPRF) INLLIBL(*JOBD) +0011.00 JOBD(QUSRNOTES/DOMINO02) +0012.00 CPYENVVAR(*YES)0013.00 ENDPGM

****************** End of data ****************************************

F3=Exit F4=Prompt F5=Refresh F9=Retrieve F10=Cursor F11=ToggleF16=Repeat find F17=Repeat change F24=More keys

(C) COPYRIGHT IBM CORP. 1981, 1999.

Do not forget to specify the parameter REPLACE(*YES), as shown on line 5. If youdo not do so, and an environment variable of the same name already exists, itis not replaced by the new value. The parameter is in the additional parametersand its default value is set to *NO. To change it, press F10.

Attention


The argument should be located before the path, and must be surrounded bysingle quotes.

To check for errors, complete the following tasks:

1. Enter WRKUSRJOB or WRKSPLF and press F4. Then enter your user name. Only ifthere are errors you will find any output here. You can press F11 to see thedates and times.

2. At the Domino server console, you can follow the progress of your command.Enter the WRKDOMSVR command, and select option 8.

3. After the job has ended, the following steps can be used to check, forexample, if FIXUP and UPDALL -R ran successfully:

a. On the AS/400 command line, enter WRKSBMJOBb. Look under the User column for QNOTES.c. Type 8 next to that user under the Option column, and then press Enter.d. Type 5 next to that user under the Option column, and then press Enter.e. Look through the display for completion or errors.

If an error occurs, such as “Unable to locate Notes executable directory”, makesure that PATH has been typed in capital letters (it is case-sensitive). Customersoften overlook this. This applies to all environment variables.

6.1.7 Alternative method for running FIXUP and UPDALLThere is an alternative method for running the FIXUP and UPDALL commands.

In this case, the server starts, but it is necessary to run FIXUP and UPDALL -Ron the Domino for AS/400 entire server with the equivalent of the server beingcompletely down. The NAMES.NSF and ADMIN4.NSF need to be givenconsistency checks and re-indexed without any interference. Other databases,such as mail files, may need to be checked for corruption without interruption aswell.

For any release up to Release 5.0.1, enter the following command to find thepartition number for your Domino server:

DSPF STMF(‘/QIBM/UserData/LOTUS/NOTES/DOMINO_SERVERS’)

For Release 5.0.2, enter:

DSPF STMF(‘/QIBM/UserData/LOTUS/LOTUS_SERVERS’)

More information on the Display File (DSPF) command, which should be usedwith OS/400 V4R4, or the Display Stream File (DSPSTMF) command, whichshould be used with OS/400 V4R2 and V4R3, can be found in 7.1, “Displayingor editing a stream file from a 5250 display” on page 117.

Do not change anything in those files, because then they become corrupted,and the WRKDOMSVR does not work anymore.

To find the job description for your Domino server, enter the WRKOBJ commandand press F4. Enter *ALL for the object, QUSRNOTES for the library, and *JOBD forthe object type.

Tip


The correct method to run FIXUP and UPDALL -R in this manner is as follows:

1. End the Domino server.

2. Rename NAMES.NSF (for example, NAMES2.NSF) to a temporary name. Dothe same for ADMIN4.NSF (for example, ADMIN4B.NSF). Enter the followingcommand:

wrklnk ('domino/as0656r2/data')

Type 5 for next level and then option 7 for rename. Press F4 to fill in the newobject name, and press Enter. See Figure 39 and Figure 40.

Figure 39. Renaming admin4.nsf

Work with Object Links

Directory . . . . : /domino/as0656r2/data

Type options, press Enter.3=Copy 4=Remove 5=Next level 7=Rename 8=Display attributes11=Change current directory ...

Opt Object link Type Attribute Text7 admin4.nsf STMF

admin4.ntf STMFalog4.ntf STMFarchlg50.ntf STMFbilling.ntf STMFbinary.gif STMFbrowser.cnf STMFbusytime.nsf STMFbusytime.ntf STMF

More...Parameters or command===>F3=Exit F4=Prompt F5=Refresh F9=Retrieve F12=Cancel F17=Position toF22=Display entire field F23=More options


Figure 40. New object admin4b.nsf

3. Use the CPY command to copy both the renamed NAMES.NSF and theADMIN4.NSF files. Select option 3 from Work with Object Links screen, andpress F4. Fill in directory path and object name as shown in Figure 41. Tocheck authority, press F23 (More options) and select option 9 to work withauthority. Verify that the two new files have QNOTES ownership. Use theChange Owner (CHGOWN) command if they do not.

Figure 41. Copy Object

Rename Object (RNM)

Type choices, press Enter.

Object . . . . . . . . . . . . . > '/domino/as0656r2/data/admin4.nsf'

New object . . . . . . . . . . . admin4b.nsf

...

BottomF3=Exit F4=Prompt F5=Refresh F12=Cancel F13=How to use this displayF24=More keys

Copy Object (CPY)


Object . . . . . . . . . . . . . > '/domino/as0656r2/data/namesb.nsf'

To directory . . . . . . . . . . '.'

To object . . . . . . . . . . . > '/domino/as0656r2/data/names.nsf'

Symbolic link . . . . . . . . . *NO *NO, *YESFrom Code Page . . . . . . . . . *OBJ 1-32767, *OBJ, *PCASCIITo Code Page . . . . . . . . . . *OBJ 1-32767, *OBJ, *CALC...Data Format . . . . . . . . . . *BINARY *BINARY, *TEXT



4. Start the Domino server. When startup is complete, end all tasks exceptSERVER and QNNINSTS (if present). Use the TELL <task> Q command. Tasksare listed on the WRKACTJOB list. Verify that nothing but SERVER andQNNINSTS (if originally present) are running. It may desirable to use the drop

all command from the Domino console to end all user sessions at this point. Itmay also be useful to end the Domino port 1352 to keep anyone else fromsigning on.

5. Run LOAD FIXUP. Let it run to completion.

6. Run LOAD UPDALL -R.

7. When complete end the server.

8. Delete NAMES.NSF and ADMIN4.NSF which are the temporary copies.

9. Again, rename the files renamed in step 2 (for example, NAMES2.NSF andADMIN4B.NSF) back to NAMES.NSF and ADMIN4.NSF.

10.Restart the server.

6.2 Transaction logging

Domino Release 5 supports transaction logging and recovery. With this featureenabled, a transactional log provides a sequential file to which everybody writes.

A database operation will complete successfully or unsuccessfully. When thatoperation completes, and you have received a response, it needs to beguaranteed that the operation is recorded on disk, not just in memory.Transactional logging is what records the operation you performed.

6.2.1 The database instance ID (DBIID)When you enable transaction logging, Domino assigns a database instance ID(DBIID) to each Domino Release 5 database. When Domino records atransaction in the log, it includes the DBIID. During recovery, Domino uses theDBIID to match transactions to databases.

Some database maintenance activities, such as compaction with options, causeDomino to assign a new DBIID to a database. From that point forward, all newtransactions recorded in the log use the new DBIID. However, any oldtransactions still have the old DBIID and no longer match the database's newDBIID. As a result, Domino cannot restore these old transactions to the database.

To avoid losing data, you should immediately perform a full database backupwhenever a database receives a new DBIID. When you perform this backup, you

Transaction logging works with databases in Domino Release 5 format, but notwith databases that use formats from earlier releases. After you enabletransaction logging, all databases in Release 5 format are automaticallylogged. To check database formats, select the Files tab in DominoAdministrator.

You can run the Compact task on your server to upgrade your databases to thenew R5 format.

Important


capture all the database transactions up until that point and ensure that Dominoneeds only the new transactions (with the new DBIID) to restore the database.

Domino assigns a new DBIID to Domino Release 5 databases when:

• You enable transaction logging for the first time.

• You run the Compact server task with any options (for example, the option toreduce file size).

• You run the Fixup task on corrupt databases.

• You move a Domino Release 5 database from one logged server to anotherlogged server or from an unlogged server to a logged server.

6.2.2 Setting up transaction loggingTo set up transaction, follow the process explained here:

1. Ensure that all databases to be logged reside in the Domino data directory,either at the root or in subdirectories.

2. From the Domino Administrator, click the Configuration tab.

3. In the Use Directory on field, choose the server's Domino Directory.

4. Click Server Configuration->Current Server Document.

5. Click the Transactional Logging tab.

6. Complete the fields, as described in Table 9, and then save the document.

Figure 42 on page 97 shows you an example of how the server document forenabling Transaction logging may look.

Table 9. Transaction logging settings

Field Enter

Transactional Logging Choose Enabled. The default is Disabled.

Log path Path name location of the transaction log.The default path name is \LOGDIR in the Domino data directory,although we strongly recommend that you store the log on aseparate, mirrored device, such as a Redundant Array ofIndependent Disks (RAID) level 0 or 1 device with a dedicatedcontroller. On the AS/400 system, it would be an auxiliary storagepool, also known as ASP.The separate device should have at least 1 GB of disk space forthe transaction log.

Maximum log space The maximum size, in MB, for the transaction log. The default is192 MB. Maximum is 4096 MB (4 GB).Domino formats at least three and up to 64 log files, dependingon the maximum log space you allocate.

The DBIID has no relation to the REPLICAID (replica ID) or DBID (databaseID). Changing the log path or maximum log size (after initial set up and use)does not trigger a DBIID change.

Note


Use all available spaceon log device

Choose one:-Yes to use all available space on the device for the transactionlog. This is not recommended on the AS/400 platform if noseparate device (auxiliary storage pool) is used.-No to use the default or specified value in the Maximum logspace field.

Automatic fixup ofcorrupt databases

Choose one:-Enabled (default). If a database is corrupt and Domino cannotuse the transaction log to recover it, Domino runs the Fixup task,assigns a new DBIID, and notifies the administrator that a newdatabase backup is required.-Disabled. Domino does not run the Fixup task automatically andnotifies the administrator to run the Fixup task with the -Jparameter on corrupt logged databases.

Runtime/Restartperformance

This field controls how often Domino records a recoverycheckpoint in the transaction log, which affects serverperformance.To record a recovery checkpoint, Domino evaluates each activelogged database to determine how many transactions would benecessary to recover each database after a system failure. WhenDomino completes this evaluation, it:-Creates a recovery checkpoint record in the transaction log,listing each open database and the starting point transactionneeded for recovery.-Forces database changes to be saved to disk if they have notbeen saved already.Choose one:-Standard (default and recommended). Checkpoints occurregularly.-Favor runtime (recommended to use less CPU). Dominorecords fewer checkpoints, which requires fewer systemresources and improves server runtime performance.-Favor restart recovery time. Domino records more checkpoints,which improves restart recovery time because fewer transactionsare required for recovery.

Field Enter


Figure 42. Server document for enabling transaction logging

6.2.3 Changing the transaction logging set upPerform these procedures to change transaction logging after you initially set itup and use it.

6.2.3.1 Disabling transaction loggingPerform a full backup of all databases.

Edit the Transactional Logging field, and then save the Server document.

Logging style Choose one:-Circular (default, and recommended to save space) tocontinuously re-use the log files and overwrite old transactions.You are limited to restoring only the transactions stored in thetransaction log.-Archive (recommended) to not re-use the log files until they arearchived. A log file can be archived when it is inactive, whichmeans that it does not contain any transactions necessary for arestart recovery. Use Backup Recoveryand Media Services (BRMS) for AS/400 (backup/recoveryproduct for the AS/400) to copy and archive the existing log.When Domino starts using the existing file again, it incrementsthe log file name. If all the log files become inactive and are notarchived, Domino creates additional log files.

Note: Backup Recovery and Media Services (BRMS) for AS/400is the IBM backup/recovery product for the AS/400 system.On-line Domino backup is possible with V4R4 5769BR1 (BRMS)PTF SF60285, in combination of Domino for AS/400 R5.02A.For implementation instructions, see the BRMS external Website at: http://www.as400.ibm.com/service/brms.htm

Field Enter


Restart the server so that the change takes effect. Domino runs restart recoveryto ensure all databases are consistent and then disables transaction logging.

6.2.3.2 Changing the log pathPerform a full backup of all databases.

Edit the Log path field, and then save the Server document.

Stop the server, and use the operating system to move the existing log files to thenew path.

Restart the server so that the changes take effect.

Perform another full backup of all databases because Domino assigns newDBIIDs to all databases.

6.2.3.3 Resizing the logPerform a full backup of all databases.

Edit either the Maximum log space field or the Use all available space on logdevice field. Then save the document. As Domino logs transactions, changestake effect. You do not need to restart the server.

Perform another full backup of all databases because Domino assigns newDBIIDs to all databases.

6.2.3.4 Changing the logging stylePerform a full backup of all databases.

Edit the Logging style field, and then save the Server document.

Restart the server so that the changes take effect. Domino does not assign newDBIIDs, so you do not need to perform a full database backup.

6.2.3.5 Changing automatic fixup or performanceEdit either the Automatic fixup of corrupt databases field or the Runtime/Restartperformance field. Then save the Server document. Changes take effectimmediately. You do not restart the server.

6.2.3.6 Checking your transaction log statisticsYou think that you have configured transaction logging in Domino 5.x, but you want toconfirm that your server is actually running with the setting from your NOTES.INI file.How is this done?

To check your transaction logging statistics, enter the following server consolecommand:

sh stat database.rm.*

A list of recovery manager (RM) statistics appear, including:

Database.RM.Current.K.Restart.Redo.Hold = 14669Database.RM.Current.K.UnCommited.Undo.Hold = 0Database.RM.Current.MaxChkptScan.msec = 4Database.RM.LastChkpt.Avg.K.Logged.PerTran = 2Database.RM.LastChkpt.Interval.Sec = 3876Database.RM.LastChkpt.K.Logged = 17067Database.RM.LastChkpt.Time = 07/26/1999 16:04:33 EDTDatabase.RM.LastChkpt.Total.DBs = 12


Database.RM.LastChkpt.Trans = 8233Database.RM.Peak.Avg.K.Logged.PerTran = 2Database.RM.Peak.Interval.Sec = 484Database.RM.Peak.K.Logged = 17077Database.RM.Peak.Time = 07/26/1999 14:50:12 EDTDatabase.RM.Peak.Total.DBs = 12Database.RM.Peak.Trans = 7983Database.RM.Peak.Trans.PerMin = 989Database.RM.Restart.Duration.Sec = 0Database.RM.Restart.K.Applied = 0Database.RM.Restart.Time = 07/26/1999 14:17:56 EDTDatabase.RM.SinceChkpt.K.Logged = 14301Database.RM.SinceChkpt.Trans = 1131Database.RM.SinceStartup.Aborts = 0Database.RM.SinceStartup.Critical.Log.Times = 0Database.RM.SinceStartup.Log.Recs.Analyzed = 2Database.RM.SinceStartup.Log.Recs.Applied = 0Database.RM.SinceStartup.Log.Recs.Read = 2Database.RM.SinceStartup.Log.Recs.Written = 232316Database.RM.SinceStartup.M.Logged = 80Database.RM.SinceStartup.Trans = 32581Database.RM.Sys.Logged = EnabledDatabase.RM.Sys.Log.Type = ArchiveDatabase.RM.Sys.M.Chkpt.Interval = 16Database.RM.Sys.M.Log.Size = 192Database.RM.Sys.M.Redo.Limit = 49Database.RM.Sys.Phase = Normal

6.2.3.7 NOTES.INI parameter TRANSLOG_StatusThe TRANSLOG_Status NOTES.INI parameter is used to enable transactionlogging for all R5.0 databases on the server. In this parameter, “0” is disabled and“1” is enabled.

6.2.4 Disabling transaction logging for a specific databaseIn most cases, we do not recommend disabling transaction logging. If you do, youneed to run the Fixup task to recover databases following a system failure.

After you set up transaction logging, all databases that are in the DominoRelease 5 format are logged. You can disable transaction logging of specificdatabases. For example, you can disable logging of a database that stores onlyattached files, because changes to those files are not recorded as transactions.

First, perform any of the following actions:

• When creating a new database, select Disable transaction logging on theAdvanced Databases Options dialog.

• For an existing database, select Disable transaction logging on theDatabase Properties box.

• In Domino Administrator, select a database on the Files tab. SelectTools->Database->Advanced Properties. Then, choose Disabletransaction logging.

• Use the Compact task with the -t parameter.

Second, ensure that all users have closed the database. Next, use the dbcache

command with the flush parameter to close the database in the database cache.Finally, open the database.


6.2.5 Fixing corrupted databasesCorrupted databases don't occur frequently when you use Release 5 databasesand transaction logging. When you use transaction logging to log changes toRelease 5 databases, a server automatically uses the transaction log to restoreand recover databases after a system failure, for example after server failures orpower failures.

Transaction logging is an integral part of recovering from system and mediafailures. A system failure causes the server to stop and requires you to restart theserver. During restart, Domino automatically performs database recovery. Thesystem uses the transaction logs to apply or undo database transactions notflushed to disk for databases that were open during the system failure.

Domino also runs the Fixup task on databases that use formats from earlierreleases, databases that are in Release 5 format but have transaction loggingdisabled, and on corrupt databases if the Auto fixup of corrupt databases field inthe Server document set to Yes.

FIXUP -JFixup -J causes Fixup to run on Release 5 databases that are enabled fortransaction logging. Fixup -J should only be run if a database is corrupt and youhave no backup of the database from which to roll forward.

Without this -J option, Fixup generally doesn't run on logged databases. TheFixup task interferes with the way transaction logging keeps track of databases.

6.2.6 Transaction logging: TroubleshootingThis section describes common problems with transaction logging. You can alsosearch for solutions to common problems on the Lotus Customer Support Website at: http://www.support.lotus.com

6.2.6.1 Invalid transaction log pathIf Domino cannot access the transaction log path, the server console displayserror messages indicating: the invalid log path, databases requiring mediarecovery or Fixup, and a panic. In this case, follow these steps:

1. Check the log path to make sure it exists.

2. Check that the server has write access to the log path.

3. If the log path is correct and the device is good, restart the server. Theproblem should be fixed and you do not need to continue to step 4.

4. If the log path is correct but the device is bad, replace the device on the logpath, or edit the TRANSLOG_Path setting in the NOTES.INI file to point to adifferent log path.

The dbcache flush command closes all databases that are currently open in thedatabase cache. This command can be used to maintain databases in thecache, for example, to back up or restore the databases.

Note


If you edit TRANSLOG_Path, be sure to make the same edit to the Log pathfield in the Server document upon restarting the server, or Domino reverts tothe old path upon the next server restart.

5. Restart the server. Domino creates new log files and a control file, and assignsnew DBIIDs to all Release 5 databases.

6. If the Automatic fixup of corrupt databases field is set to Yes in the Serverdocument, the Fixup task runs on the databases that require media recoveryor Fixup. Otherwise, you must run the Fixup task manually.

7. Perform full database backups.

6.2.6.2 Transaction log is damaged or corruptIf the transaction log appears to be damaged or corrupt, the server consoledisplays error messages indicating: the log is damaged, databases requiringmedia recovery or Fixup, and a panic.

The error may occur due to a failed read of or write to the transaction log. In thissituation, follow these steps:

1. Restart the server to correct the error. If the damaged log error does notappear again, the log is not damaged.

2. Stop the server again so it shuts down cleanly.


4. Restart the server.

If you continue to received the damaged log error, the active transaction log isdamaged or corrupt.

5. Delete the transaction log files and the control file.

6. Restart the server. Domino creates new log files and a control file, and assignsnew DBIIDs to all Release 5 databases.

7. If the Automatic fixup of corrupt databases field is set to Yes in the Serverdocument, the Fixup task runs on the databases that require media recoveryor Fixup. Otherwise, you must run the Fixup task manually.


If the error occurred during media recovery, an archived log file may becorrupt.

9. Restart the server to correct the problem, and then stop the server so it shutsdown cleanly.

10.While the server is down, use the third-party backup utility to perform mediarecovery. If the archived log still cannot be used, allow database backups tobe restored without the transactions in the corrupt log.

11.Perform full database backups.

12.Restart the server.


6.3 Troubleshooting tools in Domino Administrator

The Domino Administrator client provides several tools to help you troubleshootproblems. Most of the tools are available through the Domino Administrator. Table10 summarizes the available tools.

Table 10. Administrator tools

6.4 NOTES.INI file and its parameters

Customers have often asked for quick tools to help understand Domino relatedproblems. In specific situations, customers are frequently advised to activatecertain debug parameters in the server NOTES.INI file to collect more informationfor further analysis or diagnosis of problems.

While some customers may be reluctant to activate the debug parameters, othersmay request that such parameters be activated by default into the system.

In fact, you should rarely, if ever, need to modify the NOTES.INI file. TheNOTES.INI file contains many settings that Domino and Notes rely on to workproperly. An accidental or incorrect change may cause Domino or Notes to rununpredictably. Before you make changes to the NOTES.INI file, ensure that youfully understand what you are doing, and why the changes are necessary.

Tool Problems that thetool resolves

How to access the tool

Server log file(LOG.NSF)

All problems From the Server - Analysis tab in theDomino administrator

Server MAIL.BOX Mail routingproblems

From the Messaging - Mail tab in theDomino Administrator

Mail trace Mail routingproblems

From the Messaging - Mail tab in theDomino Administrator

ISpy Slow mailServer problems

Configured in the Statistics and Eventsdatabase on the Configuration tab in theDomino Administrator

Mail tracking Lost mail From the Messaging - Tracking Center tab inthe Domino Administrator

Network trace Connectionproblems

In User Preferences. ChooseFile->Preferences->Notes Preferences

Replicationschedule

Replication problemsfor a particular server

From the Replication tab in the DominoAdministrator

Database analysis Database problems From the Files tab in the DominoAdministrator

Of course, we always recommend that you save the NOTES.INI beforeproceeding to any modification of its content.

Note


6.4.1 Debug parameter versus activity log parameterFor the purpose of our discussion, let us divide the parameters into two differentcategories:

• Debug parameters• Activity log parameters

Although the use of these parameter categories is similar, the approaches andintentions are quite different.

6.4.1.1 Debug parametersDebug parameters are often not published publicly, and are used only for specificneeds and circumstances. They are activated with close supervision and onlyupon direct advice between the customer and the IBM/Lotus Support Team. Theyare activated only to resolve specific problems that are happening in yourenvironment.

Debug parameter exampleIf you want to debug an agent, you have to enable the debug parameterDebug_AMGR=* in the NOTES.INI file. This is an undocumented NOTES.INIparameter that allows the Domino server to report agent manager activities to thelog.nsf or to an outfile, enabling further analysis and diagnosis.

6.4.1.2 Activity log parametersActivity log parameters are published in Administering the Domino System(Volume 1 and 2). The primary focus is to allow Notes Administrators tounderstand the behavior of the Domino server and its applications. Both volumesof Administering the Domino System can be accessed on the Web at:http://notes.net/notesua.nsf

Under View the documentation by, select Printed Book Title. Search forAdministering the Domino System (Release 5.0).

Most of the time, there is no real problem. Using these parameters is simply aproactive step in collecting more data and making appropriate analysis tofine-tune the Domino server and its applications.

Notes Administrators don’t have to engage IBM/Lotus Support team to decidewhich activity log parameters to enable. Administering the Domino System(Volume 1 and 2) is comprehensive on the options of these parameters. However,we encourage you to consult the IBM/Lotus Support Team for further clarificationon how to use these parameters for your own analysis.

Activity log parameter exampleIf you want to understand the behavior of your Agent activities, you have toenable the parameter Log_AgentManager=<value 0-2> in the NOTES.INI file.

More details on such activities logging parameters are found in Administering theDomino System (Volume 1 and 2).

Note: In these two examples of Agent Manager, we can see the distinctionbetween the two in the approach and intention of activating these parameters.However, there are cases where the activity log parameter can be use todetermine the cause of the problem.


6.4.2 Understanding debug parameters in the NOTES.INI fileLet us understand the issues revolving around the Notes debug parameters in theNOTES.INI file and examine them in a broader perspective.

Applying debugging parameters is always a reactive approach, and it must betreated with caution. It is necessary only in certain circumstances. Often, itrequires close supervision by Notes administrators with advice from theIBM/Lotus Support Team.

Such close monitoring is necessary to avoid any potential adverse effects whenapplying the debug parameters. There are always some trade-offs in applyingdebug parameters. If applied indiscriminately, they may have implications on yourserver performance.

Keep in mind that applying debug parameters does not generally cause serverfailures, but may contribute to the increment of your server workloads. Therefore,server performance may degrade.

The use of debug parameters often occurs only after a specific problem appearsin a customer environment. These parameters should be used on a short-termbasis during the troubleshooting period to find the cause of the problem.

Some of the debug parameters are described in Appendix C, “Debuggingsettings” on page 287.

6.4.3 Activity log parameters in the NOTES.INI fileMany of the published NOTES.INI parameters documented in Administering theDomino System (Volume 1 and 2), are generally for administrative and serverperformance tuning purposes. In contrast, the parameters introduced byIBM/Lotus Support are specifically meant for debugging purposes.

If you intend to monitor your server and application behaviors, you can customizeyour NOTES.INI file to collect additional information. The output of these activitieswill be captured in the log.nsf file.

When you record additional information in the log file, choose a logging level thatspecifies the amount of detail that is recorded. If you set a high logging level, thelog file may become quite large. Therefore, after you troubleshoot the problem,remember to reset the logging level. Table 11 lists some debug information youmay want to record in the log file.

Table 11. Collecting additional information

To record information about Use this setting, field, or command

Agent Manager Log_ AgentManager in NOTES.INI

Indexer activity Log_update in NOTES.INI

Mail routing Mail_log_to_MiscEvents in NOTES.INI

Replication Log_Replication in NOTES.INI“Log all replication events” setting specifiedduring server setup

Server tasks Log_Tasks in NOTES.INI

User sessions Log_Sessions in NOTES.INI


6.4.3.1 Log_ AgentManagerThis parameter specifies whether the start of an agent execution is recorded inthe log file and shown on the server console.

The syntax for this parameter is:

Log_AgentManager=value

Here, value can be one of the following numbers:

0 Do not log agent execution events1 Log agent execution events (partially and completely successful)2 Log agent execution events (completely successful only)

6.4.3.2 Log_UpdateThis parameter specifies the level of detail of Indexer events displayed at theserver console and in the log file.


Log_Update=value

Here, value can be one the following numbers:

0 Records when the Indexer starts and shuts down.

1 Records when the Indexer starts and shuts down and when the Indexerupdates views and full text indexes for specific databases.

2 Records when the Indexer starts and shuts down and when the Indexerupdates views and full text indexes for specific databases. Also records thenames of views the Indexer is updating.

6.4.3.3 Mail_Log_to_MiscEventsThis parameter, in the NOTES.INI file, determines whether all mail eventmessages are displayed in the Miscellaneous Events view of the log file.


Mail_Log_To_MiscEvents=value

Here, value can be one of the following possibilities:

0 Does not display mail events in the Miscellaneous Events view1 Displays mail events in the Miscellaneous Events viewDefault None. However, if this setting is omitted, mail events are not

displayed in the Miscellaneous Events view.

6.4.3.4 Log_ReplicationThis parameter specifies whether the start and end of replication sessions arerecorded in the log file and displayed on the console.

Directory cataloger task Log_DirCat in NOTES.INI

Messages generated when views are rebuilt Log_View_Events in NOTES.INI

To record information about Use this setting, field, or command


The syntax is:

Log_Replication=value

Here, value can be one of the following numbers:

0 Do not log replication events

1 Log server replication events

2 Log replication activity at the database level

3 Log replication activity at the level of database elements (views, document,etc.)

4 Log replication activity at the field level

5 Log summary information

6.4.3.5 Log_TasksThis parameter specifies whether the current status of server tasks is recorded inthe log file and displayed on the console.

The syntax is:

Log_Tasks=value

Here, value is one of the following numbers:

0 Do not send status information1 Send the status of server tasks to the log file and to the console

6.4.3.6 Log_SessionsThis parameter specifies whether individual sessions are recorded in the log fileand displayed on the console.

The syntax is:

Log_Sessions=value

Here, value is one of the following numbers:

0 Do not log individual sessions1 Log individual sessions

6.4.3.7 Log_DirCatLogs the following information about the Directory Cataloger task to theMiscellaneous Events view of the log file:

• When the Directory Cataloger starts• On which directories the Directory Cataloger runs• When the Directory Cataloger finishes

The syntax is:

Log_DirCat=value

Here, value is one of the following possibilities:

None (default) Do not log1 Logs the above listed information


6.4.3.8 Log_View_EventsThis parameter specifies whether messages generated when views are rebuiltare recorded in the log file.

The syntax is:

Log_View_Events=value

Here, value is:

0 Do not log messages when views are rebuilt1 Log messages when views are rebuilt

6.5 Editing the NOTES.INI file

The different ways to edit the NOTES.INI file include:

• Using the Set Configuration command at the console• Editing the Server Configuration document• Using the Domino Administrator client• Using AS/400 Operations Navigator• From a 5250 display

We explain each of these methods in the following sections.

6.5.1 Using the Set Configuration commandTo edit the NOTES.INI file with the Set Configuration command, you need to enterthis command at the Domino server console.

Usage: Set Configuration setting

Example:

> set config SERVERTASKS=Router,Replica,Update,Amgr,AdminP,CalConn,Sched,Event,Stats,HTTP,DECS

6.5.2 Using the configuration documentTo use the configuration document, follow these steps:

1. From the Domino Administrator, open the Domino Directory, and click theConfiguration tab.

2. To edit an existing Configuration Settings document, highlight it, and then clickEdit Configuration. To create a new configuration document, highlight theserver for which the Configuration Settings document will apply. Then, clickAdd Configuration.

• Before updating the NOTES.INI file, make a backup copy of the file. Toenable the new parameters take effect, you need to recycle the Dominoserver.

• Examine the characteristics of your editor. If you use a different editor thanthose mentioned, choose one that does not add carriage returns (CR) to thefile. Most workstation editors add CRs.

Notes


3. To modify NOTES.INI settings on the server, click the NOTES.INI Settingstab. This tab lists a number of current settings in the server NOTES.INI file.

4. To add or change a setting, click Set/Modify Parameters to display allsettings that you can set in the Configuration Settings document. Select thesettings you want to add or modify.

5. Save and close the document.

6.5.3 Using the Domino Administrator clientThis section explains how to use the Domino Administrator client. Follow thesesteps:

1. To edit the NOTES.INI file, from your Domino Client screen select the DominoAdministrator icon or click File->Tools->Server Administration. The displayshown in Figure 43 appears. Press Esc to move the Administration menu.

Figure 43. Domino Administrator welcome page

2. On the display that appears (Figure 44), select the Configuration tab andclick Server->Configurations.


Figure 44. Notes Administrator Client main menu

The box in the center shows a list of servers. Figure 45 shows one server.

3. Highlight the server you want to administer, and double-click it.

Figure 45. Server Configurations

4. Figure 46 on page 110 shows the Configuration Settings panel. When you firstenter this panel, click the NOTES.INI Settings tab. This shows the Currentparameters and Last parameter set.


Figure 46. Configuration settings

5. Click Edit Server Configuration. This enables you to edit certain NOTES.INIparameters. The display shows the current and last parameter set. It alsoshows the value to which the last parameter was set and the person whoupdated that parameter. Click Set/Modify Parameters.

Figure 47 shows the pop-up display that appears when you set or modify theparameters.

Figure 47. Selecting parameters

6. If you click the drop-down arrow, it shows a list of parameters that can beedited. Highlight Log_AgentManager, and click OK. As demonstrated inFigure 48, to set a value, enter the value you want to set for this parameter,and click Next. Then click OK.


Figure 48. Setting parameter value

This adds it to the list of current parameters if this parameter was never setbefore, as shown in Figure 49.

Figure 49. Updated server configuration parameters

The list of servers is also updated, as shown in Figure 50 on page 112.


Figure 50. Updated servers view

6.5.4 Using AS/400 Operations NavigatorTo edit the NOTES.INI file from AS/400 Operations Navigator, select the AS/400system (AS/400 connection). Then click File Systems->Integrated FileSystem->Root. Select Directories until you get to the subdirectory where theNOTES.INI file is stored. Find and right-click the NOTES.INI file, and select Edit.

Figure 51 shows the editing of the NOTES.INI file using the AS/400 OperationsNavigator editor.


Figure 51. NOTES.INI file, edited through the AS/400 Operations Navigator editor

6.5.5 Editing the NOTES.INI file from a 5250 displayTo edit the NOTES.INI file from a 5250 display, enter the Work with DominoServers (WRKDOMSVR) command. Position the cursor in front of the Domino servercorresponding to the NOTES.INI file you want to edit. Then, type 13 (EditNOTES.INI).

Figure 52 on page 114 shows the editing of the NOTES.INI file from a 5250display.

For more information on the Edit File (EDTF) command, refer to 7.1,“Displaying or editing a stream file from a 5250 display” on page 117.

Note


Figure 52. NOTES.INI file edited from a 5250 display

6.6 Tools for monitoring the Domino server

The Domino server provides services and tasks that create and reportinformation about the Domino server. This information comes in two forms:statistics and events.

Statistics show the status of processes running on the server. For example, the“Free space on drive C” statistic indicates the amount of free space available ondrive C. On the AS/400 system, there is no disk drive letter. However, thispresentation is maintained when the show stat command is used or in the Dominoserver Statistics Reports database.

From the Domino server console, use the show stat disk command to display thefollowing disk statistics:

Disk.C.Free = 63,317,577,728Disk.C.Size = 103,079,215,104Disk.Fixed = 1Disk./.Free = 63,317,577,728

An example of the statistics report, kept in the Domino server Statistics ReportsDomino database, is shown in Figure 53.

The Domino server continuously updates statistics. To view server statistics atany time, use the Show Stats command at the server console. To obtain statisticsto monitor the Domino server, use the Collect task, which collects statistics and

Edit File: /DOMINO/DOMSVR3/NOTES.INIRecord . : 1 of 17 by 10 Column: 1 of 74 by 126Control :

CMD....+....1....+....2....+....3....+....4....+....5....+....6....+....7....+....8....+....9....+....0....+....1....+....2....+..

************Beginning of data**************[Notes]Directory=/DOMINO/DOMSVR3KitType=2NPN=1UNICODE_DISPLAY=1NSF_HOOKS=QNNDIHKCleanupScriptPath=CALL QNOTES/NSDPassthru_LogLevel=0Console_LogLevel=2VIEWIMP1=Lotus 1-2-3Worksheet,0,_IWKSV,,.WKS,.WK1,.WR1,.WRK,.WK3,.WK4,,4,VIEWIMP2=Structured Text,0,_ISTR,,.LTR,.CGN,.STR,,1,VIEWIMP3=Tabular Text,0,_ITAB,,.PRN,.RPT,.TXT,.TAB,,1,VIEWEXP1=Lotus 1-2-3 Worksheet,0,_XWKS,,.WKS,.WK1,.WR1,.WRK,,4,VIEWEXP2=Structured Text,0,_XSTR,,.LTR,.CGN,.STR,,1,VIEWEXP3=Tabular Text,1,_XTAB,,.LTR,.RPT,.CGN,.TAB,,1,EDITIMP1=ASCII Text,0,_ITEXT,,.TXT,.PRN,.C,.H,.RIP,,1,EDITIMP2=MicrosoftWord RTF,0,_IRTF,,.DOC,.RTF,,2,

F2=Save F3=Save/Exit F12=Exit F15=Services F16=Repeat find F17=Repeat change F19=LeftF20=Right

(C) COPYRIGHT IBM CORP. 1980, 1999.


puts the information into the statistics database (STATREP.NSF). Then to viewstatistic reports, click Server->Analysis tab->Statistics Reports in the DominoAdministrator.

Figure 53. Domino server Statistics Report

Events are generated when something has occurred in your Domino serverenvironment. For example, the event “Replicating files with servername” occurswhen a file replicates with a specified server.

The Domino server also provides monitor documents that you use to configurestatistic thresholds. When the Collect task collects a statistic and places it into theStatistics database (STATREP.NSF), it compares the statistic to the thresholdconfigured in the monitor document. The first time the statistic reaches thespecified threshold in the monitor document or in the Statistic Names document,an alarm report is generated. An alarm report is nothing more than a documentinforming you that a statistic has reached its threshold. To set up an alarm, youmust specify a collection alarm interval in the Server Statistic Collectiondocument. Alarms documents are automatically created in the Server->AnalysisTab->Statistics Reports->Alarms view in the Domino Administrator.

Events happen continuously on the Domino server. To notify you about importantserver information, the Domino server has a notification facility called the Eventtask. The Event task sends notification about the event to a destinationconfigured in the Event Notification document.

You can create statistics and events monitoring documents for your particularserver and network needs by creating Statistic Monitors and Event Notificationdocuments from the Statistics & Events view on the Configuration tab in the


Domino Administrator. If you do not set up an Event Notification document, youare not notified that an event has occurred.

6.6.1 Starting the Event and Collect tasksThe Event task and the Collect task are two of the server monitoring tasks thatgather information about the Domino server.

Note: You must run the Event task on each server that you want to monitor.

6.6.1.1 Starting the Event taskThe Event task monitors server activities when you start the Domino server. TheEvent task also creates the Statistics & Events database (EVENTS4.NSF) on theserver, if it doesn’t exist.

The Event task is automatically loaded at server startup. If the Event task is notloaded, you must load it manually from the Domino Administrator or the serverconsole. To do so, follow these steps:

1. From the Domino Administrator, click the Server - Status tab.2. Click Task - Load.3. In the Load task box, click Event.4. Click OK.

You may also start the task from the server console by typing load event.

6.6.1.2 Starting the Collect taskThe Collect task collects and monitors statistics from the servers configured inthe Server Statistic Collection document. To start the Collect task, load it on theserver designated to collect statistics. To do so, follow these steps:

1. From the Domino Administrator, click the Server - Status tab.2. Click Task -> Load.3. In the Load task box, click Collect.4. Click OK.

You may also start the task from the server console by typing load collect.

For more information on these monitoring tools, refer to the online Domino 5Administrator Help guide from the Domino Client (or Notes help database forRelease 4.6).


Chapter 7. Additional considerations for problem determination

As mentioned earlier in this redbook, it is important to rely on Domino for AS/400documentation to manage Domino for AS/400 servers. The redbook LotusDomino for AS/400 R5: Implementation, SG24-5592, is also helpful inunderstanding the Domino for AS/400 environment.

In this chapter, our objective is to highlight some aspects that are not alwaysgiven the attention they deserve:

• Displaying or editing a stream file from a 5250 display• The DOMINO_SERVERS file• The Domino Web Server configuration file• Ending a Domino for AS/400 server• IPLing the AS/400 system (not really a solution)

7.1 Displaying or editing a stream file from a 5250 display

After you set up your Domino server on the AS/400 system, you may need todisplay or edit some stream files that are important for your Domino serverenvironment or for debugging purposes.

Different commands are available, depending on the OS/400 release you areusing on your AS/400 system:

• For displaying stream files:

– The Display File (DSPF) command is available directly in OS/400 fromOS/400 V4R4.

– The Display Stream File (DSPSTMF) command is available for OS/400V4R2 and V4R3 through an OS/400 PTF.

To use the DSPSTMF command on OS/400 V4R2 or V4R3, you must add aPTF to OS/400: SF45296 for OS/400 V4R2 or SF49052 for OS/400 V4R3.For details, see the Domino for AS/400 Release Notes in the documentAS/400 PTF requirements.

• For editing stream files:

The Edit File (EDTF) command is available directly in the OS/400 from V4R4.It is also available for OS/400 V4R2 and V4R3 through the same OS/400 PTFrequired for the Display Stream File (DSPSTMF) command.

For OS/400 releases previous to V4R4, the Edit File (EDTF) command, whichis required for option 13 (Edit NOTES.INI), when using the Work with DominoServers (WRKDOMSVR) command, is supplied in the separate QGPTOOLSlibrary. If a message indicates that the command was not found, try typingqgptools/edtf.

For example, you may need to add or change settings in the NOTES.INI file. TheAS/400 Edit File (EDTF) command provides a simple editor for changing a streamfile, such as the NOTES.INI file. You must have the AS/400 authority to changethe NOTES.INI file.


The EDTF command is supplied in the QGPTOOLS library. If a messageindicates that the command was not found, try typing qgptools/edtf. To use theEDTF command on some versions of OS/400, you may need an OS/400 PTF. Fordetails, see the Domino for AS/400 Release Notes.

On any AS/400 command line, type edtf, and press the F4 key. Then, type thepath that identifies the location of the file in the AS/400 Integrated File System.For example, if the NOTES.INI file is in the /LOTUS/NOTES/DATA directory, type:

/lotus/notes/data/notes.ini

Press Enter.

Note: The path for the Notes data directory is specified when the Domino serveris set up.

While in the editing session, you can use the i line command to insert a line andthe d line command to delete a line.

Press the F3 key to end the session and save the changes. Press the F12 key toend the session without saving the changes.

7.2 The DOMINO_SERVERS and LOTUS_SERVERS files

This file lists all your servers that are defined, the subsystems they are in, and thenumber of the last partition used. In OS/400 V4R2 and V4R3, the file was difficultto read with DSPSTMF. In V4R4, it is presented as a text file.

The display in Figure 54 shows an example of the result of DSPF on theDOMINO_SERVERS file. This file cannot be read with a PC editor. For anyrelease up to Release 5.0.1, this file can be found in the IFS directory/QIBM/UserData/LOTUS/NOTES.

Changing the NOTES.INI file is recommended only for experiencedadministrators. Changes to the NOTES.INI file can introduce errors into the fileand impair the operation of a Domino server.

Attention

You can also edit the NOTES.INI file from the Work with Domino Serversdisplay, using option 13 (Edit NOTES.INI). To request this display, enter theWork with Domino Servers (WRKDOMSVR) command.

The EDTF command, which is required for this option, is supplied in theseparate QGPTOOLS library. If a message indicates that the command wasnot found, type:

qgptools/edtf

Tip


Figure 54. The DOMINO_SERVERS file displayed using the DSPF command

In Domino for AS/400 Release 5.0.2, the DOMINO_SERVERS file has beenrenamed LOTUS_SERVERS and can be found in the /QIBM/UserData/LOTUSIFS directory. The display in Figure 55 shows an example of the result of DSPPon the LOTUS_SERVERS file.

Figure 55. The LOTUS_SERVERS file displayed using the DSPF command

Browse : /QIBM/USerData/LOTUS/NOTES/DOMINO_SERVERSRecord . : 1 of 10 by 15 Column: 1 of 59 by 79Control :....+....1....+....2....+....3....+....4....+....5....+....6....+....7....+...************Beginning of data**************LAST_PARTITION=02SERVER=*HTTPSETUPDIRECTORY=/QIBM/UserData/HTTPSetup/Notes/DataSBSD=QDOMINOHTSERVER=DOMSVR3DIRECTORY=/LOTUS/DOMINO/DOMSVR3SBSD=DOMINO01SERVER=DOMSVR5DIRECTORY=/DOMINO/DOMSVR5SBSD=DOMINO02************End of Data********************

F3=Exit F10=Display Hex F12=Exit F16=Repeat find F19=Left F20=Right(C) COPYRIGHT IBM CORP. 1980, 1999.

Browse : /QIBM/UserData/LOTUS/LOTUS_SERVERSRecord . : 1 of 18 by 18 Column: 1 of 59 by 131Control :

....+....1....+....2....+....3....+....4....+....5....+....6....+....7....+....8..

...9....+....0....+....1....+....2....+....3.************Beginning of data**************SERVER=*HTTPSETUPTYPE=DOMINODIRECTORY=/QIBM/UserData/HTTPSetup/Notes/DataSBSD=QDOMINOHTPARTITION=8000SERVER=DOMSVRCTTYPE=DOMINODIRECTORY=/domino/domsvrctSBSD=DOMINO04PARTITION=4SERVER=DOMSVR5TYPE=DOMINODIRECTORY=/lotus/domino/domsvr5SBSD=DOMINO02PARTITION=2

F3=Exit F12=Cancel F16=Repeat find F19=Left F20=Right(C) COPYRIGHT IBM CORP. 1980, 1999.

Chapter 7. Additional considerations for problem determination 119

Do not change anything in these files because they can become corrupted andthe WRKDOMSVR will not work anymore.

The DOMINO_SERVERS file is in UNICODE, and cannot and should not beaccessed by either a PC editor or word processor, or by EDTF on the AS/400system. This file stores information about your Domino servers, including thedirectories and subsystem descriptions, as well as partition information.

7.2.1 When the DOMINO_SERVERS file is corruptedIf a Domino for AS/400 customer edits the file DOMINO_SERVERS (in theQIBM/USERDATA/LOTUS/NOTES/ path), or the customer copies it from anotherAS/400 system, this file does not work as expected or is corrupted.

First, establish whether the file is corrupted. If the file was copied from anotherbox, it will never work. This file is created or updated when a Domino serverconfiguration command is run. Therefore, we do not recommend that you movethe file to another box. If the file is copied, however, it may be owned by anAS/400 user profile other than QNOTES. Check the ownership first.

If the file is corrupted, perform the following steps:

1. End Directory Synchronization (if enabled) by calling the QNNDIEND programfrom the QNOTESINT library with the *all parameter.

2. End all Domino servers on this AS/400 system.

Note: Use the WRKUSRJOB command to ensure that no jobs owned byQNOTES are running or are in JOBQ status.

3. Make a backup copy of the DOMINO_SERVERS file. You may use the CPYcommand on the AS/400 system to copy it to another filename temporarily.

4. Make backup copies of the NOTES.INI and HTTPD.CNF (HTTPD.CNFdisappears after you use the Configure Domino Server (CFGDOMSVR)command).

5. For each Domino server, you must use the CFGDOMSVR command toreconfigure. You must put all current information in it, as you configured theDomino server before (server name, domain, mail, data path, and so on).Then, set the REPLACE Configuration parameter to “No”. Now the file will becreated with the new information.

6. Enter the WRKDOMSVR command and the new information will appear.

7. You may delete the file created during step 3 (the one that you renamed).

8. You may have to manually update the NOTES.INI if there are any special INIvariables or settings that you had set originally. You may use either (on theAS/400 system) the EDTF command or the CA/400 Operations Navigator toedit the file. Do not edit the NOTES.INI using Wordplay or any other wordprocessor.

7.3 The Domino Web Server configuration file (httpd.cnf)

The Domino Web Server configuration file, called httpd.cnf, can be found in theDomino data server directory. The contents of this file are used by the DominoWeb server. Modifications, additions, or deletions of directives in this file, exceptfor MIME types, are unsupported.


When the Domino Web server is started (or restarted), it reads two configurationfiles: httpd.cnf followed by domino.cnf.

The domino.cnf file is generated from scratch each time the Web server isstarted. Modifications to this file will not be honored. These settings should bechanged in the server's Domino Directory (Name & Address Book) and in theDomino configuration database. In some situations, this file can becomecorrupted and needs to be restored.

7.4 Ending the Domino for AS/400 server

The Domino Server for AS/400 must be ended in order to run a successfulbackup of the data directory. Many Domino for AS/400 administrators want toautomate the process of ending the Domino Server so that they can run a backupduring off peak hours, with nobody monitoring the process. How can this bedone?

To following example shows a program that uses the QWDRSBSD API. Theprogram uses that API to check for active jobs in the Domino subsystem. In thisprogram, the ENDDOMSVR *CNTRLD command is issued, and the API is calledin a loop to see if all the Domino jobs have ended. After 15 minutes (that is 30iterations of the loop with a 30 second delay in each loop), the ENDDOMSVR*IMMED command is issued and the loop starts over again.

*************** Beginning of data **********************************************0001.00 PGM0002.000003.00 /* Below are the variable declarations */0004.000005.00 /* This pgm has hardcoded the Domino subsystem name as DOMINO01. */0006.00 /* Your Domino subsystem may have a different name, so you would change */0007.00 /* the value of this variable to whatever your Domino subsystem is called. */0008.000009.00 DCL VAR(&SBSD) TYPE(*CHAR) LEN(20) +0010.00 VALUE('DOMINO01 QUSRNOTES ')0011.00 DCL VAR(&ERRCODE) TYPE(*CHAR) LEN(8) +0012.00 VALUE(X'0000000000000000')0013.00 DCL VAR(&HEXZERO) TYPE(*CHAR) LEN(4) +0014.00 VALUE(X'00000000')0015.00 DCL VAR(&RCVVAR) TYPE(*CHAR) LEN(256)0016.00 DCL VAR(&RCVLENBIN) TYPE(*CHAR) LEN(4) VALUE(X'00000100')0017.00 DCL VAR(&FMTNAM) TYPE(*CHAR) LEN(8) VALUE('SBSI0100')0018.00 DCL VAR(&ERRCDE) TYPE(*CHAR) LEN(256)0019.000020.00 /* We named our Domino Server 'SERVERNAME'. */0021.00 /* Your DOMSVR value will be whatever your Domino Server Name is. */0022.000023.00 DCL VAR(&DOMSVR) TYPE(*CHAR) LEN(10) VALUE('SERVERNAME')0024.00 DCL VAR(&CURACTJOB) TYPE(*CHAR) LEN(4) VALUE(X'00000000')0025.00 DCL VAR(&DNUMJOBS) TYPE(*DEC) LEN(10 0)0026.00 DCL VAR(&CNUMJOBS) TYPE(*CHAR) LEN(30)0027.00 DCL VAR(&COUNT) TYPE(*DEC) LEN(3 0) VALUE(0)0028.000029.00 /* Below is the body of the pgm */0030.00 /* The command after the DLYJOB will end all the Directory Synchronization jobs */0031.000032.00 ENDDOMSVR &DOMSVR0033.00 DLYJOB DLY(60)0034.000035.00 CALL QNOTESINT/QNNDIEND *ALL0036.000037.00 LOOP:0038.00 CALL QWDRSBSD PARM(&RCVVAR &RCVLENBIN &FMTNAM &SBSD &ERRCODE)0039.000040.00 /* The next section of code is to convert the current number of active jobs into a */0041.00 /* value that can be displayed in a msg. This is done so that a user could see */0042.00 /* that the program is working. If you change the TOUSR field to *SYSOPR, the */


0043.00 /* msg will be sent to the QSYSOPR msgq. */0044.000045.00 CHGVAR &CURACTJOB %SST(&RCVVAR 73 4)0046.00 CHGVAR &DNUMJOBS %BINARY(&CURACTJOB)0047.00 CHGVAR &CNUMJOBS &DNUMJOBS0048.00 SNDMSG MSG('Jobs:' *CAT &CNUMJOBS) TOUSR(QSECOFR)0049.000050.00 /* This portion tests to see if the number of currently active jobs equals 0 */0051.00 /* If the number of jobs = 0, then it goes to the NEXT label and exits the pgm */0052.00 /* If not, then the pgm checks to see how many times it has gone through the loop */0053.00 /* If the loop has been executed 30 times, it will go to the END label */0054.000055.00 IF (&CURACTJOB =&HEXZERO) GOTO NEXT0056.00 IF (&COUNT > 30) GOTO END0057.000058.00 CHGVAR &COUNT (&COUNT + 1)0059.00 DLYJOB DLY(30)0060.00 GOTO LOOP0061.000062.00 NEXT:0063.00 SNDMSG MSG('NO JOBS!' *CAT &CNUMJOBS) TOUSR(QSECOFR)0064.00 ENDPGM0065.000066.000067.00 END:0068.000069.00 /* The code comes to this section if the ENDDOMSVR did not end all the jobs */0070.00 /* in the fifteen minutes it takes to run through the above loop.If the pgm enters */0071.00 /* this section of the code, it issues an ENDDOMSVR *IMMED, resets the */0072.00 /* COUNT variable, and returns to the loop to check if the jobs in the sbs have ended0073.00 */0074.00 ENDDOMSVR &DOMSVR *IMMED0075.00 CHGVAR VAR(&COUNT) VALUE(0)0076.000077.00 GOTO LOOP0078.000079.00 ENDPGM

****************** End of data *************************************************

This information is also found in Lotus Domino for AS/400 R5: Implementation,SG24-5592.

7.5 IPLing the system

There is a strong belief that a system IPL is the solution for many Dominoproblems, including how to restart after a Domino server crash.

When a Domino server crash happens on the AS/400 system, there are othersolutions other than restarting the system where Domino runs. More informationon handling a server crash is provided in Chapter 8, “Server crash” on page 127.

There are very few problem situations where a restart of the AS/400 system isnecessary.

7.5.1 The Power Down System (PWRDWNSYS) commandThe Power Down System (PWRDWNSYS) command prepares the system forending and starting the power-down sequence. All active subsystems are notifiedthat the system is being powered down. No new jobs or routing steps can bestarted by any subsystem.

The How to end (OPTION) parameter specifies whether the system allows theactive subsystem to end processing of active jobs in a controlled manner (whichlets the application program perform end processing), or whether the systemends the jobs immediately. In either case, the system performs certainjob-cleanup functions.


The Delay time, if *CNTRLD (DELAY) parameter specifies the amount of time, inseconds, that the system allows a controlled end to be performed by the activesubsystems. If the end of job routines are not finished in the specified delay time,any remaining jobs are ended immediately.

The Restart after power down (RESTART) parameter specifies whether thesystem ends and powers down, or whether the system ends and then starts againin unattended mode.

Two other parameters (Restart type, and IPL source) are not described here.

A very common usage of the Power Down System (PWRDWNSYS) command is:

PWRDWNSYS RESTART(*YES)

Note the following points:

• The How to end (OPTION) is *CNTRLD (controlled, which is the default value).• The Delay time is 3600 seconds (default value).

In this example, the system is ended in a controlled way and restartsautomatically afterwards.

We strongly recommend that you stop all Domino servers running on the systemand to run the CALL QNOTESINT/QNNDIEND *ALL command before stopping the system(or re-IPLing it, for example with the PWRDWNSYS RESTART(*YES) command).

If the system stops automatically (for example, through the Power On/OffSchedule, available in AS/400 Operational Assistant), the program described in7.4, “Ending the Domino for AS/400 server” on page 121, should be used as anexample to automate that process.


Part 2. Looking at specific problem areas

This second part of this redbook offers details on how to handle problems in thefollowing areas:

• Server crashes• Where has my mail disappeared or gone?• HTTP server• Database integration problems• Performance problems• Network problems


Chapter 8. Server crash

This chapter gives you a quick checklist of what to do first if your Domino servercrashes. Are you on the latest software levels and fixes available? Which datahas to be collected to get an overview of the problem. Was it only a user error oris it a severe problem? What is the essential information in the data you needcollect and provide, as a problem summary, to the Domino for AS/400 supportspecialist. Read on to learn how to handle a Domino server crash.

8.1 What is a server crash

A server crash is the moment when one of the Domino tasks ends in error and, asa result, all other Domino tasks are ended. The server no longer responds to anyuser requests. In most cases, this ending of the tasks happens in a controlledway more or less, and the Notes Server Diagnostics tool (NSD) is started to dumpall available data of this failing task to the NSD file.

The following quote from an article found at http://www.notes.net is another viewof a server crash:

“There's nothing fun about a Domino server crash. Everything stops working,users get frustrated, and everyone blames you—the Domino administrator.Fortunately, most administrators can quickly troubleshoot a server crash and gettheir machine back online before angry users storm the data center. However,what if you're not so fortunate? You try everything and still your server refuses tocooperate. That's when it's time to call Lotus Support.”

8.2 A beginner’s checklist

When a Domino for AS/400 server crashed, and does not restart, what is the bestmethod for someone new to Domino on the AS/400 system for diagnosing theproblem?

Follow this general checklist to gather all possible information about the servercrash. This data gives the Domino for AS/400 analyst a general idea of the natureof the problem.

1. Check for the release of the OS/400 operating system and, for current CUMlevels and OS/400 PTFs required for Domino. Check this on the Web at:http://www.as400.ibm.com/domino/DomSupport.htm

For example, for OS400 Release R440, click List of V4R4 PTFs related toDomino for AS/400.

2. Check for the Domino for AS/400 release and its proper QMU level at:http://www.as400.ibm.com/domino/qmulist.htm

IBM and Lotus support people should check if there are hot fixes relevant to thatQMU. If so, they should be checked and applied. See the AS/400 DominoInformation database.

3. Is there an NSD file generated? Look for the QSYSOPR message queue, allspooled files for user QNOTES, and especially the QNNINSTS job log. Refer toChapter 3, “Where to find error information in the system” on page 33.

4. Verify whether the server has ever worked?


5. Identify the function of the Domino server. For example, is this a mail server, adatabase application, the HTTP task, the SMTP MTA task, a resource reservationserver, or a combination of some of these items?

6. Obtain a description of the crash sequence from STRDOMSVR to fault and endthe server functionality. Which server tasks and applications were started? Didanything unusual happen before or when the server was running?What changes have been made? Has any PTF been recently applied?

7. What was the last shutdown procedure like? Did any errors happen when theENDDOMSVR command was run? Is there an automated shutdown?As an example for an automated shutdown, see 7.4, “Ending the Domino forAS/400 server” on page 121, or Lotus Domino for AS/400 R5: Implementation,SG24-5592.

8. Does the server start up at all? Are there incorrect parameters in the NOTES.INIfile, such as backslashes used instead of forwardslashes? The NOTES.INI fileshould be studied if there is no NSD file.

9. Check the STATREP.NSF - System View initially. Statistic reports where theserver has been running for several days are preferred. The STATREP.NSF filecan provide very useful information in the event of a crash. Many times, servercrashes are due to resource issues that can be addressed by analyzing thestatistics on the server.

10.Are there authorities issues? Are there any authority error messages in the joblogs or spooled files?

11.Are there QNOTES ownership issues? Do all data directory objects haveQNOTES ownership?

12.Do errors appear in the current Notes log? If the LOG.NSF Domino databasecannot be accessed through the server, map your AS/400 system locally, andopen the LOG.NSF database as a local database.

13.Do errors appear in the QHST history log? Refer to 3.1.4.3, “Messages in QHST”on page 41, for more details.

14.Check the system library list. Type DSPLIBL at any AS/400 command line andpress Enter. The list should include QSYS and QSYS2, preferably at the top ofthe list.

15.Check for damaged semaphores and shared memory. Refer to 3.1.6,“Semaphores and shared memory” on page 42, for more details.

16.Do you have any memory or storage problems? Type WRKSYSSTS on the commandline, and press Enter. Does it reveal any problems?

17.How many users are on the system?

18.Are there non-Domino for AS/400 applications running?

19.Is directory synchronization active? For a quick check, type WRKACTJOB JOB(QNND*)

on the AS/400 command line. For more information, see 3.1.5, “Directorysynchronization” on page 41.

20.Is there any bad mail crashing the server? Refer to Chapter 11,“Troubleshooting e-mail delivery” on page 169, for e-mail troubleshooting.

21.Check for /QIBM/UserData/LOTUS/NOTES/DOMINO_SERVERS for anychanges being made on authority and ownership. Since R5.02, thisinformation is stored in /QIBM/UserData/LOTUS/LOTUS_SERVERS.


22.Is the job queue for Domino held? To determine if the job queue of yourDomino server is in hold status, complete these tasks:

a. Look for the subsystem name in which your Domino server is running byentering WRKDOMSVR in an command line. Press Enter.

b. Note the subsystem in which your server is running.

c. Type WRKJOBQ in the command line. Press Enter. The Work with All JobQueues display appears. Look for the subsystem in which your Dominoserver is running, and check for the status of the queue. It must have RLS(released) status.

23.Identify all other processes or programs running on this server, including:

• Any APIs running “against” this server either from the server or from aworkstation

An API does not need to be running on this server. It can be executed froma workstation.

• Any gateways

• Backup programs

• Executable scripts

• Third-party programs

All third-party applications should be documented, as well as any patchesor version upgrades.

8.3 An in-depth analysis

We now look at the information provided in the NSD file in case of a server crash.This data can be used as input for database research. Sometimes, it helps todetermine immediately what repair actions should be considered.

Keep in mind that this information varies with each crash. The approach to theproblem source is different with every failure. The NSD offers information for bothAS/400 and Domino specialists.

The NSD file has a rather fixed structure. However, it changed between Dominofor AS/400 Releases 4.6.x and 5.x.

In Release 5, the following sequence of information is available in the NSD file:

1. System name, OS/400, and Domino release2. Invocation stack trace3. Environment variables4. Job log of current job (Display Job Log)5. List of all active jobs (Work with Active Jobs)6. Status of current job (Display Job)7. Dump of IPC Semaphores and Shared Memory8. Dump of the NOTES.INI file9. Dump of Console Entries

According to the severity of the error and its impact on the Domino server and theNSD tool itself, the size and completeness of the information can vary from only afew lines up to hundreds of pages.

Chapter 8. Server crash 129

The following examples are taken out of a single NSD file.

8.3.1 System name, OS/400, and Domino releaseAt the beginning of the dump, we see the original name of the file and thetimestamp when it was created, for example nsd_20000211_07.48.39.nsd in theformat nsd_(julian date)_(time).nsd. See Figure 56.

If the server can start the NSD tool in case of a error situation, this timestampshows the time of the failure. We see the AS/400 system name, OS/400 release,and the Domino server release without the QMU.

Figure 56. General NSD information

8.3.2 Invocation stack traceThe invocation stack trace gives us the primary information of what happenedimmediately before, and at the time of, the error. See Figure 57.

Figure 57. Invocation stack trace out of NSD

The first line with the _C_pep entry shows you the area within Notes where thefailure occurred. Here, it would be within the router area.

Then, look for any entries, such as exception, kill, fatal, panic, or fault. At the firstentry within the call stack, look at the line above, and you see the last functionbeing executed. In the above example, it would be the select function forQP0LLIB1. This is the first point to start when you search the databases.

nsd_20000212_16.28.55.nsd for server: DOMSVR3System: as25 OS: OS400 Release: V4R4M0Notes Build: Release 5.0.1 (Intl)|16 July 1999

Invocation stack trace

_C_pep 0 NOTES0 ROUTER [144702]main 1notes_main 14NotesMain 4 NOTESAI0AddInMain 228 MAINOSDelayThread 1 THREAD LIBNOTESunix_usleep 6 USLEEPselect 2056 QP0LLIB1 QP0LLIB1fatal_error 29 BREAK LIBNOTESOSFaultCleanup 21 CLEANUPOSRunExternalScript 14__system_a 2 STDLIB_Asystem 6 QC2SYS QC2SYS_C_pep 0 NSD NSDmain 48


8.3.3 Environment variablesThe listing in Figure 58 shows the actual setting of any environment variable usedby Domino for AS/400 server. Some examples shown are the languageenvironment assigned to user QNOTES, the path, and fault recovery.

Figure 58. Display of environment variables out of NSD

Sometimes, not all environment variables are dumped to the NSD file. Therefore,you need to set the additional environment variable NOTES_AS400_STSDEBUG=1 to getthe information about all variables being used. This information is then logged tothe QNNINSTS job log at the time of server start. To set this environmentvariable, use the Work with Environmental Variables (WRKENVVAR) command.For details on how to use this command, see 3.1.7, “Environment variables” onpage 45.

8.3.4 Job log of a current jobThe job log of the current job is what is dumped at the time of the server crash.You should look for the AS/400 error messages that caused the error. Look forsuch message IDs as LNTxxxx, CPIxxxx, CPFxxxx, or any others in the firstcolumn of the job log that have a severity (third column sev) of 40 or more. If theseverity is 40 or above, the job is terminated. Note the message ID and anyadditional information of the message, such as called program or module for furtherresearch. Sometimes the error messages already indicates the problem source. SeeFigure 59 on page 132.

Environment Variables

NOTES_AS400_STSDEBUG=1LANG=/QSYS.LIB/QNOTES.LIB/EN_US.LOCALENOTES_AS400_SERVER_TYPE=1PATH=/QIBM/PRODDATA/LOTUS/NOTES:/DOMINO/DOMSVR3:/QIBM/USERDATA/LOTUS/NOTESNOTES_CONSOLE=QUSRNOTESNOTES_AS400_SBS_NAME=DOMINO01NOTES_AS400_SERVER_NAME=DOMSVR3NOTES_AS400_DATA_DIR=/DOMINO/DOMSVR3FaultRecovery=1Notes_SubprocessConfigFileName=NOTES_BUILD_ID=Release 5.0.1 (Intl)|16 July 1999


Figure 59. Job log out of NSD

8.3.5 List of all active jobsFigure 60 shows the Work with Active Jobs display of the entire system at thetime of the server crash. You may need to scroll through the data to find theinformation about the subsystem in which your server was running.

In the following example, the subsystem is DOMINO01. Because you may seedifferent subsystems running on this system, look for the subsystem informationat different areas within this NSD file, such as:

• Environment Variables (see NOTES_AS400_SBS_NAME=DOMINO01)• Display Job (see Job Status Attributes, Subsystem)

Job log of current job

1 5769SS1 V4R4M0 990521 Display Job Log AS25 02/12/00 16:28:55 Page 10 Job name . . . . . . . . . . : ROUTER User . . . . . . : QNOTES Number . . . . . . . . . . . : 144702

Job description . . . . . . : DOMINO01 Library . . . . . : QUSRNOTES-MSGID TYPE SEV DATE TIME FROM PGM LIBRARY INST TO PGM LIBRARY INST0CPI3BC4 Information 20 02/12/00 16:27:37 QLGSJATR QSYS 0326 QWTPIIPP QSYS 01CC

Message . . . . : Job attributes not set from locale.Cause . . . . . : The user profile associated with this job has theSETJOBATR value set to use some of the locale fields as job attributes. Ifthe value in the locale is not valid for a job attribute, the job attributewill be set from the user profile. Use the reason number to determine if ajob attribute was selected and set from the locale. Job attribute for CCSID-- Reason 0 Job attribute for date format -- Reason 0 Job attribute for dateseparator -- Reason 0 Job attribute for time separator -- Reason 0 Jobattribute for sort sequence -- Reason 1 Job attribute for decimal format --Reason 0 Reason 0 - Job attribute selected and set from locale. Reason 1 -Job attribute selected but not set from locale. Reason 2 - Job attribute notselected. See previous messages in the job log for additional information.

CPF1124 Information 00 02/12/00 16:27:37 QWTPIIPP QSYS 05E5 *EXT *NMessage . . . . : Job 144702/QNOTES/ROUTER started on 02/12/00 at 16:27:37in subsystem DOMINO01 in QUSRNOTES. Job entered system on 02/12/00 at16:27:37.

CPC1126 Completion 50 02/12/00 16:28:55 QWTCCCNJ QSYS 0432 *EXT *NMessage . . . . : Job 144702/QNOTES/ROUTER was ended by user ITSCID07.Cause . . . . . : User ITSCID07 issued a controlled end job request for job144702/QNOTES/ROUTER.


Figure 60. List of all active jobs in NSD

Look for any unusually high CPU usage counts if you expect a job loop. Then,look into the appropriate spooled file for more information.

8.3.6 Status of the current jobHere you see the snapshot of the status of the current and failing job at the timeof the crash of the server. For more detailed information about the contents, youcan look at AS/400 Work Management, SC41-5306.

All or some of this information can be helpful depending on each specific crash:

• Job status attributes: Shows the identifying characteristics and the status ofthe job.

• Job definition attributes: Shows the definition attributes in the jobdescription associated with the job.

• Job run attributes: Shows run attributes that are defined in the classassociated with the job.

• Spooled output files: Shows all the spooled files that are owned by thecurrent job.

• Job call stack: Lists information for all programs and procedures in the callstack. It gives the same information as the invocation stack trace. However,depending on the crash, it can happen that the invocation stack trace is notcomplete. You can then find the necessary information about the crash inhere. Look for any statements such as exception, kill, fatal, panic, or fault. Foran example, see Figure 61 on page 134.

List of all active jobs

1 5769SS1 V4R4M0 990521 Work with Active Jobs 2/12/00 16:28:55 Page 10 Reset . . . . . . . . . . . . . . . . . : *NOSubsystems . . . . . . . . . . . . . . : *ALLCPU Percent Limit . . . . . . . . . . . : *NONEResponse Time Limit . . . . . . . . . . : *NONESequence . . . . . . . . . . . . . . . : *SBSJob name . . . . . . . . . . . . . . . : *ALL

0 CPU % . . . : .0 Elapsed time . . . . . . . : 00:00:00 Active jobs . . . . . . : 270-

--------Elapsed---------Subsystem/Job User Number Type Pool Pty CPU Int Rsp AuxIO CPU% Function Status Threads

0 DOMINO01 QSYS 143945 SBS 2 0 .1 0 .0 DEQW 1ADMINP QNOTES 144707 BCI 2 20 .7 0 .0 SELW 1AMGR QNOTES 144705 BCI 2 20 9.5 0 .0 SELW 1AMGR QNOTES 144706 BCI 2 20+ .9 0 .0 SELW 1BILLING QNOTES 144700 BCI 2 20 .2 0 .0 SELW 1CALCONN QNOTES 144709 BCI 2 20 .2 0 .0 SELW 1DECS QNOTES 144711 BCI 2 20 1.2 0 .0 SELW 1EVENT QNOTES 144710 BCI 2 20 .5 0 .0 SELW 13HTTP QNOTES 144712 BCI 2 20+ 2.2 0 .0 SELW 51NOTESMM QNOTES 143991 BCI 2 20 6.5 0 .0 SELW 1NSD QNOTES 144720 BCI 2 20 .1 0 .0 RUN 1QNNINSTS QNOTES 144698 BCH 2 20 .0 0 .0 PGM-QNNINSTS EVTW 1QRWTSRVR QNOTES 144716 BCI 2 20 .3 0 .0 TIMW 1REPLICA QNOTES 144701 BCI 2 20 .3 0 .0 SELW 1ROUTER QNOTES 144702 BCI 2 20 .7 0 .0 EVTW 1SCHED QNOTES 144708 BCI 2 20 2.0 0 .0 SELW 1SERVER QNOTES 144699 BCI 2 20 1.7 0 .0 SELW 72STATS QNOTES 144704 BCI 2 20 .4 0 .0 SELW 1UPDATE QNOTES 144703 BCI 2 20 .5 0 .0 SELW 1

QBATCH QSYS 143676 SBS 2 0 .0 0 .0 DEQW 1QCMN QSYS 143679 SBS 2 0 .0 0 .0 DEQW 1QACSOTP QUSER 143706 PJ 2 20 .0 0 .0 PSRW 1QLZPSERV QUSER 143707 PJ 2 20 .0 0 .0 PSRW 1QNMAPINGD QUSER 143698 PJ 2 25 .0 0 .0 PSRW 1


Figure 61. Job call stack in NSD

• Job locks, job member locks, job record locks: Shows all external objectlocks held by the current job, including both held locks and locks being waitedfor.

• Job library list: Shows the names, types, and text of the libraries that are inthe job's system and user library list. Check for the QSYS and QSYS2 libraries,preferably at the top of the list. See Figure 62.

Figure 62. Job library list in NSD

• Display open files: Shows the files that are open for the specified job and thestatus of system or user files.

• All file overrides: Shows file overrides at any active call level for the currentjob. All file overrides, or file overrides for a specific file name, can be shown.

- Job Call Stack0 Thread: 000000020Rqs Program or ILE ILE CtlLvl Procedure Library Statement Instruction Activation Group Module Program Bdy0 _CXX_PEP QSYS 0000000139 0000000001 *DFTACTGRP QP0ZSPW2 QBUILDSS1 QP0ZSPW2 Y

main QSYS 0000000127 0000000001 *DFTACTGRP QP0ZSPW2 QBUILDSS1 QP0ZSPW2 N< SpawnChild QSYS 0000000372 0000000001 *DFTACTGRP QP0ZSPWC QBUILDSS1 QP0ZSPWC N

Procedure: Qp0zSpawnChild< d__FPviPPc QSYS 0000000469 0000000001 *DFTACTGRP QP0ZSPWC QBUILDSS1 QP0ZSPWC N

Procedure: Invoke_Child__FPviPPc_C_pep QNOTES 0000000097 97 NOTES0 Q1NOTESLIB ROUTER Ymain QNOTES 0000000001 0000000097 97 NOTES0 Q1NOTESLIB ROUTER Nnotes_main QNOTES 0000000014 0000000097 97 NOTES0 Q1NOTESLIB ROUTER NNotesMain QNOTES 0000000004 0000000097 97 NOTESAI0 Q1NOTESLIB ROUTER NAddInMain QNOTES 0000000228 0000000097 97 MAIN Q1ROUTER ROUTER N

< elayThread QNOTES 0000000001 0000000097 97 THREAD Q1OS LIBNOTES NProcedure: OSDelayThread

< nix_usleep QNOTES 0000000006 0000000097 97 USLEEP Q1LIBUNIX LIBNOTES NProcedure: unix_usleep

select QSYS 0000002056 0000000097 97 QP0LLIB1 QBUILDSS1 QP0LLIB1 N< atal_error QNOTES 0000000029 0000000097 97 BREAK Q1OS LIBNOTES N

Procedure: fatal_error< ultCleanup QNOTES 0000000021 0000000097 97 CLEANUP Q1OS LIBNOTES N

Procedure: OSFaultCleanup< rnalScript QNOTES 0000000014 0000000097 97 CLEANUP Q1OS LIBNOTES N

Procedure: OSRunExternalScript__system_a QNOTES 0000000002 0000000097 97 STDLIB_A Q1LIBASCII LIBNOTES Nsystem QSYS 0000000006 0000000097 97 QC2SYS QBUILDSS1 QC2SYS NQCMDEXC QSYS 011F 0000000001 *DFTACTGRP_C_pep QNOTES 0000000097 97 NSD Q1NSD NSD Ymain QNOTES 0000000057 0000000097 97 NSD Q1NSD NSD NNSDPart2 QNOTES 0000000032 0000000097 97 NSD Q1NSD NSD N

< p0zWaitpid QSYS 0000000007 0000000097 97 QP0ZWAT QBUILDSS1 QP0ZSPWN NProcedure: Qp0zWaitpid

< 0zSWaitpid QSYS 0000000621 0000000001 *DFTACTGRP QP0ZSWAT QBUILDSS1 QP0ZSSPW NProcedure: Qp0zSWaitpid

Job Library List-- Library Type Text0 TSYS SYS TOOLS product command library

QSYS SYS System LibraryQSYS2 SYS System Library for CPI'sQHLPSYS SYSQUSRSYS SYSTUSRSYS SYS TOOLS product user object libraryQNOTES USRQGPL USRQTEMP USR


• Commitment control status: Shows a list of commitment definitions activefor the current job. A commitment definition can be established by the StartCommitment Control (STRCMTCTL) command or by other system functions.

• Display communications status: Shows status information for thecommunications connections the Domino server has with a remote system orprogram.

• Job activation groups: Shows information for all activation groups for thisjob.

• Job mutexes: Shows all mutexes associated with a specific thread, includingmutexes that are held and the mutex for which the thread may be waiting.When displaying mutexes for a job, only mutexes associated with the initialthread are displayed. The Work with Threads or display threads option shouldbe used to obtain mutex information about secondary threads within a job.

Figure 63. Job mutexes in NSD

• Threads: Figure 64 shows the list of threads in the current job.The first thread in the list is the initial thread for the job. Some processes arecapable of supporting additional threads, which are called secondary threads.

Figure 64. Thread display in NSD

To get the information about all possible thread status, place the cursor on thethread status field in the Work with Threads display (Figure 17 on page 37),and press F1.

• MLB (media library) attributes: Shows the resource allocation attributes of amedia library device.

8.3.7 Dump of IPC semaphores and shared memoryThis shows shared memory at first, and then the semaphores.

Look into the mode field for shared memory and the semaphores to check if anyof them are damaged. If it is damaged, it will have a “D” in the mode field asshown in the spooled file sample in Figure 22 on page 44 (line beginning with S

1055 0XF6).

Job Mutexes-- MutexStatus Mutex

0 Held QC2_SYSTEMF_MTX

1 5769SS1 V4R4M0 990521 Display Job 2/12/00 16:28:56 Page 70 Job . . . : ROUTER User . . . : QNOTES Number . . . . : 144702- Threads0 Total Aux RunThread Status CPU I/O Priority00000002 EVTW .733 207 20


If you find a damaged semaphore within your dump, and your Domino server canonly be restarted after an IPL, and this happens again, you can avoid the IPL byremoving the damaged semaphore. To do so, perform the following steps:

1. To confirm the damaged semaphore in the NSD file, you should again dumpthe IPC by using:

CALL PGM(QP0ZIPCS) PARM('-Emsa')

2. Look in the spooled file for the damaged semaphore, indicated with the “D” inthe mode field, and note the ID number, which is in the example.

3. Use the following command to remove a damaged semaphore. It allows theDomino server to restart without IPLing the AS/400 system:

CALL PGM(QP0ZIPCR) PARM('-Xs1055')

Please note the parameter ID would be substituted by the actual value of theID, which is 1055 in our case. The complete call would appear as:

CALL PGM(QP0ZIPCR) PARM('-Xs1055')

For more information, see IBM APAR SA70795.

8.3.8 The NOTES.INI fileThis section provides a complete listing of the NOTES.INI file as it is read by thesystem. Check for any of the following occurrences:

• ServerTasks=: Look for what tasks are being loaded.

• ServerTasksAtXX: Look for what tasks are loaded at the specified times.Compare these ServerTasksAt lines with any Program documents or batchfiles the customer may be running. If the customer has stacked processes(multiple processes scheduled to run at the same time), there could beproblems with multiple tasks kicking off at the same time.

• NSF_BUFFER_POOL_SIZE= or NSF_BUFFER_POOL_SIZE_MB=: Thissetting must be exact in bytes or megabytes. An invalid setting can haveunpredictable results such as corrupted mail files. Setting the NSF Buffer Poolshould always be discussed with a member on the Performance team. Thisparameter is discussed in more detail in 3.2.1.1, “Domino server consolecommands” on page 48.

• Directory=: Check for forward slashes (/) and not backslashes (\)

• Debug parameters: Check for any debug parameters set and why they areset.

The following command should only be used by an analyst experienced inthe OS/400 operating system. This command has the potential to causeoperating system interruption, among other issues.

Important

All Domino servers should be stopped before using these commands toavoid potential problems.

Important


For a listing of published debug parameters settings for the NOTES.INI file,see Appendix C, “Debugging settings” on page 287.

• Third-party software programs

• Look for parameters that may have been set and are causing problems or areno longer useful, for example:

– Server_Session_Timeout set too low.

– Multiple Updaters (Indexer) set up?

– Debug parameters that may have been set for previous problems that areno longer valid (for example, NSF hooks from third-party applications).These can have negative effects on the server's performance if they arenot set correctly.

8.3.9 Dump of console entriesThe dump of the console entries shows all data that was written to the consolebefore the server crashed. Here, you would look for:

• Specific error messages that may have been displayed when the server wasstarted up before the crash.

• Is there any special activity of the server? What is the server doing? Do yousee any trends?

• The time of the crash. Was it on the hour? Were there any batch files, backupprograms, third-party applications, server tasks, or program documentskicking off at the time of the crash? What was the time that the server wasrestarted?

• Can you detect any odd behavior? Did any task kick off before the crash thatdid not finish?

8.3.10 Successfully processed console entriesBased on our experience with server crashes in the past, the messageSuccessfully processed console entries is displayed when the NSD tool coulddump all available information from the Domino server.

However, if the server is constantly rebooting, the NSD files may contain onlysome parts of the above information. Therefore, check all dumped files, eventhough they seem to be empty, for any information that may be of interest for theDomino for AS/400 analyst to solve the problem.

8.4 Server crash and automatic restart

An exception (unhandled error) in the Domino server or add-in programs thatruns under the server usually leaves the server in an unpredictable condition.Therefore, if an exception occurs, Domino for AS/400 automatically cleans up theserver and restarts it. The clean-up processing ends all server and add-in jobs.

You can add two AS/400 environment variables to control this automatic recoveryand the number of attempts to restart the server:

• The FaultRecovery environment variable specifies whether automaticrecovery is on or off. When you set up a Domino server on AS/400, automaticrecovery is on. If you want to turn off automatic recovery, set the


FaultRecovery value to 0. For example, you may want to turn off automaticrecovery to debug add-in jobs.

If this variable is set to a value other than 0, it defaults to 1 and the serverrestarts if a 5 completion code is received on the server end. A 5 completioncode indicates that the job went down and the server went down as a result.When a user ends the server (via quit), the completion code is a 0, and the jobdoes not restart.

• The AS400_DOMSVR_RESTART environment variable specifies the numberof restart attempts after an exception occurs. Initially, Domino for AS/400 triesup to 10 times to restart the server after an exception. You can useAS400_DOMSVR_RESTART to change the value to a larger or smallernumber.

Use the Add Environment Variable (ADDENVVAR) command or the Work withEnvironment Variable (WRKENVVAR) command, to add these environmentvariables and set their values. See 3.1.7, “Environment variables” on page 45, tolearn how to set environment variables within your job or for the entire system.

8.5 Troubleshooting examples and scenarios

These examples should give you an idea of what problems can occur and how anadministrator would assess and solve such problems.

Server crashes continuouslyIf a server is crashing continuously (for example, every five minutes), a usefultroubleshooting step is to remove the ServerTasks= line from the NOTES.INI filetemporarily. The server can than be restarted and tasks can be loadedindividually to determine which process is causing the crash.

Server is crashing on the Router taskIf the server crashed on the Router task, it could be an issue with the MAIL.BOXfile. An easy first step in troubleshooting Router crashes is renaming theMAIL.BOX file because it is susceptible to corruption. Rename the MAIL.BOX fileand restart the server. By renaming the MAIL.BOX file, you can preserve anymessages currently in MAIL.BOX, in case it is necessary to try and resend themessages to prevent data loss. If renaming the MAIL.BOX does not resolve theproblem, adding the Log_Mailrouting parameter to the NOTES.INI file is anotheroption. See 11.2.10, “Settings in the NOTES.INI file to obtain error reporting” onpage 203. You can also increase the log level.

Server is crashing on the Update taskIf the server crashed on the Update task, look in the log file to see what databasewas accessed last by the Update task. If nothing can be determined from the log

A mail message that crashes a server may not be corrupt. It may be due toinsufficient resources, especially in the case of an extremely large message(for example, with file attachments). A restart of the server (thus dropping allsessions on the server and freeing up resources) may let the message passwithout any problem.

Note


file, refer to Administering the Domino System (Volume 1 and 2) for loggingparameters for the Update task (Log_update, Log_view_events). Theseparameters provide detailed information on what the Update task is doing in theevent of another crash. The Update task is the most resource intensive of theNotes tasks. If the log information does not reveal anything, explore performanceproblems (for example, review the STATREP.NSF).

Server is crashing on Agent ManagerExamine the log to see if any agents were executing at the time of the crash. Ifnot already implemented, add Log_AgentManager=1 to the NOTES.INI file.

The crash is database or view relatedOften times a server can crash on a particular task, but the task is not theproblem. There may be a database that the process is acting upon that causesthe server to crash. A corrupt database can cause a server to crash. If there is anNSD file, examine the failing thread to see if a database is called out. This is mostlikely the offending database (although not always).

In the case of database corruption, we do not recommend running FIXUP,UPDALL, and COMPACT just because the database is corrupt. If possible,determine the nature of the corruption. For example, if the server is crashing on adatabase while updating the database, the most likely cause is a corrupt view. Inthis case, it is only necessary to run UPDALL, not FIXUP. This will save time andhelp reduce the effect on a production server. The Compact task is specificallyused to remove white space. It does not help with database corruption.

QNNINSTS and SERVER jobs start but end immediatelyIf starting Domino server shows the QNNINSTS and SERVER jobs start but thenend almost immediately, there may be a potential problem with damagedsemaphores or shared memory. You should confirm this by using the CALLQP0ZIPCS -Emsa command, as described in 3.1.6.1, “Displaying through a 5250interface” on page 43. Then, look for “D” in the spooled file, which indicatesdamage in either a semaphore or shared memory.

Domino R5.x server crashes when HTTP tasks load and latest Javagroup PTF is appliedThis issue may be due to corruption of the view for HTTP in the Name andAddress Book (NAB). It can be fixed by FIXUP and UPDALL -R or a restore frombackup of the Domino directory.

Adding verbose logging should be done only when troubleshooting problems.These parameters should be removed or reverted back to their original loglevel when the problem is resolved or the task is ruled out as a potentialproblem.

Note


Chapter 9. HTTP server troubleshooting

The HTTP server can incur such problems as crashes or hang conditions. Hangconditions can also lead to the server being restarted by the administrator.

9.1 Troubleshooting an HTTP crash

The HTTP task can crash for a number of different reasons, including:

• Agents or CGIs

• Bad icons or images

• Corrupted data

• Unusual form properties, for example, controlled access sections, hide-whenformulas, etc.

• Corrupted disk cache (R4.6.x)

• Corrupted httpd.cnf file (see 7.3, “The Domino Web Server configuration file(httpd.cnf)” on page 120)

9.1.1 Where to look firstFirst, make sure the server has the most recent operating system updates andpatches. See 8.2, “A beginner’s checklist” on page 127, for a quick check to seethat all prerequisites are fulfilled.

Ensure that HTTP is, in fact, in the crashing process Review the NSD file to makethis determination.

You should also perform these quick checks:

1. Check in the server document in the section Internet Protocols to see if therehave been any changes to the HTTP settings.

2. Check whether someone started the HTTP server included in TCP/IP for theAS/400 system, and if the HTTP server is using the same port as your HTTPtask within Domino for AS/400.

9.1.2 Troubleshooting a crashOften, a specific URL causes a crash, either because it is corrupted or has anattribute that exposes a bug in the HTTP code. The intent in troubleshootingHTTP crashes is to gather diagnostics for every crash to determine:

• What the server was doing when it crashed• What URL the server was processing at the time of the crash

It is important to discover a pattern to the URL if there is one. Appropriatediagnostics are needed to pinpoint a problematic URL, or to provide an idea ofwhat the server was doing at the time of the crash. Generally, it takes a minimumof three crashes to detect a pattern to the crash.

Troubleshooting crashes includes disabling any suspected agents (by removingthe Action button or WebQueryOpen contents), clearing the disk cache,performing database maintenance, rebuilding views, and so on.


9.1.3 What data is neededThe data files covered in the following sections must be collected for every crash.

9.1.3.1 NSD fileAs with conventional crashes, you should collect as many files as possible (at leasttwo to three) to establish a pattern. For detailed information on the NSD file, see3.2.4, “NSD diagnostic tool” on page 57, and 8.3, “An in-depth analysis” on page129.

9.1.3.2 Domino Web server log database DOMLOG.NSFTo help determine the flow of Web activity on the server, you should enable logging tothe Web server log files, the Domino Web server log database, or both. Thislogging is set in the server document.

To activate the logging, you perform the following steps:

1. Open the Names and Address Book of the server.

2. Go to the Server - Servers section and double-click your server document.

3. Click Edit Server.

4. Click the Internet Protocols tab.

5. Go to the Enable Logging To section and set either:

• Log files to Enabled• Domlog.nsf Enabled• Both log files and domlog.nsf to Enabled

Within this section, you can also modify the log settings and the log filenames and locations.

6. Click Save and Close to leave the document

7. To let these changes take effect, you have to recycle the HTTP server. Youcan do this by typing TELL HTTP RESTART on your Domino console. Press Enter.

If logging is enabled, an entry is written to the Domino Log for all Domino Webserver activities. The entry includes information that is relevant to each clientrequest, including:

• Date and time the request was made

• User's IP address (or the DNS address if DNS lookup is enabled in the Serverdocument)

• User's name (if the user supplied a name and password to access the server)

• Status code the server returns to the browser to indicate its success or failurein generating the request

• Length of the information, in bytes, sent from the server to the browser

• Type of data accessed by the user (for example, text/html or image/gif)

• HTTP request sent to the server from the browser

• Type of browser used to access the server

• Internal and Common Gateway Interface (CGI) program errors

• URL the user visited to gain access to a page on this site


• Server's IP address or DNS name (useful for virtual server statistics, availablein Domino R4.6 and later)

• Amount of time, in milliseconds, to process the request

See Figure 65 for an example of the Domino Web server log.

Figure 65. Domino Web server log

The Domino log database is usually the best method for logging activity on theDomino Web Server if:

• You want to generate reports from Notes views.

• You want to trigger workflow events, such as sending mail when certain eventsoccur.

9.1.3.3 Domino Web server log text filesDomino Web server logging can also be set to be logged to text files. All files arecreated after modifying the server configuration document and restarting theserver. You find them in the location specified in the server document in Directoryfor log files in the HTTP section. The files stay empty until the HTTP server isstopped (TELL HTTP QUIT, TELL HTTP RESTART, or Domino server stop).

The name of the generated files consists of the name specified in the settingsand the day it is created in the format (NameDDMMYYYY.log). For example,Access01252000.log was created 25 January 2000.

The Domino Web server log text files include:

• ACCESS-LOG: The first field of each records shows the IP address of theuser.

Chapter 9. HTTP server troubleshooting 143

The third field shows the name of a user who had to connect with a userprofile and a password to the Web server,

The last field of each records shows (or tries to indicate) information about thebrowser used to access the Web server.

• AGENT-LOG: The agent log keeps a record of the Web browsers that accessthe server.

• REFERER-LOG: The referrer log has records indicating sites that containlinks to this site.

• ERROR-LOG: The error log records internal server error and CGI programerrors.

• CGI-ERROR-LOG: The CGI error log records the Common Gateway Interfaceerrors.

The text files are usually the best method for logging activity on the Domino WebServer if:

• The Web site is large and frequently accessed.• Other HTTP server reporting and management tools are already being used.

To display the text log files, you use the Display File (DSPF) command on anyAS/400 command line, for example DSPF Access01252000.log. Details about theDisplay File (DSPF) command can be found in 7.1, “Displaying or editing astream file from a 5250 display” on page 117.

There are also tools available to analyze these text log files. These tools areproposed by Lotus business partners and have not been checked within thisredbook.

Some information about these tools can be found in an article in IRIS Today,entitled “Logging your Web Site traffic”. This article can be browsed at:http://www.notes.net/today.nsf

In the Archives section, click Perform a full-text search of all Iris Today, andsearch for “Logging your Web Site Traffic”. The author is by Barbara Burch, the lastupdate is 01 August 1998.

Logging to text files is useful when the site is large and busy or when the siteuses a third-party HTTP server-reporting and management tool. In addition, youcan set up workflow events. To help manage the information in the database, forexample, you can set up an event that sends mail when a page is accessed morethan a specific number of times. Logging to a database is somewhat slower thanlogging to text files, especially at busy sites.

9.1.3.4 Notes log and Domino consoleConsole output is preferred since this data is typically not buffered. In some cases,errors display only on the console. You can redirect console output to a text file by:

• Using the DEBUG_OUTFILE parameter in the NOTES.INI file. This parameteris documented in Appendix C, “Debugging settings” on page 287.

Note: This parameter requires a restart of the Domino server to take effect.

• Dumping the Domino console to a spooled file (see 3.2.1, “Domino console”on page 47).


9.1.3.5 The NOTES.INI file and server documentThese documents are needed to determine if any abnormal HTTP settings areaffecting the crash.

9.1.3.6 REQ###.LOG fileREQ###.LOG files are created for each active thread when dynamic threadlogging is activated (see 9.1.4, “Dynamic thread logging” on page 145, for moreinformation).

These files are named REQ###.LOG, where ### is the thread ID for the activethread. It records specific URL requests on the Web server. It is used to trackwhich URL has caused a crash or hang. It should be deleted regularly, until thecrash or hang happens (it can become very big).

9.1.4 Dynamic thread loggingDomino R5 provides a dynamic method of enabling thread logging that wasintroduced on Domino 4.6x and later. For R5.0 and R5.01, this logging wrote alldebugging information into a single file, named, for example, req2147483648.logon our AS/400 test system. Since R.5.0.2a, a REQ###.log file is created for eachsingle task that is started during an HTTP session.

To set up this thread logging, you have two possibilities:

• For permanent debugging, perform these tasks:

a. Edit the HTTPD.CNF file located in the server data directory using the EditFile (EDTF) command.

b. Add the following debug parameter line entry to the end of the file:

debugthreadlogging on

c. Restart the HTTP task by typing tell http restart on the server console.

• To set up thread logging temporarily, enter the following command at theserver console:

tell http debug thread on

This dynamically sets the thread logging debug flag, and the server begins tocreate thread logs immediately.

However, this debug flag remains in effect only until the HTTP server isrestarted. This method of turning on debug does not place the parameter inthe HTTPD.CNF. Therefore, with the dynamic method of enabling threadlogging, the HTTP task will not continue to update the REQ###.LOG files uponrestart. In order for this parameter to remain in effect over several restarts ofHTTP, the debugthreadlogging on parameter must be placed in theHTTPD.CNF file.

You would use the dynamic thread logging temporarily when you want toprevent too much data in the log files, because you know that the failure onlyoccurs at or after a certain time. The tell command lets you easily switch thelogging on or off without interfering with the HTTP server.

You can stop the thread logging for the actual HTTP server session by typing thefollowing command on the server console:

tell http debug thread off


Keep in mind that this command does not edit the HTTPD.CNF file.

A REQ###.LOG file is created and updated each time there is an HTTP request.There will be no more log files besides the number of active threads defined inthe server document, section Internet Protocol and HTTP settings. If themaximum number of threads (Default = 40) and, therefore, also the number of logfiles is reached, the new information will be appended to the existing log files. Ifyou set debug thread logging on in a heavily used Web server environment, it caneasily create very large files. These log files can be deleted as long as there is noserver crash or hang occurring. They can be deleted while the server is running,and new logs will be created dynamically.

See Figure 66 for an example of a REQ###.log file. Note the following points:

• The URL being served is GET /eBoats.nsf/Products?OpenView

This is the starting point for a new browser request being served by thethread.

• The time stamp and date are Tue, 22 Feb 2000 01:09:47 GMT.

For further problem determination, you have to review the NSD file and theREQ###.LOG files to see what URL the server was processing at the time of acrash or hang. The suspected URL is the last request listed in the specificREQ###.LOG file for the faulting or hung thread.

Domino logging (Access Log or DOMLOG.NSF) is buffered. In a crash or hang,you cannot rely on the last URL listed in these logs to be accurate. This is whyREQ###.LOG files are required for troubleshooting.

If you know the start time of HTTP (from server statistics or from the Log), youcan look for a correlation between the requests from the REQ###.LOG files torequests in the Access Log or DOMLOG.NSF and confirm that the requestscorrespond to the crash or hang time. Domino R5 writes the date and time stampto the REQ###.LOG files, so this information is now available for troubleshooting.

There was a change in the Domino coding from R4.6 to R5.02. That means theactual thread number is not given as a decimal number in the ### field of theREQ###.log file name. What does this mean?

In 4.6. and from 5.03 on, this value was and will be a decimal value to beconverted to hex code to match the thread ID in the NSD file. In R5.01andR5.02 this value does not match the actual thread. In these releases, you haveto check, using the time stamp of the REQ###.log file, to see if it matches withthe time of the HTTP crash or hang.

Note


Figure 66. REQ###.log file display

9.2 HTTP hang

Unfortunately, HTTP hangs are much more difficult to troubleshoot, due to thelack of complete diagnostics. More investigative work is required with HTTPhangs.

Hangs can result from a number of issues, including:

• CPU spin (excessive view rebuilds, corrupted views, or documents)

• Excessive agent executions or agent hangs

• DNSLookup enabled in the Server document

• Semaphore timeout issues

• Disk thrashing

• Network or port binding problems

************Beginning of data**************GET /eBoats.nsf/Products?OpenView HTTP/1.0Referer: http://10.8.62.13:8080/eboats.nsf/NavSideMenu?OpenNavigator&BaseTarget=rightUser-Agent: Mozilla/4.7 [en] (WinNT; I)Host: 10.8.62.13:8080Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*Accept-Encoding: gzipAccept-Language: enAccept-Charset: iso-8859-1,*,utf-8Forwarded: by http://proxy1.site3.company.com:81 (Netscape-Proxy/3.52)

HTTP/1.1 200 OKServer: Lotus-Domino/5.0.2Date: Tue, 22 Feb 2000 01:09:47 GMTConnection: closeContent-Type: text/html; charset=US-ASCIIContent-Length: 1147

<HTML><HEAD><META NAME="robots" CONTENT="noindex">

<SCRIPT LANGUAGE="JavaScript"><!--document._domino_target = "_self";function _doClick(v, o, t) {var url="/eBoats.nsf/Products?OpenView&Click=" + v;if (o.href != null)o.href = url;

else {if (t == null)t = document._domino_target;

window.open(url, t);}

}

• Too much traffic on the server (can be determined from Domino statistics thatindicate the number of requests serviced for an interval)

• Whether the system is still sized for the increasing usage by more and moreusers

To determine the problem, you have to know the exact circumstances of theproblem. You have to determine if the server is truly in a hang state or is simplyencountering performance problems that cause it to become unresponsive for acertain period of time.

One way to determine if the behavior is only a performance slow down and not atrue hang is to remove any REQ###.LOG files from the program directory duringthe hang state. If new REQ###.LOG files are created, the server is activelyserving up new requests. If the Domino Web server creates new REQ files, theserver has not actually hung, but has experienced a performance slow down. Ifthis is the case, you should troubleshoot such an issue as a performanceproblem, and not as a hang. If no new REQ###.LOG files are created over thecourse of a minute or two, the server is in an actual hang state, indicated by thefact that no new Web browser requests are getting serviced, not even queuedrequests.

The files to collect for every hang are:

• REQ###.LOG files

• Access log or DOMLOG.NSF

• LOG.NSF

• Console output with DEBUG_THREADID set before the hang (see 9.1.3.4,“Notes log and Domino console” on page 144)

• DMPJOB

• CALL STACK of all HTTP threads

• Output of Domino console command Tell HTTP Show Thread State at the time ofthe hang condition

• SEMDEBUG.TXT (relevant for R4.6x)

These additional files can also be used:

• NOTES.INI file and Server document• STATREP.NSF

A majority of HTTP hangs are caused by a hang of a Web-triggered agent. Anagent can be executed via the Web in one of three ways:

• Using the URL http://host/db.nsf/AgentName?OpenAgent

• An Action button using the formula @Command([ToolsRunMacro])

• WebQueryOpen or WebQuerySave events of a Notes form

Because REQ###.LOG files are appended to, and one is created for, eachthread, it is critically important that you move the REQ###.LOG files from theprogram or data directory after each hang so that you have the most isolated datafor each hang. If we do not move these files and then restart the HTTP server,certain thread IDs may be reused, which will cause the thread to append newrequest data to existing REQ###.LOG files. This hampers the ability to determine


which GET requests are relevant to the hang, as well as which REQ###.LOG filesare valid.

Keep in mind, when you discover the hang condition, move the REQ###.LOGfiles first and then restart the HTTP server by issuing the TELL HTTP RESTART

command on the Domino server console.

9.3 New HTTP Web server commands in R5

New HTTP Tell commands for the Web server (HTTP) can be issued at the serverconsole or from the Domino Administrator since Domino Release 5.0. The outputis logged to the LOG.NSF database.

The following list highlights the new commands. Additional information can befound in the Domino 5 Administration Help.

• Tell HTTP Show Thread State: Domino R5 introduces a new, undocumentedconsole command that helps troubleshoot HTTP hangs on R5. When enteredat the console, this command displays the current status of each active threadand which URL, if any, the thread is processing. This output is logged in thelog.nsf. Figure 67 shows a sample output for three threads. The first twothreads are idle. The third thread is processing the URL GET

/reference.nsf/Refresh?OpenAgent HTTP/1.0.

Figure 67. Tell HTTP show thread state

If the server is in a hung or partially hung state, this command can be used todetermine if a particular thread has been processing the same URL for toolong. If the thread is still processing the same request or URL for more than afew minutes, the thread is likely in a hung state.

In many cases, if the HTTP task is hung, customers attempt to shut the HTTPserver down, but the task does not shut down gracefully. In Domino R5, whenan administrator issues the command Tell HTTP Quit, if HTTP is waiting for ahung thread to complete during shutdown, HTTP outputs this thread ID andthe URL it is working on to the console.

This information can be used to determine the hung thread and which URL thethread is processing, in the same way the REQ###.LOG files can be used.The thread ID can be correlated against the REQ###.LOG file that pertains tothat thread. This new feature helps to decrease the amount of time it takes todetermine the problem URL that may cause a hang.

> tell http show thread state02/23/2000 10:25:12 AM HTTP Thread State: HiId: [0] LoId: [1cc] Handle:[218] State: [Worker waiting for work] reservedC: [ff] Other Info:

02/23/2000 10:25:12 AM HTTP Thread State: HiId: [0] LoId: [1d0] Handle:[221] State: [Worker waiting for work] reservedC: [103] Other Info:

02/23/2000 10:25:12 AM HTTP Thread State: HiId: [0] LoId: [1c5] Handle:[171] State: [Worker processing request] reservedC: [f8] Other Info:GET /reference.nsf/Refresh?OpenAgent HTTP/1.0


• Tell HTTP Show Users: This command can be used only if the server isconfigured to use session-based tracking for the Web. This command showsthe User Name, IP address, and the time of expiration, which is 30 minutes bydefault. The results display only users who are authenticated. Anonymoususers cannot be tracked.

• Tell HTTP Restart: This command results in the HTTP task shutting down,reloading, and refreshing certain Web server settings. It is the equivalent toissuing Tell HTTP Quit followed by Load HTTP. However, the down time for theserver is much shorter because it is not completely reloading all HTTPmemory. This command deletes the in-memory page and user-authenticationcaches.

• Tell HTTP Show File Access: This command displays information about filesystem protection on the machine and each virtual server if configured.

• Tell HTTP Show Security: This displays the current status on the use of SSLfor the server and each virtual server or virtual host.

• Tell HTTP Show Virtual Servers: This command displays a list of allconfigured virtual servers or virtual hosts running on the server.

See Figure 68 for an example of the new HTTP Tell commands.

This undocumented Tell HTTP Show Thread State command can crash theDomino for AS/400 server R5.0a. This problem is fixed in R5.0.1.

Important note


Figure 68. Example of new Tell HTTP commands

> Tell HTTP Show File Access

02/23/2000 08:56:27 AM Base server:02/23/2000 08:56:27 AM Redirect URLs with path / to /homepage.nsf?Open02/23/2000 08:56:27 AM Execute files in/lotus/domino/DOMSVR5/domino/cgi-bin/* via URLs with path /cgi-bin/*02/23/2000 08:56:27 AM Execute files in /QSYS.LIB/QNOTES.LIB/*.PGM viaURLs with path /adm-bin/*02/23/2000 08:56:27 AM Path protected: /QSYS.LIB/QNOTES.LIB02/23/2000 08:56:27 AM Get files in /lotus/domino/DOMSVR5/domino/icons/*via URLs with path /icons/*02/23/2000 08:56:27 AM Get files in /lotus/domino/DOMSVR5/domino/java/*via URLs with path /domjava/*02/23/2000 08:56:27 AM Execute files in/lotus/domino/DOMSVR5/domino/servlet:/lotus/domino/DOMSVR5/domino/servlet/jt400.jar/*via URLs with path /servlet/*02/23/2000 08:56:27 AM Get files in /lotus/domino/DOMSVR5/domino/html/*via URLs with path /*02/23/2000 08:56:27 AM File system protection configuration errors:02/23/2000 08:56:27 AM Path NOT protected: /QSYS.LIB/QNOTES. File systemaccess control path is not related to any URL access paths

> Tell HTTP Show Security

02/23/2000 08:57:39 AM Base server:02/23/2000 08:57:39 AM SSL enabled02/23/2000 08:57:39 AM Key file name: /lotus/domino/DOMSVR5/keyfile.kyr02/23/2000 08:57:39 AM Secure server started

> Tell http show users

02/23/2000 09:02:14 AM There are 2 current HTTP user sessions02/23/2000 09:02:14 AM User Name IP Address Expires02/23/2000 09:02:14 AM User1 Test 10.8.62.72 10:01:47 AM02/23/2000 09:02:14 AM Redbook User 10.8.62.69 10:01:24 AM


Chapter 10. Threads and semaphores

A job can be hung due to threads. The parent thread waits for a termination signalfrom one or more of its children. When looking at the job via the Work withThreads screen, only the parent thread is visible. What may have happened isthat the child thread may have ended abnormally while holding an exclusive lockon a gate. This has only been seen with the server job so far, but could happen toany multi-threaded job.

APAR SA76447 and MA18737 were created to address this problem. APARMA18737 is specific for a TCP module at V4R3 that caused this problem. APARSA76447 is a request for the ENDJOBABN command to be changed to handlethis situation, so an IPL is not necessary.

After trying the ENDDOMSVR *CNTRLD, ENDJOB, and ENDJOBABNcommands, the job is still running. Only one thread is visible on the Work withThreads screen, and it is probably in a THDW status. You then need to get thefully qualified job name and take a process dump of the job.

A job can also be hung due to semaphores. For example, cancel handler calls ablocked semaphore operation which causes errors and faults so that the jobcannot end. This has only been seen with the Server job so far, but can happen toany Domino job. This has been fixed in Domino Release 4.62.

This chapter gives an insight in two different techniques that help solve jobhangs:

• Gathering call stack information on a secondary thread• Handling semaphore timeouts

10.1 Gathering call stack information on a secondary thread

A secondary thread is any thread in a single job listed after the first thread (calledthe initial thread) and is dumped from a DSPJOB OUTPUT(*PRINT) commandautomatically as the call stack for that job.

To gather the call stack information for secondary threads on an AS/400 systemrunning V4R2 and above, print screens or process the dumps.

10.1.1 Print screensThe print screen way to display the call stack for secondary threads can be arather time consuming way to gather information, when you look at the possiblenumber of threads within the server and HTTP tasks. For detailed information onthe call stack, see 3.1.1, “Job information” on page 33.

To capture the information for the server job, you perform the following tasks:

1. Enter the following command:

WRKDOMSVR

2. Select option 9 (Work server jobs) for the required server.

3. Press F11 until you see the number of threads for the job.


4. Select option 12 (Work with threads) for the required job, which is the serverjob as shown in Figure 69.

Figure 69. Work with Threads

5. Select option 10 (Display call stack) for each thread. See Figure 70.

Figure 70. Display Call Stack

6. Copy the call stack. Ensure you get the entire stack. If any of the procedureshave a less than (<) character in front of them, position the cursor on theprocedure, and press F22 to display the entire name. See Figure 71. Performthe copy function on the entire name.

Work with ThreadsSystem: AS25

Job: SERVER User: QNOTES Number: 150575

Type options, press Enter.3=Hold 4=End 5=Display attributes 6=Release 10=Display call stack11=Work with thread locks 14=Work with thread mutexes

Total Aux RunOpt Thread Status CPU I/O Priority

00000076 SELW 25.425 1 2010 00000075 CNDW 377.657 0 20

00000074 SELW 20.015 311 2000000073 SELW 559.941 1024 2000000072 SELW 571.081 0 2000000071 TIMW .911 0 2000000070 TIMW 1.396 0 200000006F TIMW .919 0 200000006E TIMW .932 0 200000006D TIMW 1.569 6 20

More...F3=Exit F5=Refresh F9=Command line F12=Cancel F17=Top F18=Bottom



Thread: 00000075

ProgramRqs orLvl Procedure Library Statement Instruction

< tth_parm_t QSYS 0000000826< eate_part2 QSYS 0000001008< eadWrapper QNOTES 0000000008Scheduler QNOTES 0000000029

< CacheTask2 QNOTES 0000000018< essDbCache QNOTES 0000000019< SWaitEvent QNOTES 0000000002< eSemaphore QNOTES 0000000017< rThreadSem QNOTES 0000000015< _timedwait QSYS 0000000921< 0wMutexiT3 QSYS 0000001226

More...F3=Exit F10=Update stack F11=Display activation group F12=CancelF17=Top F18=Bottom F22=Display entire name


Figure 71. Display Entire Procedure Name

7. Paste the info into a workstation document or print it. You may prefer to savethe name into a document, because it is then easier to handle and to searchwithin the data you collected.

Remember that both outputs, the Work with Threads display and the CallStack display may be more than one page. Press PageDown to go to the endof the display and view that information as well.

8. Press F11 to see all different screens that are displayed:

• Display activation group• Display module• Display statement ID• Again, the first screen Display instruction

Repeat steps 6 and 7 to copy all stack information.

9. Press F12 to access to the Work with Threads display.

10.Repeat steps 5 through 8 until you have copied all call stacks for all threads.

Because this is really labor and time intensive work, you would do this processonly if you need the information for a single thread.

10.1.2 Process dumpsDumping each thread individually is the recommended procedure for jobs withless than four threads running. Dumping all threads at one time is recommendedfor jobs with four or more threads still running.

10.1.2.1 Dumping each thread individuallyThis procedure is recommended for jobs with less than four threads running.



..............................................................................: Display Entire Procedure Name :: WaitOnNativeSemaphore :: :: :: :: :: :: Bottom :: F12=Cancel :: ::............................................................................:

< eSemaphore QNOTES 0000000017< rThreadSem QNOTES 0000000015< _timedwait QSYS 0000000921< 0wMutexiT3 QSYS 0000001226

More...F3=Exit F10=Update stack F11=Display activation group F12=CancelF17=Top F18=Bottom F22=Display entire name

Chapter 10. Threads and semaphores 155

To dump each thread individually, complete the following tasks:

1. Write down the fully qualified job name from the Work with Active Jobs display(see 3.1.1, “Job information” on page 33), for example:

Job: ADMINPUser: QNOTESNumber: 150583

2. On the AS/400 command line, type:

STRSST

Press Enter.

3. Select option 1 (Start a service tool), and press Enter.

4. Select option 4 (Display/Alter/Dump), and press Enter.

5. Select option 2 (Dump to printer) or option 4 (Dump to tape). Press Enter.

6. Select option 4 (Tasks/Processes), and press Enter.

7. Select option 2 (Process), and press Enter.

8. Select option 5 (Display list of processes), and press Enter.

9. Find the job that is hung (use the fully qualified job name from Step 1), andtype 2 in front of it. Press Enter to display the threads. See Figure 72.

The option chosen depends on how the support representative decides totransmit the information. Dumping it to a printer puts the information in aspooled file that could be over 1,000 pages. In this case, it is likely that thesupport representative would be required to sign on to the system andtransfer the files to support.

This is, therefore, the fastest way to obtain the information if this is a criticalsituation. Dumping to tape puts the information on a tape that can be mailedto IBM for analysis.

Note


Figure 72. Work with Processes

10.Type option 1 (Select thread to work with) in front of the thread, and pressEnter.

11.The Display Process Found Information screen is shown (Figure 73). PressEnter.

Figure 73. Display Process Found Information

12.Select option 2 (Dump in hexadecimal (logical blocks)). Press Enter.

Work with Processes

Type option, press Enter.1=Select process to work with 2=Display threads

TDE TDE TDE ThreadProcess Name Number ID Address CountQDIA QSNADS 150601 000000000FF6 00000471 B00010000D389000 1RCHASM12 QSNADS 150600 000000000FF5 00000470 B00010000D387000 1RCHASM01 QSNADS 150599 000000000FF4 0000046F B00010000D385000 1TCPIPLOC QGATE 150598 000000000FF3 0000046E B00010000D373000 1LPAR2OPC QSNADS 150597 000000000FF2 000003DC B00010000D375000 1QROUTER QSNADS 150596 000000000FF1 000003D8 B00010000D37B000 1QZDSTART QSNADS 150595 000000000FF0 0000036B B00010000D37D000 1QSNADS QSYS 150594 000000000FEF 000001B5 B00010000D37F000 1DECS QNOTES 150587 000000000F96 0000041E B00010000D2C5000 1EVENT QNOTES 150586 000000000F89 0000040F B00010000D2AB000 13CALCONN QNOTES 150585 000000000F88 0000040D B00010000D2A9000 1SCHED QNOTES 150584 000000000F87 0000040C B00010000D2A7000 1

2 ADMINP QNOTES 150583 000000000F86 0000040B B00010000D2A5000 2AMGR QNOTES 150582 000000000F85 00000409 B00010000D2A3000 3

More...F3=Exit F5=Refresh F12=Cancel

Display Process Found Information

Output device . . . . . . : Printer

Process (PCS) . . . . . . : ADMINP QNOTES 150583

Process subtype . . . . . : FF

Process TDE address . . . : B00010000D D3D000

Process TDE number . . . . : 000000001560

Process TDE ID . . . . . . : 0000019D

Press Enter to continue.

F3=Exit F12=Cancel


13. If Dump to Printer is chosen, type a title for the dump, leave Perform seizes at1, and increase the Through page field to 999999 (as a precaution). See Figure74.

Figure 74. Specify Dump Title

14.If Dump to Tape is chosen, type a title for the dump and the name of the tapeon the Volume ID line. Then, give the dump a random file name. If the tape isblank, assign a sequence number of 0001. If you know that the tape is blank ordo not care about what is on the tape, change Check for active files to 2 (No).

15.Press F12 three times. Then you return to the Find Process screen.

16.Repeat steps 8 through 14 until you have completed this process for everythread running under that job.

Specify Dump Title



Dump title . . . . . . . . . . . ADMINP Thread 0000000000000005

Perform seizes . . . . . . . . . 1 1=Yes, 2=No

Partial print page numbers:From page . . . . . . . . . . . 1 1-2147483647Through page . . . . . . . . . 9999999 1-2147483647

F3=Exit F12=Cancel

In the case of Dumping to Tape, the dump first formats itself and theinformation. Once complete, a screen appears that asks for the name of thedevice (for example, TAP01). In the case of Dumping to Printer, a messageis shown at the bottom of the screen that says Dump to printer successfully

submitted.

Note

You cannot exit SST until the dumps have been completed or you lose theinformation. This may take some time. To view the status, at theDisplay/Alter/Dump Output Device menu, select option 7 (Display dumpstatus). See Figure 75. Press F5 to see the current status. When Dumprequests not complete equals 0, exit SST.

Note


Figure 75. Display Status of Dump

10.1.2.2 Dumping all threads at one timeThis procedure is recommended for jobs with four or more threads still running.Follow these steps:

1. Write down the fully qualified job name from the Work with Active Job display(see 3.1.1, “Job information” on page 33), for example:

Job: HTTPUser: QNOTESNumber: 151134

2. On the OS/400 command line, type the following:

STRSST

Press Enter.

3. Select option 1 (Start a service tool), and press Enter.

4. Select option 4 (Display/Alter/Dump), and press Enter.

5. Select option 1 (Display/Alter storage), and press Enter.

6. Select option 4 (Tasks/Processes), and press Enter.

7. Select option 2 (Processes), and press Enter.

8. Select option 5 (Display list of processes), and press Enter.

9. Find the job that is hanging (use the fully qualified job name from step 1), andtype 1 next to it to select the process to work with. See Figure 76 on page 160.Press Enter.

Display Status of Dump

Dump requestsnot complete . . . . . . . . : 1

Title ofactive dump . . . . . . . . . : ADMINP Thread 0000000000000005

Press Enter to continue.

F3=Exit F5=Refresh F12=Cancel


Figure 76. Work with Processes

10.The Display Process Found Information is shown. Press Enter.

11.Select option 1 (Display unformatted data), and press Enter.

12.In the Control field, type P8, and press F10 (Push Stack). See Figure 77.

Figure 77. Display Storage

13.You will know that it worked if you can see the fully qualified job name to theright. Copy down the complete address listed in the address field. Here, it isFBAA5812C7 000000. See Figure 77.

Work with Processes

Type option, press Enter.1=Select process to work with 2=Display threads

TDE TDE TDE ThreadProcess Name Number ID Address CountHTTP QNOTES 151275 000000001A1E 000001FF B00010000E7D5000 46DECS QNOTES 151274 000000001A1D 000001FE B00010000E7D3000 1LOGASIO QNOTES 151273 0000000019F0 000001CF B00010000E783000 1SERVER QNOTES 151272 0000000019EF 000001CE B00010000E781000 70QNNINSTS QNOTES 151271 0000000019EE 0000019A B00010000E77F000 1AS0656R1 QSYS 151270 0000000019ED 00000117 B00010000E77D000 1QZLSFILE QUSER 151263 0000000019DE 00000190 B00010000E753000 1QTTFT00095QTFTP 151195 000000001894 000003BE B00010000E4C5000 1QPADEV000NITSCID07 151188 000000001887 00000172 B00010000E4A7000 1QPADEV000NITSCID07 151159 000000001837 000003BC B00010000E411000 1QZSCSRVS QUSER 151139 000000001802 0000046A B00010000E39D000 1QRWTSRVR QNOTES 151137 0000000017FE 0000044D B00010000E329000 1

1 HTTP QNOTES 151134 0000000017CC 00000171 B00010000E33B000 51QZRCSRVS QUSER 151118 00000000179E 0000048E B00010000E2D9000 1

More...F3=Exit F5=Refresh F12=Cancel

Display Storage

Control . . . . . . . P8 nnnnn, Pnnnn, Lcccccc, .cccccc, >Address . . . . . . . FBAA5812C7 000000

0000 00010008 00800001 FBAA5812 C7000000 * ............G... *0010 00010000 00000000 DFA76377 6A000020 * ................ *0020 00001AEF C8E3E3D7 40404040 4040D8D5 * ....HTTP QN *0030 D6E3C5E2 40404040 F1F5F1F1 F3F44040 * OTES 151134 *0040 40408800 00000FE0 00014038 FF1C0101 * ........ ..... *0050 8040F2DD D0F30000 00000000 00000000 * . 2..3.......... *0060 D41C7340 7C000000 00000000 00000000 * M.. @........... *0070 00000000 00000000 00020000 01000000 * ................ *0080 804335D0 9E798001 00000000 00000000 * ................ *0090 00000000 00000000 00000000 00000000 * ................ *00A0 00000000 000000CC 00000000 00000000 * ................ *00B0 00000000 00000000 00000000 00000000 * ................ *00C0 00000000 00000000 E0010000 00000000 * ................ *00D0 00000000 00000000 00000000 00000000 * ................ *00E0 00000000 00000000 00000000 00000000 * ................ *00F0 00CC0000 00000000 00000000 00000000 * ................ *F3=Exit F4=Alter labels F5=Refresh F6=Display stackF9=Pop stack F10=Push stack F11=Alter storage F12=Cancel


14.Press F12 six times to return to the Display/Alter/Storage Output Devicescreen.

15.Select option 2 (Dump to printer) or option 4 (Dump to tape). Press the Enterkey.

16.Select option 1 (Machine Interface (MI) object), and press Enter.

17.Select option 26 (Process control space (1A)), and press Enter.

18.Select option 2 (Find by object address), and press Enter.

19.Type the address from step 1, and press Enter. See Figure 78.

Figure 78. Find by Object Address

20.The Display Object Found Information is shown. Press Enter.

21.Select option 2 (Dump in hexadecimal (logical blocks)), and press Enter.

22.To Dump to printer, type a title for the dump, leave Perform seizes at 1, andincrease the Through page field to 999999 (as a precaution). Press Enter.

23.If Dump to Tape is chosen, type a title for the dump, type the name of the tapeon the Volume ID line, and give the dump a random file name. If the tape isblank, assign a sequence number of 0001. If you know that the tape is blank ordo not care about what is on the tape, change Check for active files to 2 (No).

The option chosen depends on how the support representative decides toget the information transmitted. Dumping to printer puts the information in aspooled file that could be over 1,000 pages. In this case, it is likely that thesupport representative would be required to sign on the system and transferthe files to support. This is, therefore, the fastest way to get the informationif this is a critical situation. Dumping to tape puts the information on a tapethat can be mailed to IBM for analysis.

Note

Find By Object Address



Object address . . . FBAA5812C7 000000 0000000000 000000-FFFFFFFFFF FFFFFF

F3=Exit F12=Cancel


10.2 Handling semaphore timeouts

In this section, we document:

• What a semaphore and a semaphore timeout are• An example of a semaphore timeout in Domino• Reasons for semaphore timeouts• How to determine if a semaphore timeout has occurred• Parameters in the NOTES.INI file for troubleshooting semaphore timeouts

10.2.1 What a semaphore and a semaphore timeout areIn a multitasking environment, there is often a requirement to synchronize theexecution of various tasks or ensure one process has been completed beforeanother begins. This requirement is facilitated by the use of a software switchknown as a semaphore or a flag. The function of this is to work in much the sameway that a railway signal would: only allowing one train on the track at a time. Asemaphore timeout is where the railway signal has been set in one state too long,maybe because the train has broken down.

10.2.2 Example of a semaphore timeout in DominoAn example of a semaphore timeout in Notes/Domino is when the indexer needsto completely rebuild an index. It locks a semaphore so that other tasks cannotuse the index until it is rebuilt. If a user task now tries to open that index while it isbeing rebuilt, it has to wait for the indexer to finish the rebuild and then unlock thesemaphore. As a result, the user task is stuck until that semaphore is unlocked.

While it is stuck waiting for the semaphore, it keeps track of how long it has beenwaiting. If it is stuck for more than 30 seconds, this is considered a semaphoretimeout and, in debug mode, a message is logged to the console. The taskcontinues to wait for the semaphore, timing out every 30 seconds, until thesemaphore is unlocked or the task is ended.

For most operations, a task may only wait a few microseconds and, therefore, nottimeout. With a complicated view on a large database, the task may have to waitseveral minutes for the index semaphore.

If an important semaphore is locked by a task and is never unlocked, all tasks canbe stopped waiting for that semaphore. This can happen in several differentways. The most common is where a task locks the semaphore and then crashes.This can also happen if a task locks the semaphore and then goes into anendless loop or it gets an error and forgets to unlock it.

In the case of Dumping to Tape, the dump first formats itself and theinformation. Once complete, a screen is shown that asks for the name of thedevice (for example, TAP01). In the case of Dumping to Printer, a message isshown at the bottom of the screen that says Dump to printer successfullysubmitted.

Note


Semaphore deadlock can occur when two tasks try to lock two differentsemaphores in a different order. For example, Task A locks semaphore 1 andthen tries to lock semaphore 2. In the meantime, Task B has already lockedsemaphore 2 and is now trying to lock semaphore 1. Task A is stuck waiting forsemaphore 2, and Task B is waiting for semaphore 1. This means there is adeadlock.

10.2.3 Reasons for semaphore timeoutsWhen you receive semaphore timeout messages, the messages are usually theresult of one of the following reasons:

• A heavy load on the server is causing processes to be delayed from releasingsemaphores.

• A process has crashed while holding a semaphore, causing other processesto block when trying to acquire the semaphore.

• A deadly embrace, semaphore contention occurs where two tasks are waitingon each other and neither task can break the loop. In the simplest case,thread A is trying to get a semaphore that is owned by B, while B is trying toget a different semaphore that is owned by A. More complex combinations arealso possible: A wants a semaphore owned by B, who wants a semaphoreowned by C, who wants a semaphore owned by A, and so on.

• If a process fails to set a semaphore during execution, another process thatdepends on the semaphore is blocked awaiting the semaphore.

10.2.4 Determining if a semaphore timeout has occurredIf there is an issue with a semaphore, it will be reported in two ways:

• A specific answer to the Show stat Sem. Timeouts command appears on theconsole.

• The error message Session semaphore held for [n] seconds is reported on theDomino console.

10.2.4.1 Show stat Sem.Timeouts on the consoleAt the server console, you see the Sem.Timeouts statistic. To view this statistic,type:

sh stat sem.timeouts

If the problem has occurred, you see text similar to the following example,depending upon the nature of the semaphore timeout:

Sem.Timeouts = 4143:4 0244:3 412C:2

The first number is the semaphore ID. The ID tells us what the semaphore isused for. For example, 0x030B is the collection semaphore used by Notes IndexFacility (NIF). The second number is a decimal number that shows the number oftimes the semaphore timeout occurred. It is not unusual in a heavily used serverto get semaphore timeouts.


Table 12 lists some of the semaphores you may encounter.

Table 12. List of semaphores and their functions

10.2.4.2 Error: Session semaphore held for [n] secondsThe second way of determining whether there is a semaphore issue is when yousee (at the server console) the message Session semaphore held for [n] seconds.

Note: This error does not print to the log file.

10.2.5 Troubleshooting semaphore timeoutsThe following parameters can be added to the NOTES.INI file:

Semaphore Function

0244 NSF per-database semaphore

410F OS file system semaphore

0266 NSF per-database full-text semaphore

4245 NSF database opening semaphore

030B NIF collection semaphore

430D NAMELookup semaphore

0A13 Log commit semaphore

0116 Console semaphore

0A12 Buffered log package semaphore

030B Collection semaphore

4117 Handle table free chain consistencysemaphore

0A0B Session table semaphore

4113 Handle table movement semaphore

33d5 AdminP's semaphore for ACL modification

The statistic Sem.Timeouts will not appear in the Statrep.NSF database if theserver is not experiencing semaphore timeouts.

Note

Before enabling any debug parameters, it is imperative that you discuss themwith a Lotus Notes/Domino Support Analyst. There may be issues surroundingtheir use or special precautions that must be considered. These debugparameters may require a large amount of disk space, depending on when theserver encounters problems. The longer the server stays up, the larger thedebug files will be. These files can grow large enough to cause disk spaceshortages.

Note


• Debug_Capture_Timeout=1• Debug_Show_Timeout=1• DEBUG_SEM_TIMEOUT=xx in milliseconds• Debug_threadid=1

After adding these parameters, if semaphores are generated, they are displayedin the Domino console. They are not logged to the log.nsf database. To save thedata for further analysis, you can dump the complete Domino console to aspooled file by first pressing F5 (Refresh) to capture all actual data and thenpressing F6 (Print) on the Domino console. You may also want to use theparameter Debug_Outfile=filename.txt, which writes the general console outputand the Debug_threadid information to the defined text file in the Domino datadirectory. Unfortunately it does not dump the semaphore timeouts information tothe file.

If the system experiences a semaphore timeout, it would look like the exampleshown in Figure 79 on page 166 in the Domino console.

The Debug_Capture_Timeout=1 and Debug_Show_Timeout=1 statements arealso used to create a logging file for the semaphore timeouts in the Notes datadirectory. The name of this logging file should be SEMDEBUG.TXT. Due to anerror in the actual releases, until R4.6.6 and R5.0.2, this name comes out as amangled file name with unreadable characters. This problem will be fixed inR4.6.7 and R5.0.3.

Until the fix is available, you can either read it by using a workstation editor,such as Windows Notepad. Or, you can rename the file with AS/400 OperationsNavigator, and then use the Display File (DSPF) command to display thestreamfile within an 5250 screen.

Note


Figure 79. Semaphore timeout in the Domino console

The output mean is explained here:

• 147511 is the job number experiencing this timeout. Here, it is the DECS jobrunning, as shown in Figure 80.

• 1069 is the number of the semaphore you get of the IPC dump (see Figure 81).

• 01545 is the process identifier, also referred to in AS/400 Operations Navigatorin the semaphore properties of semaphore 1069. For an example, see Figure23 on page 44.

• 00000013 is the thread number within the job.

• 0x0244 indicates the type of semaphore. Here, it is the open databasesemaphore.

• /domino/DOMSVR3/decsadm.nsf is the database involved here.

• STREADER=147499:01533:0000002C FOR 30 MS gives us the information about thejob and thread within the job that wants to read the information. Here, it is theserver job 147499, with process identifier 1533 and thread 2C within the serverjob.

Work with Domino ConsoleServer: DOMSVR3

Previous subcommands and messages:THREAD [147511:01545:00002-00000013] WAITING FOR FRWSEM 0x0244 database semaphore (@8000000000000000F8CDC023140FA7F4) (/domino/DOMSVR3/decsadm.nsf) (R=1,W=0,WRITER=000000:00000:00000000,1STREADER=147499:01533:0000002C) FOR 30 ms02/17/2000 02:42:27 PM DECS: Activity finished: Contacts02/17/2000 02:42:27 PM DECS: Activity finished: eItems02/17/2000 02:42:27 PM DECS: Activity finished: eStockTHREAD [147511:01545:00002-00000013] WAITING FOR FRWSEM 0x0244 database semaphore (@8000000000000000F8CDC023140FA7F4) (/domino/DOMSVR3/decsadm.nsf) (R=1,W=0,WRITER=000000:00000:00000000,1STREADER=147499:01533:0000002C) FOR 30 msTHREAD [147511:01545:00002-00000013] WAITING FOR FRWSEM 0x0244 database semaphore (@8000000000000000F8CDC023140FA7F4) (/domino/DOMSVR3/decsadm.nsf) (R=1,W=0,WRITER=000000:00000:00000000,1STREADER=147499:01533:0000002C) FOR 30 ms

> show stat sem.timeouts

Sem.Timeouts = 4143:4 0244:3 412C:2

Enter a Domino subcommand.===>

F3=Exit F5=Refresh F6=Print F9=RetrieveF17=Top F18=Bottom F21=Command line


Figure 80. Work with Active Jobs display

All this information is an example of which information can be taken out of thesystem if you are experiencing problems, such as crashes or hang conditions.

Figure 81. IPC dump



Type options, press Enter.9=Exclude 10=Display call stack 11=Work with locks12=Work with threads 14=Work with mutexes ...

Opt Subsystem/Job User Number Type CPU % ThreadsDOMINO01 QSYS 147497 SBS .0 1ADMINP QNOTES 147507 BCI .0 1AMGR QNOTES 147505 BCI .0 1AMGR QNOTES 147506 BCI .0 3BILLING QNOTES 147500 BCI .0 1CALCONN QNOTES 147509 BCI .0 1DECS QNOTES 147511 BCI .0 1EVENT QNOTES 147510 BCI .2 13SERVER QNOTES 147499 BCI .7 68

More...Parameters or command===>F3=Exit F5=Refresh F7=Find F10=Restart statistics F11=Display statusF12=Cancel F17=Top F18=Bottom F23=More options F24=More keys

Display Spooled FileFile . . . . . : IPCS Page/Line 6/6Control . . . . . Columns 1 - 130Find . . . . . .

*...+....1....+....2....+....3....+....4....+....5....+....6....+....7....+....8....+....9....+....0....+....1....+....2....+....3

0 1 147505S 1056 0XF6010502 --RW-RW-RW- QNOTES *NONE QNOTES *NONE 3 17:37:13 14:35:49

0 31 P0015801 9999 1474992 0 147499

S 1057 0XF6010503 --RW-RW-RW- QNOTES *NONE QNOTES *NONE 3 14:35:49 14:35:490 1 1474991 9999 1474992 0 147499

S 1058 0000000000 --RW-RW-RW- QNOTES *NONE QNOTES *NONE 1 14:35:51 14:35:510 1 147508




S 1075 0XF60F0501 --RW-RW-RW- QNOTES *NONE QNOTES *NONE 3 17:46:50 15:45:330 1 P001581

More...F3=Exit F12=Cancel F19=Left F20=Right F24=More keys


Chapter 11. Troubleshooting e-mail delivery

E-mail troubleshooting needs to be broken into parts. Determination needs to bemade regarding whether the issue involves the SMTP servers, the Domain NameSystem server (DNS), the firewall, or the network.

When doing problem resolution, it is important to determine if e-mail has everbeen sent successfully. If it has not, you are probably looking for basicconfiguration errors.

Take extra care in evaluating the status of the internal workings of the AS/400system, including:

• SMTP server for inbound mail• SMTP client for outbound mail• Mail Server Framework (MSF)• OS/400 Resolver• System Distribution Directory (SDD)• SMTP server attributes• POP server attributes

Also, with the release of Lotus Domino Server Version 5.0, two separate Dominoconfigurations are possible. The first configuration consists of using the DominoRouter, the Domino SMTP server function, and the OS/400 Resolver and TCP/IPStack. The second configuration consists of using the Domino Router, OS/400SMTP Servers, the OS/400 MSF, and the OS/400 Resolver and TCP/IP Stack.

Whether *DOMINO or *MSF was selected as the SMTP method duringconfiguration of the Domino Server on the AS/400 system dictates how e-mailtroubleshooting is performed on the AS/400 system and Domino.

11.1 Domino and Internet mail flow

There are many places to lose mail, so we look at the mail flows, tasks or jobs,and databases that are needed for:

• Domino SMTP MTA• Domino for AS/400 Release 4.6 SMTP MTA• Domino R5 SMTP• Domino for AS/400 AnyMail/400 integration with MSF

If you understand how Domino mail flows through the AS/400 system, it may helpyou resolve e-mail problems.

11.1.1 What MSF doesAnyMail/400 is a set of registered exit points. The AnyMail/400 Mail ServerFramework (MSF) is a framework for handling e-mail:

• It simply calls exit programs in a specific order, passing programs informationabout e-mail. It includes these features:

• The exit programs do all the work.

• It is used for Domino and OfficeVision/400 integration (and Domino Release4.6 SMTP).


E-mail is queued to MSF via APIs that create an internal MSF object that holdsinformation about the mail:

• Recipient address• Pointers to mail content (IFS files, OfficeVision/400 documents, etc.)

The Mail Server Framework runs in QMSF jobs under QSYSWRK subsystem,and is started when QSYSWRK is started.

To start and stop MSF, use the Start Mail Server Framework (STRMSF) and EndMail Server Framework (ENDMSF) commands.

Figure 82. What MSF does

The AnyMail/400 Mail Server Framework is an open structure for electronic maildistribution that is provided with the OS/400 operating system. AnyMail/400functions are a set of mail-related functions that provide open and flexibleinterfaces to support mail on the AS/400 System. The MSF is the distributionMSF is to allow the distribution of electronic mail messages (for example, voice,video, or text).

On the AS/400 system, the Mail Server Framework performs the followingfunctions:

SDD

QSYS/QZD...

Address Resolution

Local Delivery

Message Forwarding

?

InternalMSFQueueSNADS

OV/400

POP3Mailbox

QTCP/QTM...OV/400Mailbox

QTCP/QTM...Attachment Conversion

QMSF

Internet SMTP

DominoMailbox

QNOTES/QANI.

QNOTES/QANI.

QTCP/QTM...

QSYS/QZD...

QSYS/QZD...


• It creates and queues MSF messages.• It distributes MSF message information by calling configured exit programs.

An exit point is a specific point in a system function or program where control ispassed to one or more specified programs. An exit point passes control to an exitprogram. The exit program can be written by the user, or it can be a program thatis already on the system. An application program interface (API) is a functionalinterface supplied by the operating system that allows an application programwritten in a high-level language to use specific data or functions of the operatingsystem.

The Mail Server Framework allows programs to be configured and accessedthrough an API. A snap-in is a registered exit point program that is called from aMail Server Framework exit point. The user-written exit programs provide the mailmessage processing required for each MSF message that is created. The MSFmessage contains information that defines the electronic mail message. The MailServer Framework does not process the contents of a mail message. Instead, itdetermines which exit program is allowed to work with the MSF message, and ittracks the flow of the MSF message through the framework. When an MSFmessage is sent, the MSF data type is stored and used asynchronously. Theframework also provides support for electronic mail messages that arrive fromother systems.

The MSF message may be created by an application and, as a result of usingspecific exit point programs, the information may be sent to other applications onthe same system, and applications on other systems. Also, messages couldarrive from other systems. As a result of using specific exit point programs, thatinformation could be both delivered to the application on that system andforwarded to another system.

The exit points allow you to install software to affect MSF message flow throughthe Mail Server Framework. Each of the Mail Server Framework exit points arepresented in the following list along with the functions of the exit programs:

• List expansion: Allows for expansion of the distribution lists for the MSFmessage. It expands any distribution list found in a recipient list. Note thatdistribution lists may contain other distribution lists that can be expandedfurther.

• Address resolution: Allows for address mapping for each destinationaddress and adds the correct information to the MSF message recipientinformation so appropriate exit programs can be called at later exit points.

• Envelope processing: Allows the message envelope to be changed to aformat that the mail delivery system accepts.

• Attachment conversion: Allows for changes or additions to MSF messageattachments.

• Security and authority: Allows for verification of user authority to distributethe MSF message information to addresses in its recipient list, before the MSFmessage is delivered using the local or remote delivery exit programs.

• Local delivery: Allows for delivery of the MSF message to local recipients.

• Message forwarding: Allows forwarding of the MSF message information toremote recipients.

• Non-delivery: Allows for reporting the non-delivery of an MSF message.

Chapter 11. Troubleshooting e-mail delivery 171

• Attachment management: Allows for the management of attachmentsreferenced by an MSF message as part of the message distribution.

• Accounting: Allows for an audit trail of activity required in the electronic mailenvironment.

You can find and work with the exit points using the WRKREGINF FORMAT(MSFF0100)

command on an AS/400 command line. The screen shown in Figure 83 appears.For more information about MSF Exit program, refer to AnyMail/400 Mail ServerFramework Support V4, SC41-5411.

Figure 83. Work with Registration Information display

11.1.2 What POP3 isThe Post Office Protocol (POP3) is an electronic mail protocol that allows a clientrunning POP3, such as Netscape Communicator, Eudora Pro, or MicrosoftOutlook, to retrieve e-mail messages. POP3 allows mail for multiple users to bestored in a central location until a request for delivery is made by an electronicmail program. POP3 uses SMTP for communication between the e-mail clientand the server.

11.1.2.1 The POP3 directory entryThe POP3 directory entry can be a source of confusion for an AS/400administrator configuring POP3 for the first time. What makes a directory entry aPOP3 directory entry?

There are two parameters in the directory entry that determine if the entry is aPOP3 directory entry. The parameters are:

• Mail Service Level = 2 (System message store)• Preferred address = 3 (SMTP name)

Work with Registration Information

Type options, press Enter.5=Display exit point 8=Work with exit programs

ExitExit Point

Opt Point Format Registered TextQIBM_QZMFMSF_ACT MSFF0100 *YES MSF Accounting ExitQIBM_QZMFMSF_ADR_RSL MSFF0100 *YES MSF Address ResolutionQIBM_QZMFMSF_ATT_CNV MSFF0100 *YES MSF Attachment ConversionQIBM_QZMFMSF_ATT_MGT MSFF0100 *YES MSF Attachment ManagementQIBM_QZMFMSF_ENL_PSS MSFF0100 *YES MSF Envelope ProcessingQIBM_QZMFMSF_LCL_DEL MSFF0100 *YES MSF Local DeliveryQIBM_QZMFMSF_LST_EXP MSFF0100 *YES MSF List ExpansionQIBM_QZMFMSF_MSG_FWD MSFF0100 *YES MSF Message ForwardingQIBM_QZMFMSF_NON_DEL MSFF0100 *YES MSF Non DeliveryQIBM_QZMFMSF_SEC_AUT MSFF0100 *YES MSF Security and AuthorityQIBM_QZMFMSF_TRK_CHG MSFF0100 *YES MSF Track Mail Message ChangeQIBM_QZMFMSF_VLD_TYP MSFF0100 *YES MSF Validate Type

===> WRKREGINF FORMAT(MSFF0100)F3=Exit F4=Prompt F9=Retrieve F12=Cancel


11.1.2.2 The POP3 mailbox on the AS/400 systemWhen POP3 mail is successfully delivered on the AS/400 system, it is located in aPOP mailbox on the AS/400 system until the POP3 user issues the GET MAILcommand from the POP3 client. It is possible to review the contents of an AS/400IFS directory to determine if a POP3 user has any mail distributions in the POP3mailbox. This is useful when debugging a mail problem because an administratordoes not have to continue to use the POP3 client and issue the GET MAILcommand to see if mail is finally working. Instead, they can check for mail withone command:

WRKLNK '/QTCPTMM/MAIL/JONEST2'

Here, JONEST2 in the command is the system directory user ID of the POP3client. This may be different from their SMTP user ID.

If the POP3 mailbox exists, the previous command produces the display shown inFigure 84.

Figure 84. Locating a POP3 mailbox on the AS/400 system

The POP directory entry needs to be configured on the AS/400 system that isthe final location for the mail, until the user “gets the mail”. This is the AS/400system, defined in the POP3 client as its incoming POP server. The AS/400system is where the POP3 client “gets” mail. There is another kind of directoryentry that can be used to forward mail. It is a different type of directory entryfrom the POP directory entry.

Tip


Directory . . . . : /QTCPTMM/MAIL


Opt Object link Type Attribute TextJONEST2 DIR

BottomParameters or command


From the display shown in Figure 84, select option 5 to view the next level. Thenext level shows any mail distributions that exist in the POP3 mailbox. The screendisplayed in Figure 85 shows that the two mail distributions are located in theJONEST2 POP3 mailbox. These distributions disappear after the POP3 userissues the GET MAIL command from the POP3 client. You cannot read thecontents of these mail distributions from an AS/400 screen.

Figure 85. Mail distributions

11.1.3 SMTP MTA tasks and databasesA Message Transfer Agent (MTA) is a program that runs under the control of aDomino or Notes server to exchange messages (e-mail) with mail applicationsusing different mail protocols.

The Simple Message Transfer Protocol/ Multipurpose Internet Mail Extensions(SMTP/MIME) is a standard based on TCP/IP. It is often used to connect unlikemail systems. The MTA provides advanced messaging features and extensions tothe Domino 4.6 server, allowing it to transfer messages in the SMTP/MIME formatvia TCP/IP networks, which can be the Internet, connecting to many othercompanies, or the intranet within one company.

Integrated into Domino Release 4.6, the SMTP MTA provides the contentconversions and address mappings necessary to enable Notes users toexchange SMTP/MIME formatted mail. Every Domino Release 4.6 server has theSMTP MTA function. It has to be enabled for the server.

If the previous command is issued and the error message Object not found isissued to the user’s job log, the POP3 mailbox does not exist. It is important torealize that the POP3 mailbox does not exist until the first distribution of mail isdelivered to it. If the POP3 mailbox (in the form of the directory listed in Figure84 on page 173) is missing, it does not necessarily mean that the POP3directory entry was misconfigured. It may mean that mail has never beendelivered to this mailbox.

Important


Directory . . . . : /QTCPTMM/MAIL/JONEST2


Opt Object link Type Attribute TextJW122040.NOT STMFJW122735.NOT STMF

Bottom


All Domino servers in a network don't need to use SMTP MTA. A single server orspecific servers can serve as gateways for an entire network of servers.

At Release 4.6, an MTA is used for sending mail to and receiving mail from theInternet. There are a number of tasks that are loaded with the load smtpmta

command.

Figure 86 shows the flow of mail to and from the Internet and the tasks anddatabases that are involved. The router does all the work. The conversion task issingle threaded, converting SMTP to Notes CD (compound document) and viceversa. All tasks must be working for it to work.

SMTPMTA is the spawning task for the other jobs. Delivery Reporting Task (DRT)moves objects around the databases. Both are necessary for anything to work.

Figure 86. General SMTP MTA tasks and databases

11.1.4 SMTP MTA and MSF: R4.6 AS/400On a single AS/400 system, only one partition can run the MTA. For an AS/400system, this is how mail is exchanged between Domino and OfficeVision/400.

AS/400-based mailing applications, such as OfficeVision/400, can also use theOS/400 SMTP support to convert from and to SMTP, which allows direct mailexchange with a Domino for AS/400 server on the same or other AS/400systems.

A Domino for AS/400 Release 4.6 server has a slightly different SMTP MTA:

• The tasks and database structures are the same.• Conversion and address mapping functions of MTA are the same.

Inbound and Outbound session handlers and controllers are completely different:

• Inbound gets SMTP/MIME messages for Notes users from MSF via an IFSdirectory.

• Outbound uses an API and an IFS file to pass SMTP/MIME messages fromNotes users into MSF.

Internet

OMSGCNV

ROUTER

mail.boxsmtpibwq.nsf

smtp.box

smtpobwq.nsf

ISESHNDLISESCTL

OSESHNDLOSESCTL

SMTPMTA

DRT

IMSGCNV

Domino Server

Port25

SMTP MailServer

25

TCP/IP

SMTP MailServer

TCP/IP


Figure 87. SMTP MTA and MSF: R4.6 AS/400

The CFGDOMSVR ... MAIL(*SMTPMTA) command installs some MSF exits andcreates directories used to integrate MTA into the AS/400 SMTP andOfficeVision/400 functions.

AS/400 Domino is not listening on a port. Messages are dumped into a directory,and an API is called. If this layer is not configured correctly or not turned on, youdon't get the same feedback. Mail may just disappear. All jobs have to be activefor it to work.

If we look at inbound SMTP mail, the AS/400 system determines if it is destinedfor an OV or AS/400 POP user based on SDD entries. If not, it places the mail inthe IFS directory/new. This happens even if the MTA is not running. If it isrunning, the inbound session controller queries this directory for new mail. If so, itinvokes up to eight inbound session handlers to route the e-mail to the Dominoserver inbound SMTP work queue. The inbound message converter converts theMIME into a Notes document, attachments and all. The document is sent to theMAIL.BOX file on the server, awaiting delivery via the Domino router task. Thedelivery report task then deletes successfully sent messages, or generates anon-delivery report, for example if no user exists for the address.

For outbound mail, we have a similar process. E-mail that can't be delivered bythe Domino router goes to the SMTP.BOX mail file. The document is convertedfrom a Notes format to MIME via the outbound message converter task and sentto the outbound SMTP work queue. The outbound session controller monitors thework queue and calls the outbound session controller to basically send the mail tothe AnyMail Framework on the AS/400 system. In fact, the e-mail is copied in the/TMP directory to be handled later by the Mail Server Framework.

The AS/400 Mail Server Framework and SMTP and TCP/IP support areresponsible for the actual sending of the e-mail.

OMSGCNV

ROUTER

mail.box

smtpibwq.nsf

smtp.box

smtpobwq.nsf

Internal MSFQueue

SMTPMTA

DRT

IMSGCNV

Domino for AS/400 Server

Local DeliveryQTCP/QTM...

QTCP/QTM...

OSESHNDLOSESHNDL

OSESHNDLOSESCTL

ISESHNDLISESHNDL

ISESHNDLISESCTL

QMSFAddress Resolution

/mail, /new,/inprog

/tmp

QNOTES/QANI...

QNOTES/QANI...

QTSMTPnnnnJobs in

QSYSWRK


11.1.4.1 Tasks and AS/400 jobssmtpmta, the SMTP MTA “parent” task, is started with LOAD SMTPMTA. All MTAtasks are jobs in the Domino server's subsystem:

• drt: Delivery report task• isesctl: Inbound session controller• iseshndl: Inbound session handler (up to eight)• imsgcnv: Inbound message conversion• omsgcnv: Outbound message conversion• osesctl: Outbound session controller• oseshndl: Outbound session handler (up to eight)

The router task, which is not part of the MTA, is started with the Start DominoServer (STRDOMSVR) command.

QMSF (AnyMail/400) in subsystem QSYSWRK is started with the Start MailServer Framework (STRMSF) command or when the QSYSWRK subsystem isstarted.

AS/400 SMTP/MIME jobs (QTSMTP...) are started in QSYSWRK subsystem withthe STRTCPSVR *SMTP command.

11.1.4.2 Used databases and directories (IFS files)Databases, created for the SMTP MTA task in the server data directory, include:

• smtp.box• smtpibwq.nsf: SMTP MTA inbound work queue• smtpobwq.nsf: SMTP MTA outbound work queue• mail.box

The following directories are created (under the/QIBM/UserData/LOTUS/SMTPMTA/ directory):

• tmp is used by oseshndl for creating outbound MIME

• inbound is used by iseshndl when working on inbound MIME and itssubdirectories:

– mail– new– inprog

11.1.5 R5 server Domino SMTP supportSMTP routing is built into Domino R5. Some remapping is done in the router, butno conversion is performed. Figure 88 on page 178 illustrates the new R5architecture.


Figure 88. R5 server Domino SMTP support

When you enable an R5 server to send messages over SMTP outside the localInternet domain (configured in the Server Configuration document), the server nolonger needs to use the Foreign SMTP Domain and SMTP Connectiondocuments. The server can connect via TCP and route e-mail via SMTP. Theserver can use a relay host as needed and make connections using DNS or usinga hosts file. The server uses the router to route e-mail over SMTP and performconversions between Notes format messages and MIME format messages.Therefore, the R5 server replaces the outbound services of the MTA.

When you enable an R5 server to listen for inbound SMTP connections(configured in the Server document), the server performs the same functions asthe inbound MTA services. Inbound messages are stored in MAIL.BOX andtransferred or delivered by the router. MIME messages are stored in that formatand routed over either Notes RPC or SMTP to their destination. If needed, therouter converts between MIME and Notes format messages.

11.1.5.1 Databases and directories (IFS files) usedThe MAIL.BOX database, under the server data directory, is the place where:

• Inbound messages are stored.• Outbound messages are stored.

11.1.6 R5 server integrated with AS/400 SMTP: MSFNow let's look at Domino R5 and MSF. For Domino R5, the integration withAS/400 SMTP and Mail Server Framework is much simpler and less resourceintensive. This integration is shown in Figure 89.

R5ROUTER

mail.box

Internet

Domino for AS/400 R5 Server

Port25

SMTP MailServer

25

TCP/IP

SMTP MailServer

TCP/IP SMTPlistenertask

Notes R5 Client

NotesMailbox


Figure 89. R5 server integrated with AS/400 SMTP: MSF

A Release 5.0 Domino server can be set to do SMTP mail routing, even if theSMTP listener task is disabled (which is needed to prevent it from interfering withAS/400 SMTP). This is used, as well as other R5 enhancements to the mailrouter, with AS/400 integration for Domino R5.

Let's look at inbound mail first. As in Release 4.6, the AS/400 AnyMail and SMTPservices determine if a message should be delivered locally to anOfficeVision/400 or POP3 user or sent to Domino. Assuming it is destined for aNotes user, the AnyMail framework places the MIME message in the indicatedIFS directory temporarily. Then the Domino server processes the message(converts it from MIME to Notes Rich text if needed, determines the recipient,etc.). The message is placed in the Domino server's MAIL.BOX file for the routerto deliver to Notes users. Many of the tasks from Release 4.6 are gone, including:

• The inbound session controllers and handlers• The inbound work queue• The message converter

For outbound SMTP mail, the router takes the message out of the mail.box fileand converts any Notes documents to MIME. Then, it places the message in theAS/400 IFS for the AnyMail framework and the SMTP server can send themessage to the proper server. The following items are gone:

• The SMTP.BOX file• The outbound work queue• Outbound session controllers and handlers

As you can see, this is a much simpler solution than the Domino for AS/400Release 4.6 SMTPMTA.

Internal MSFQueue

Domino for AS/400 R5 Server

Local Delivery

QTCP/QTM...

QTCP/QTM...

QMSFAddress Resolution

/inbound

/outbound

QNOTES/QANI...

QNOTES/QANI...

QTSMTPnnnnJobs in

QSYSWRK

R5ROUTER

mail.box

Notes R5 Client

NotesMailbox

/qibm/userdata/lotus/smtpmtainboundoutbounddeadletter


11.1.6.1 Tasks or AS/400 jobsThe router, which is not part of the MTA, is started automatically with the server. Athread reads out of inbound and writes to the MAIL.BOX and writes to outbound.

QMSF (AnyMail/400) in the QSYSWRK subsystem is started with STRMSF orwhen QSYSWRK subsystem started.

QTSMTP... (AS/400 SMTP/MIME Jobs) in the QSYSWRK subsystem is startedwith STRTCPSVR *SMTP.

Address resolution is the same.

11.1.6.2 Database and directories (IFS files) usedThe only database (under the server data directory), created by the configuration,is: MAIL.BOX (created as part of server)

The directories, under /QIBM/UserData/LOTUS/SMTPMTA, are:

• Inbound (much simpler than with 4.6)• Outbound• Deadletter

11.2 E-mail troubleshootingWhen mail is not being delivered as expected, a DNS or mail administrator isfaced with one of the most challenging troubleshooting areas in TCP/IP.

To begin, it is important to determine if e-mail has ever been sent successfully. Ifnot, you may be looking for basic configuration errors. For additional informationabout configuring the Domino for AS/400 server, refer to:

• Lotus Domino for AS/400: Installation, Customization, Administration,SG24-5181, when using Domino for AS/400 Release 4.6.

• Lotus Domino for AS/400 R5: Implementation, SG24-5592, when usingDomino for AS/400 Release 5.0.

11.2.1 The starting and ending placeThe first step in debugging mail is to always know exactly what the users areusing to address the mail. If possible, visit the users at the client location andwatch them type in the Mail To value: <user Id@smtp domain name>. Watch formis-typing. Make sure that the user is using the @ symbol and not using the wordat.

The second step is to find the SMTP User ID and the SMTP Domain name in theAS/400 system alias table on the AS/400 system for the POP client to which themail should be delivered.

These two pieces of information are the starting and ending place for mail. Maildelivery starts by using the Mail To: information and ends by delivering the mail tothe POP mailbox on the AS/400 system associated with the SMTP user ID andthe SMTP domain name.

What the user types to the right of the @ sign in the Mail To field should match theSMTP Domain name in the AS/400 SMTP system alias table for the POP3 user


who should receive the mail with one exception: when aliases are used. Forexample, consider the example where mail is addressed [email protected]

However, the AS/400 SMTP system alias table lists this user ’s SMTP Domainname as AS1.mycompany.com. This discrepancy is okay. Mail is successfullydelivered if the AS1 local host table lists mycompany.com as an alias toAS1.mycompany.com and the Search First parameter in CFGTCP option 12 is setto *LOCAL.

11.2.2 Internet mail flow on an AS/400 system running DominoIn many organizations, it is a usual situation nowadays to have users send orreceive Internet mail in addition to e-mail inside the organization. In this section,we look specifically at how mail goes from the Internet to the Domino server, orleaves the Domino server and goes toward the Internet, and how to troubleshootproblems happening there.

11.2.2.1 Domino Release 4.6For R4.6, Domino must rely on the AS/400 for SMTP serving and MSF forprocessing of mail before it will ever have the chance to be routed to or from theDomino server itself. If a Domino R4.6x server is configured for *SMTPMTA, itautomatically creates the registry snap-ins to deliver mail to any Domino user notlisted expressly in the AS/400 SDD.

The R4.6 flow of inbound mail follows this process:

1. Internet mail is received by the AS/400 SMTP server and is placed in the IFSdirectory QTCPTMM/SMTPBOX. You only see it appear here if MSF is ended.Otherwise, it may happen too fast to catch by just viewing the directory, whereif MSF is ended, you see inbound Internet mail sit in this directory.

2. With MSF running, the mail processes through the MSF exit points.

3. If *SMTPMTA was selected for the Domino server and the intended userdoesn't exist in the SDD, mail is shipped toQIBM/USERDATA/LOTUS/SMTPMTA/INBOUND/NEW. Then it is processedby the Domino inbound session controller to be shipped by the inboundsession handler to the inbound work queue.

4. From there, the inbound message converter transfers the converted mail tothe MAIL.BOX for eventual delivery by the Notes router to the intended user.

The R4.6 flow of outbound mail follows this process:

1. When *SMTPMTA is selected during Domino configuration, an MTA GlobalDomain document, Foreign SMTP Domain document, and SMTP connectiondocument are created. Internet mail sent from a Domino user is handed by theRouter task to SMTP.BOX.

2. The outbound message converter transfers the converted mail to theoutbound work queue.

3. The outbound work queue then passes the mail, via the outbound sessioncontroller, to the outbound session handler for delivery to theQIBM/USERDATA/LOTUS/SMTPMTA/TMP IFS directory.

Again, you may not catch this flow unless MSF is ended, in which case you'llsee outbound mail pile up in this directory.


4. Once MSF is started, the Internet mail processes through MSF, and on to theAS/400 SMTP server for delivery to the Internet or mail relay if one isconfigured (set through CHGSMTPA).

11.2.2.2 Domino Release 5.0Domino mail serving at R5.x is simpler and more complex. It is simpler in thesense that either implementation you choose is easier to work with. It is morecomplex in that, due to the two choices, you need to make sure you're clear onwhich implementation you're using and why.

Depending on your prior mail environment, you have the option to use theOS/400 SMTP server or Domino SMTP functionality (new in Release 5.0). IfOV/400 or AS/400 POP3 users are involved, you should stay with *MSFimplementation. Otherwise, choose *DOMINO. From an end-user standpoint,either method is transparent, since mail will route through either method.

Administrators can switch between the two by using CHGDOMSVR, andselecting *SMTP or *ALL for “Internet Mail Packages”, and *MSF, *SMTP, or *ALLfor “Internet Mail Packages” and *DOMINO.

R5.x flow of inbound mail when configured for *SMTP and *MSFThe flow of Internet mail is much like it was at R4.6x. The exception is that, wheninbound mail is processed by MSF, it is passed on toQIBM/USERDAT/LOTUS/SMTPMTA/INBOUND, and no conversion is needed.Mail is routed directly to MAIL.BOX for the router to then deliver the mail to theintended recipient.

R5.x flow of outbound mail when configured for *SMTP and *MSFThe flow of Internet mail is much like it was at R4.6x. The exception is that whenoutbound mail is processed by MSF, it is passed on to theQIBM/USERDATA/LOTUS/SMTPMTA/OUTBOUND directory.

If *SMTP and *DOMINO are chosen for mail configuration, the AS/400 system nolonger acts as the SMTP server. Instead, the Domino server provides for this. Youcan confirm this through the OS/400 command NETSTAT *CNN and verify that theassociated user profile for SMTP is QNOTES instead of QTCP.

In this example, no MSF code is involved. In fact, the SMTP task doesn't need tobe running for outbound Internet mail to be delivered (the router is SMTP aware)and is only required to receive inbound Internet mail (so port 25 is listening).

Additional configuration is needed if a firewall or mail relay is involved. Theconfiguration setting (previously changed through the Change SMTP Attributes(CHGSMTPA) command on the AS/400 system) is now needed on theRouter/SMTP tab of the server configuration document. Also, the SMTPconnection, SMTP Foreign Domain, and MTA Global domain documents are nolonger needed.

The Global Domain document is needed if you desire to receive mail formultiple domains.

Note


11.2.2.3 Likely encountered mail problems and resolutions for R4.6If inbound Internet mail for a Domino R4.6x server is not being received, checkfor the following situations:

• If mail is not flowing, the first and best thing you can do is narrow the scope ofthe problem. If the problem is with inbound mail, end MSF and verify that mailis appearing in the QTCPTMM/SMTPBOX directory.

• If mail isn't going this far, it may be that the AS/400 system isn't accepting mailfor the domain, and changes need to be made to CFGTCP option 10 or option12. Otherwise, and more likely (especially if mail was flowing at some point),there is a problem with the Internet Service Provider (ISP) or firewall, if one isinvolved. Check with support for those products to see if logging is confirmingthat mail is making it that far.

To confirm that the ISP has the correct MX record for your mail domain, you canuse NSLOOKUP or several Internet Web sites to verify. Two examples are:

• http://www.baboon.ch

• http://www.samspade.org/

If the MX entry is incorrect, mail is likely not being forwarded to the properaddress (usually the non-secure port of your firewall, or an AS/400 IP interfaceaddress connected to your ISP router). If all checks out, make sure that SMTP isup and running on your AS/400 system (use netstat *cnn for port 25):

1. If mail makes it to the QTCPTMM/SMTPBOX, but doesn't go any further, theproblem is likely with MSF. Check the MSF job logs for information.

2. If mail disappears after entering MSF, end the Domino server's SMTPMTAtask and verify that mail is arriving in the QIBM/USERDATA/LOTUS/SMTPMTA/INBOUND/NEW directory. If not, the problem is likely with MSF.Possibly Domino registered exit programs aren't correct, or another AS/400directory entry is grabbing the mail.

A dump of the Mail Server Framework may be required to see what Anymailhas done with your message. The corresponding steps are described in11.3.9, “Dumping Mail Server Framework (MSF)” on page 218. Here is anexample of a dump file generated by MSF:

5769SS1 V4R2M0 980228 AS/400 DUMP 014026/QMSF/QMSF02/02/98 10:56:38 PAGE 1> MSF MESSAGE FORMATTED DUMP MSGID: ID 100304G9802021056030000000004 AT EXIT POINT :QIBM_QZMFMSF_ACT>>>>>MSF FORMAT CRTA0100000000 00000030 00000030 C3D9E3C1 F0F1F0F0 0000001C 00000001 00000000 *

CRTA0100 *>000000 F0F9F8F0 F2F0F2F1 F0F5F6F0 F3F7F6F6 F0F2C2F1

*098020210560376602B1 *>>>MSF FORMAT ORGL0100 01B0 SMTP originator000000 00000064 00000064 D6D9C7D3 F0F1F0F0 0000001C 00000001 00000000 * ´

´ORGL0100 *>000000 00000048 00000020 00000028 F0F1C2F0 000001F4 00000001 00000000 00000000 * ‡

01B0 4 *000020 D596A385 A26DC16D C4899393 8995877C D9C3C8C1 E2C5E2D7 4BD9C3C8 D3C1D5C4

*[email protected]*000040 4BC9C2D4 4BC3D6D4 *.IBM.COM

*


>>>>MSF FORMAT RCPL0100 Recipient still shows SMTP MIME (not system message store)03-Local Delivery000000 00000080 00000080 D9C3D7D3 F0F1F0F0 0000001C 00000001 00000000 *

RCPL0100 *>000000 00000064 00000058 0000000C 00000038 00000020 F0F1C2F0 000001F4 00000000 * ´

• 01B0 4 *000020 00000000 F0F2C2F1 00000003 00000000 00000002 00000000 84A78496 A5F47CD9 *

02B1 dxdov4@R*000040 C3C8C1E2 C5E2D74B D9C3C8D3 C1D5C44B C9C2D44B C3D6D400 C4E7C4D6 E5F40040

*CHASESP.RCHLAND.IBM.COM DXDOV4 *000060 40404060 * -

*>>>>MSF FORMAT ENVL0100000000 000000B0 000000B0 C5D5E5D3 F0F1F0F0 0000001C 00000002 00000000 * ^

^ENVL0100 *>000000 00000048 0000001C 0000002C F0F3C2F0 00000001 00000000 00000000 61D8C9C2 * ‡

03B0 /QIB*000020 D461E4E2 C5D9C4C1 E3C161D3 D6E3E4E2 61E2D4E3 D7D4E3C1 61A39497 61D8C1C3

*M/USERDATA/LOTUS/SMTPMTA/tmp/QAC*000040 E7E3C8D3 E7E5E300 *XTHLXVT

*>000000 0000004C 0000001C 00000030 F0F3C1C6 00000002 00000000 00000000 00000024 * <

03AF *000020 0000000C 00000003 0000000C 0000000A 00000001 0000000C 00000005 00000000 **000040 0000000C 00000016 00000001 *

*>>>>MSF FORMAT RCHL0100000000 000000E3 000000E3 D9C3C8D3 F0F1F0F0 0000001C 00000002 00000000 * T

TRCHL0100 *>000000 00000063 00000057 0000000C 00000038 0000001F F0F1C2F0 000001F4 40404040 * Ž

‹ 01B0 4 *000020 00000000 00000000 00000001 00000001 00000000 00000000 84A78496 A5F47C99 *dxdov4@r*000040 838881A2 85A2974B 99838893 8195844B 8982944B 839694F2 40404040 40404040

*chasesp.rchland.ibm.com2 *000060 404040 *

*>000000 00000064 00000058 0000000C 00000038 00000020 F0F1C2F0 000001F4 F0F2C2F1 * ´

• 01B0 402B1*000020 00000003 00000000 00000002 00000000 00000001 00000000 84A78496 A5F47CD9 *dxdov4@R*000040 C3C8C1E2 C5E2D74B D9C3C8D3 C1D5C44B C9C2D44B C3D6D400 C4E7C4D6 E5F40040

*CHASESP.RCHLAND.IBM.COM DXDOV4 *000060 40404060 * -

*>

PAGE 2>>>MSF FORMAT ATTL0100000000 00000064 00000064 C1E3E3D3 F0F1F0F0 0000001C 00000001 00000000 * ´

´ATTL0100 *>000000 00000048 0000001C 0000002C F0F4C2F7 00000001 00000000 00000000 61D8C9C2 * ‡

04B7 /QIB*000020 D461E4E2 C5D9C4C1 E3C161D3 D6E3E4E2 61E2D4E3 D7D4E3C1 61A39497 61D8C1C3

*M/USERDATA/LOTUS/SMTPMTA/tmp/QAC*000040 E7E3C8D3 E7E5E300 *XTHLXVT

*>>>


>MSF FORMAT MSGL0100000000 00000020 00000020 D4E2C7D3 F0F1F0F0 0000001C 00000001 00000000 *

MSGL0100 *>000000 F0F2C2F1 *02B1

*>>>>MSF FORMAT EXCH0100000000 0000016C 0000016C C5E7C3C8 F0F1F0F0 0000001C 00000004 00000000 * %

%EXCH0100 *>000000 D8C9C2D4 6DD8E9D4 C6D4E2C6 6DC1C4D9 6DD9E2D3 D8E3D4D4 C1C4D9E2 4040D8E3

*QIBM_QZMFMSF_ADR_RSLQTMMADRS QT*000020 C3D74040 40404040 00000514 F0F9F8F0 F2F0F2F1 F0F5F6F0 F5F8F1F1 F0F9F8F0 *CP

09802021056058110980*000040 F2F0F2F1 F0F5F6F2 F9F0F6F5 00000000 F1404040

*202105629065 1 *>000000 D8C9C2D4 6DD8E9D4 C6D4E2C6 6DD3C3D3 6DC4C5D3 D8E3D4D4 D7C1D9E2 4040D8E3

*QIBM_QZMFMSF_LCL_DELQTMMPARS QT*000020 C3D74040 40404040 00000640 F0F9F8F0 F2F0F2F1 F0F5F6F2 F9F1F6F0 F0F9F8F0 *CP

09802021056291600980*000040 F2F0F2F1 F0F5F6F3 F1F8F9F4 00000000 F0404040

*202105631894 0 *>000000 D8C9C2D4 6DD8E9D4 C6D4E2C6 6DC1E3E3 6DD4C7E3 D8E9C4E2 D5D7C1D4 4040D8E2

*QIBM_QZMFMSF_ATT_MGTQZDSNPAM QS*000020 E8E24040 40404040 000003E8 F0F9F8F0 F2F0F2F1 F0F5F6F3 F1F9F1F5 F0F9F8F0 *YS

Y09802021056319150980*000040 F2F0F2F1 F0F5F6F3 F2F0F1F3 00000000 F1404040

*202105632013 1 *>000000 D8C9C2D4 6DD8E9D4 C6D4E2C6 6DC1E3E3 6DD4C7E3 D8E3D4D4 C1E3E3D4 4040D8E3

*QIBM_QZMFMSF_ATT_MGTQTMMATTM QT*000020 C3D74040 40404040 00000514 F0F9F8F0 F2F0F2F1 F0F5F6F3 F2F0F7F0 F0F9F8F0 *CP

09802021056320700980*000040 F2F0F2F1 F0F5F6F3 F3F4F4F4 00000000 F1404040

*202105633444 1 *>>> END DUMP OF MSF MESSAGE MSGID: ID 100304G9802021056030000000004

3. If mail appears in the above IFS directory, start the Domino SMTPMTA task,and mail should arrive in MAIL.BOX. If not, the mail problem likely lies withSMTP inbound work queue.

4. If mail appears in MAIL.BOX and is never delivered, does the mail user exist?For example, is there a proper person document configured for this user toreceive mail?

If all is correct, the problem could lie with a corrupt MAIL.BOX (not likely ifother users are receiving mail) or with a corrupt user Domino mail database(USER.NSF).

If no users are receiving mail at this point, verify that the Domino server isconfigured to accept mail for that domain (based on the MTA Global domaindocument).

If Outbound Internet mail for a Domino R4.6x server is not being received, checkthe following situations in the order shown:

1. Confirm that mail is arriving in SMTP.BOX by ending the SMTPMTA task andopening SMTP.BOX.

2. Start the SMTPMTA task, and verify that mail is leaving the Domino server byending MSF, check QIBM/USERDATA/LOTUS/SMTPMTA/TMP, andconfirming mail message appears there.


3. If everything is OK in step 2, start MSF and verify that mail leaves the TMPdirectory. From there, MSF will process the mail and route it to the AS/400SMTP server for delivery. SMTP routes the mail directly to your ISP outer,unless a mail relay is involved, in which case the mail relay needs to bedefined under CHGSMTPA. If a mail relay or firewall is involved, you canconfirm that it is listening for port 25 requests. To do so, Telnet from theAS/400 system on port 25 to the secure port of the firewall or mail relay, andreceive the standard 220 reply.

11.2.2.4 Likely encountered mail problems and resolutions for R5.0For Release 5.0, mail can be configured for *MSF or *DOMINO. If configured for*MSF, troubleshooting is much the same as it was for R4.6. The exception is thatthe SMTPMTA task is no longer used, and SMTP.BOX, along with all the inboundand outbound conversion processing, are no longer relevant.

The IFS directory structure has also been simplified with the INBOUND, with itsthree directories being consolidated into just INBOUND and the outbound TMPdirectory being changed to OUTBOUND. Also, a deadletter directory is created.The *MSF implementation still relies on the AS/400 Mail Server Framework andSMTP server.

For Release 5.0 mail configured for Domino, you can have multiple serversrunning SMTP to allow for failover. Domino is now acting as the SMTP server,and you can verify this via the OS/400 NETSTAT CNN command, and confirm thatport 25 is associated with QNOTES profile. Note the following points:

• Outbound mail now leaves MAIL.BOX and, although still relying on the AS/400DNS settings, processes mail outbound via the Domino's SMTP server. MailRelay/Firewall settings are made via the Server\Configuration view and underthe Router/SMTP tab for “relay hosts for messages leaving Internet domain.”DNS settings on the AS/400 system (*local or *remote) should match thesettings under the Router/SMTP tab (local lookup only or dynamic then local).

• Inbound mail now processes automatically to MAIL.BOX and if not beingreceived, verify that mail domain is accepting mail for intended recipient. TheMTA Global domain document can be created if mail must be accepted mailmultiple domains.

11.2.2.5 Additional considerationsThe use of special NOTES.INI parameters is very important to this Internet maildebugging process. See 11.2.10, “Settings in the NOTES.INI file to obtain errorreporting” on page 203, for relevant R4.6x and R5.x settings.

It is also important to follow the right steps if upgrading from R4.6 to R5.0 forstable mail delivery.

An overview of the process to upgrade a Domino 4.6 server, using the MTA, to aDomino 5.0 mail server is shown here:

1. Backup important files.2. Disable the SMTP/MIME MTA housekeeping.3. End the AS/400 TCP/IP SMTP server.4. Shut down the inbound transport.5. Shut down the router.6. Shut down the outbound transport.7. End the AnyMail/400 Mail Server Framework.


8. Clear the SMTP.BOX.9. Clear the messages from the outbound MTA queue.10.Clear the messages from the inbound MTA queue.11.Clear messages from the /QIBM/UserData/LOTUS/SMTPMTA/TMP directory.12.Shut down the MTA server.13.Install the Domino Release 5 software.14.Select the type of SMTP support for your organization.15.Configure your SMTP server.

11.2.3 MAILROUTER parameterThis section applies to Domino Release 4.6 and Release 5.0 with SMTP services*MSF.

If you are configuring an AnyMail/400 server to send mail to the Internet, use theChange SMTP Attributes (CHGSMTPA) command to define the router name. Thisinformation is important and not self-evident.

When you start SMTP and have a MAILROUTER parameter configured, the client(sending task) immediately performs the gethostbyname function on this name toresolve the IP address. If the gethostbyname function fails, any mail sent to theInternet fails and the message is returned to the sender. It can fail if the routerhas not yet been set up in the DNS, even though it will have a valid address thenext day. It will continue to fail even after the DNS is set up. It will not work untilSMTP is restarted with a successful gethostbyname function. This is the only timethe gethostbyname is run.

11.2.4 HOSTS fileIf the AS/400 system does not have an entry in the local Host table for the domainname of the Domino server, the mail is not delivered. This is only a requirementfor AnyMail/400 (MSF) Integration of Lotus Domino for AS/400 and not theDomino SMTP option. You will not find the mail in any of the directories orMAIL.BOX. No message is returned to the sender. A clue is that you see SMTPprestart jobs starting and ending.

11.2.5 DNS serverIf an AS/400 DNS server is used, verify that the DNS server is started, and anactive QTOBDNS job exists in QSYSWRK subsystem. Check its job log forerrors. Verify that the IP interfaces to which the DNS server should be bound arestarted, including the Internet address listed on the same AS/400 system, usingthe Configure TCP/IP (CFGTCP) command option 12.

If changes or corrections have been made to the DNS server, make sure the DNSserver has been updated to pick up those changes.

Use the Start DNS Query (NSLOOKUP) command to verify that the DNS server isresponding with the answers you expect. For example, is the DNS serverresolving the SMTP domain name used to the right of the @ symbol in the Mail Toaddress? If not, this can be a problem, unless an alias is used in the AS/400 localhosts table, and Searched First =*LOCAL is used. For more information about theStart DNS Query (NSLOOKUP) command, refer to 11.3.4, “NSLOOKUP on theAS/400 system” on page 209.


11.2.6 The jobs or tasks that need to be verified that they are activeThe following jobs (tasks) must be active for the AS/400 Domino server to sendmail to different kinds of mail users:

• Router• SMTPMTA tasks (Domino Release 4.6)• TCP/IP• MSF (Domino Release 4.6 or 5.0 with SMTP services *MSF)

11.2.6.1 RouterThe Domino server router is used by all different kinds of mail setup and needs tobe active for all mail. Use the Domino console to check that router is active. Therouter starts automatically when the Domino server is started. If you have to startit manually, use the Domino console and key in load router.

11.2.6.2 SMTPMTA tasksFor Domino for AS/400 Release 4.6, SMTPMTA tasks need to be active in theDomino server subsystem. Use the SHOW TASKS command, at the Domino serverconsole to verify that all the following tasks are active:

• drt: Delivery report task• isesctl: Inbound session controller• iseshndl: Inbound session handler (up to eight)• imsgcnv: Inbound message conversion• omsgcnv: Outbound message conversion• osesctl: Outbound session controller• oseshndl: Outbound session handler (up to eight)

11.2.6.3 TCP/IPVerify that the SMTP client sending the mail, and the POP client receiving themail, have TCP/IP connectivity to their respective servers. Also, verify that eachclient can successfully ping their server by IP address. If the ping is notsuccessful, you need to debug a TCP/IP connectivity problem before proceedingto debug a mail problem:

• Make sure the appropriate AS/400 line descriptions are active.• Verify that the associated IP interface has been started on the AS/400 system.• Verify that the TCP/IP route exists if the client is on another subnet from the

SMTP, POP, or DNS server.

If the mail client is configured to have the SMTP Outgoing Mail Server orIncoming Mail Server to be a host.domain name rather than an IP address, verifythat a ping to the host name is successful. If a ping by IP address works but aping by host name fails, you need to debug a DNS problem before proceeding todebug a mail problem.

11.2.6.4 QMSF (AnyMail/400)This section applies to Domino Release 4.6 or 5.0 with SMTP services *MSF.

For mail to be successfully delivered on an AS/400 system, at least one QMSFjob needs to be active under the QSYSWRK subsystem. This job should autostartwhen the QSYSWRK subsystem goes active. However, certain errors can causethe QMSF job to end. Therefore, if mail is not being delivered, verify that QMSF isactive. To do so, issue the following command:


WRKACTJOB SBS(QSYSWRK)

QMSF should be listed as an active job. If it is not listed, you can start the QMSFjob by issuing the following command:

STRMSF

If you issue the Start Mail Server Framework (STRMSF) command and stillcannot find QMSF as an active job under QSYSWRK, the job may be starting butending right away with an error. If this is the case, the ended job log should bereviewed for error messages. The QMSF job runs using the QMSF user profile.To find the spooled file for the QMSF job log, issue the following command:

WRKSPLF QMSF

Press F18 to go to the bottom of the list. Many of these QMSF job log spooledfiles may be listed. Press the F11 key to display the date and time stamps ofthese jobs to help locate the one that you need.

If the QMSF job is active and mail is still not being delivered, check the activeQMSF job log for errors.

11.2.6.5 QMSF job error recoveryThere are two possible situations:

• The MSF messages are not processed.

Situations may occur in which you find that messages are not beingprocessed. Error messages or job log messages may indicate that the MailServer Framework stopped the processing of some messages and isbeginning to process new messages from the message queue.

To recover from this situation, check the job logs for the exit point and the exitprogram running when MSF stopped processing the messages. Use theinformation to determine where and why MSF stopped processing themessages.

• The Mail Server Framework ends.

If the MSF job in QSYSWRK ends, first try restarting MSF. If that does notresolve the problem, check for messages in QSYSOPR and in the job logs forthe QMSF jobs that are running that caused MSF to end. The Mail ServerFramework is finding something it cannot process, so it ends the framework.

A case where this may occur is if someone deleted the exit point program, butdid not remove it (unregister the program) from the exit point. You can use theWork with Registration Information (WRKREGINF) display to check for thissituation.

These messages may appear in the job log:

– CPFAF95: Job ended– CPFAF98: Message postponed

One of the following events may occur:

– Exit programs encounter severe errors causing the job to end.

– The Mail Server Framework encounters abnormal conditions.


– Exit programs encounter situations that stop the processing of MSFmessages and end the MSF job.

– One of the exit programs determines that an MSF message should bepostponed until the next STRMSF.

Use the Display Job Log (DSPJOBLOG) command to determine what the problem is.Display the message and use the recovery information to correct the problem.Then restart MSF.

The Start Mail Server Framework (STRMSF) command starts the Mail ServerFramework jobs in the system work subsystem (QSYSWRK). For moreinformation about QSYSWRK, see AS/400 Work Management, SC41-5306.There are two parameters you can specify on the STRMSF command.

The MSGOPT (Message Option) parameter specifies how the Mail ServerFramework processes existing MSF messages:

• *RESUME is the default option, which allows all existing MSF messages tocontinue processing from the point the Mail Server Framework was previouslyended.

• *RESET allows all existing MSF messages to be processed as if they werejust created. Users may receive duplicate messages when this parameter isused.

11.2.6.6 SMTPThis section applies to Domino Release 4.6 or 5.0 with SMTP services *MSF.

AS/400 SMTP runs in QSYSWRK. There can be one or more jobs started. If theserver is active, you should find four SMTP jobs in QSYSWRK:

• QTSMTPBRCL• QTSMTPBRSR• QTSMTPCLTD• QTSMTPSRVD

If the preceding jobs do not exist under QSYSWRK subsystem, enter thefollowing command to start the server:

STRTCPSVR SERVER(*SMTP)

Then, issue the command:


The display shown in Figure 90 appears.

*CLEAR deletes all existing MSF messages. Use the *CLEAR option onlywhen a software error is reported with the Mail Server Framework or itsassociated exit point programs that requires all MSF messages to be deleted.Use the option only when you want to remove all mail message traffic from thesystem because of errors. The possibility may exist that a message wasalready delivered before an error occurred. If this option is used, all messagesare deleted and cannot be recovered.

Important


Figure 90. SMTP jobs

11.2.6.7 AS/400 POP3 serverThe AS/400 POP3 server needs to be active for POP3 clients (NetscapeCommunicator, Eudora Pro, or Microsoft Outlook) to get the mail.

If the POP server is active, locate one or more jobs with the following threenames:

• QTPOP00622• QTPOP00635• QTPOP00681

The last five numbers in the POP job name can be any number. Also, even oneQTPOPxxxxx job active indicates that the POP server is active.

If the preceding jobs do not exist under QSYSWRK subsystem, then start thePOP server.

11.2.6.8 Starting the SMTP or POP3 serverThis section applies to Domino Release 4.6 or 5.0 with SMTP services *MSF.

Verify that the SMTP and POP servers are active. If they are active, theircorresponding jobs are listed as active jobs in the QSYSWRK subsystem. To seeif servers are started, enter the following command:


If servers are not started, start it with either or both of the following commands:

STRTCPSVR SERVER(*SMTP)STRTCPSVR SERVER(*POP)

If you use either or both of these commands, and still cannot find the associatedactive jobs in the QYSWRK subsystem, it is possible that these jobs are startingbut ending before you can locate them. First, check for any errors in the user job




Opt Subsystem/Job User Type CPU % Function StatusQTRTD05912 QTCP BCH .0 PGM-QTORSRVR SELWQTSMTPBRCL QTCP BCH .0 PGM-QTMSBRDG DEQWQTSMTPBRSR QTCP BCH .0 PGM-QTMSBRSR DEQWQTSMTPCLTD QTCP BCH .0 PGM-QTMSCLCD DEQWQTSMTPSRVD QTCP BCH .0 PGM-QTMSSRCD SELWQTTFT00077 QTFTP BCH .0 SELWQTTFT00084 QTFTP BCH .0 DEQWQTTFT00088 QTFTP BCH .0 DEQWQTTFT00097 QTFTP BCH .0 DEQW



log that issued the STRTCPSVR commands. If your own interactive job was usedto issue the commands, review your own job log with the following command:

DSPJOBLOG

Press Enter, followed by pressing F10. Then, page up to look for error messages.

Also, if the SMTP or POP jobs end with an error, review their spooled job logs forerror messages. These jobs run using the QTCP user profile. Therefore, to findthe spooled job logs of the inactive jobs, use the following command:

WRKSPLF QTCP

Press Enter, followed by F18, to go to the bottom of the list. The job name isusually displayed in the User Data field in the Work With Spooled Files display.

If the SMTP and POP jobs are active, and mail is still not being delivered, alwayscheck the SMTP and POP active job logs for any error messages. Any errormessages in these job logs can give you clues as to what is going wrong.

You can also enter the NETSTAT *CNN command on an AS/400 command line andpress F11. You will be able to see if SMTP and POP3 are active, as shown inFigure 91. In this example, the AS/400 SMTP is started, since QTCP is the user.If it is the Domino SMTP that is started, the user is QNOTES.

Figure 91. NETSTAT *CNN with F11 pressed

If changes to the AS/400 TCP/IP domain or host table were made with theCFGTCP command, option 12 or option 10, the SMTP server needs to beended and started again to pick up the changes.

Tip

Work with TCP/IP Connection StatusSystem: AS25

Local internet address . . . . . . . . . . . : *ALL

Type options, press Enter.4=End 5=Display details

Remote Remote LocalOpt Address Port Port User Bytes Out Bytes In

* * ftp-con > QTCP 0 0* * telnet QTCP 0 0* * smtp QTCP 0 0* * tftp QTFTP 0 0* * www-http QTMHHTTP 0 0* * kerberos QTMHHTTP 0 0* * pop3 QTCP 0 0* * netbios > QSYS 0 0* * netbios > QSYS 837172 10647976* * netbios > QSYS 0 4512037* * netbios > QSYS 0 0

More...F5=Refresh F11=Display connection type F13=Sort by columnF14=Display port numbers F22=Display entire field F24=More keys


If a Domino SMTP is listening on port 25, and the displayed user is QNOTES, youmust end that server before you can start AS/400 AnyMail Integration for SMTP.

If you start SMTP while a Domino server is listening on port 25, QTSMTPXXXXjobs will start and end. Running the DSPMSG QSYSOPR command shows an errormessage: SMTP job QTSMTPXXXX ended abnormally.

End Domino server that is configured with *DOMINO and start SMTP again.

11.2.6.9 OfficeVision/400For OfficeVision/400 or Object Distribution, both the QSYSWRK and QSNADSsubsystems need to be active. Unfortunately, it is not an easy problem todiagnose (if you miss the fact that the tasks are not running) because there isnothing obvious in terms of where the mail has gone. It is sitting in QZDINDEXwaiting for the QDIA* processes to handle it, but you cannot see where it is. Thebest advice is to verify that the subsystem QSNADS is active.

The SNADS distribution log can prove useful. You can use the DisplayDistribution Log (DSPDSTLOG) command, and press F4 for prompting. You canaccess this screen by typing GO NETWORK, pressing Enter, and selecting option 9.

11.2.7 Where to look for the mail that doesn’t flow throughThere are different databases and directories where the mail can be, dependingon several factors:

• The Domino for AS/400 release• Whether SMTP is configured to use Domino’s native SMTP or AS/400 SMTP• Whether you are using POP3

11.2.7.1 SMTP MTA and MSF (R4.6 AS/400)Databases under .../notes/data/ directory, created by the SMTPMTA task, are:

• smtpibwq.nsf (SMTP MTA inbound work queue)• mail.box• smtp.box• smtpobwq.nsf (SMTP MTA outbound work queue)

Directories under the /QIBM/UserData/LOTUS/SMTPMTA directory are:

• tmp: Used by oseshndl for creating outbound MIME

• inbound/: Used by iseshndl when working on inbound MIME and itssubdirectories:

– mail– new– inprog

• /QTCPTMM/ATTABOX (AS/400)

• /QTCPTMM/MAIL (AS/400)

• /QTCPTMM/SMTPBOX (AS/400)

• /QTCPTMM/TMP (AS/400)

11.2.7.2 Release 5.0 server integrated with AS/400 SMTP (MSF)The mail.box database, created as part of the server by the configuration underthe .../notes/data/ directory, is a possible place.


Directories under /QIBM/UserData/LOTUS/SMTPMTA are:

• inbound• outbound• deadletter• /QTCPTMM/ATTABOX (AS/400)• /QTCPTMM/MAIL (AS/400)• /QTCPTMM/SMTPBOX (AS/400)• /QTCPTMM/TMP (AS/400)

11.2.7.3 R5 Server Domino SMTP supportThe following places can contain the mail:

• Inbound messages are stored in MAIL.BOX• Outbound messages are stored in MAIL.BOX• POP3 mail

11.2.8 Accessing the AS/400 directoriesThe WRKLNK command can be used to look at, for example/QIBM/UserData/LOTUS/SMTPMTA/OUTBOUND, as shown in Figure 92.

Figure 92. Outbound directory

The Display File (DSPF) command can be used to look inside these objects toensure that it is the mail item that is being stuck. For the example above, thecommand is:

DSPF ‘/QIBM/UserData/LOTUS/SMTPMTA/OUTBOUND/qacx5ctdgz'


Directory . . . . : /QIBM/UserData/Lotus/SMTPMTA/OUTBOUND


Opt Object link Type Attribute TextQACX5CTDGZ STMF

BottomParameters or command===>F3=Exit F4=Prompt F5=Refresh F9=Retrieve F12=Cancel F17=Position toF22=Display entire field F23=More options

For details about the Display File (DSPF) command, refer to 7.1, “Displaying orediting a stream file from a 5250 display” on page 117.

Note


An easier way may be to access the directory by using the following command:

WRKLNK (‘/QIBM/UserData/LOTUS/SMTPMTA/OUTBOUND/')

Then, select option 11 to change the current directory. Next, on the commandline, type:

DSPF qacx5ctdgz

If the object has a long name as shown in Figure 93, you put your cursor on theObject link field Q09..., and press F22 (shift F10). You will see the full objectname as shown in Figure 94, which you can use with the Display File (DSPF)command.

Figure 93. Work with Object Links display

Figure 94. Full object name


Directory . . . . : /QTCPTMM/SMTPBOX


Opt Object link Type Attribute TextQ0919637FAB302B393 > STMF




..............................................................................: Object link :: /QTCPTMM/SMTPBOX/Q0919637FAB302B393F000000.NOTE :: :: :: :: :: :: Bottom :: F12=Cancel :: ::............................................................................:



11.2.9 What happens when jobs or tasks are not activeThis section shows you some examples of where the mail is located.

11.2.9.1 MSF is not activeThis section applies to Domino Release 4.6 or 5.0 with SMTP services *MSF.

If the AS/400 MSF jobs are not active, what happens when you send mail?

Mail that is sent from a Notes user to POP3 user will stay in the/QIBM/UserData/LOTUS/SMTPMTA/OUTBOUND directory as shown in Figure95.

If there is nothing in the OUTBOUND directory, the problem is with Dominocreating and sending the mail item and not MSF. Note that there is no errormessage if MSF is not running.

Figure 95. The /QIBM/UserData/LOTUS/SMTPMTA/OUTBOUND directory

Mail that is sent from a POP3 user to a Notes user will be in the/QTCPTMM/SMTPBOX directory because MSF is not active and can’t delivermail. See Figure 96 for an example.

If the mail is not in the SMTPBOX directory, AS/400 SMTP has not received anddelivered the message to MSF. You can use a communications trace to see ifthere is any communication between the two servers.


Directory . . . . : /QIBM/UserData/LOTUS/SMTPMTA/OUTBOUND


Opt Object link Type Attribute TextQACXTL5B1X STMF



Figure 96. The /QTCPTMM/SMTPBOX directory

Mail that is sent from an OfficeVision/400 user to a Notes user or POP3 user sitsin QZDINDEX waiting for QDIA* processes to handle it. The best advice is toverify that MSF and the subsystem QSNADS are active.

The SNADS distribution log can be useful. You can use the Display DistributionLog (DSPDSTLOG) command and press F4 for prompting. To access this display,(Figure 97), type GO NETWORK, press Enter, and select option 9.

Figure 97. Display Distribution Services Log




Opt Object link Type Attribute TextQ01176480432AFEB03 > STMF


Display Distribution Services Log

Type options, press Enter.5=Display details

Function Entry ------Logged------ ----Originator---- SeqOpt Type Type Date Time Job Name User ID Address Nbr

*ORG *NRM 2/23/00 10:15:31 QPADEV000P ASMITH AS25 0009

Bottom

F3=Exit F12=Cancel


11.2.9.2 SMTP is not activeThis section applies to Domino Release 4.6 or 5.0 with SMTP services *MSF.

If the AS/400 SMTP jobs are not active, what happens when you send mail?

Use the WRKACTJOB command so you can see the QMSF jobs. Select option 5 andoption 10 to view the job logs. One of the jobs will contain the message SMTP

distribution not sent. SMTP jobs not active., as shown in Figure 98. You also seethe messages in QSYSOPR message queue. Use the DSPMSG QSYSOPR commandto find the message.

Mail sent to your server, while SMTP is down, will be held somewhere on thesending side, most likely at the Internet Service Provider (ISP) site, for a certainlength of time before being declared undeliverable.

Figure 98. QMSF job log

Mail that is sent from a Notes user to a POP3 user stays in the/QTCPTMM/MAIL/JSPRING directory, as you can see in Figure 99, sinceJSPRING cannot contact the server.

Display Job LogSystem: AS02

Job . . : QMSF User . . : QMSF Number . . . : 123534

Job 123534/QMSF/QMSF started on 10/25/99 at 13:16:18 in subsystem QSYSWRKin QSYS. Job entered system on 10/25/99 at 13:16:18.

Job 123534/QMSF/QMSF submitted.Output file DSTLINFO created in library QTEMP.Member DSTLINFO added to output file DSTLINFO in library QTEMP.SMTP distribution not sent. SMTP jobs not active.Job 123534/QMSF/QMSF stopped processing MSF message.


F3=Exit F5=Refresh F10=Display detailed messages F12=CancelF16=Job menu F24=More keys


Figure 99. JSPRING mailbox

Mail that is sent from a POP3 user to a Notes user or OfficeVision/400 is notdelivered. The server is not responding as you can see in Figure 100.

Figure 100. No response

To start SMTP, enter the command:

STRTCPSVR SERVER(*SMTP)

Verify that the SMTP is running by using the Work with TCP/IP Network Status(NETSTAT) command. On any AS/400 command line, type:

NETSTAT *CNN


Directory . . . . : /QTCPTMM/MAIL/JSPRING


Opt Object link Type Attribute TextJW571764.NOT STMF



You see a display, as shown in Figure 101. When you press F14 to show the portnumbers, make sure you see a column with Local Port showing SMTP or thenumber 25. Confirm that it is AS/400 SMTP by typing a 5 in the Option column.The associated user profile is QTCP for AS/400 SMTP.

If the associated user profile is QNOTES, Domino SMTP is started.

Figure 101. Work with TCIP/IP Connection Status display

If a Domino SMTP is listening on port 25, you must end the server before you canstart AS/400 AnyMail Integration for SMTP.

If starting SMTP while a Domino Server is listening on port 25, QTSMTPXXXXjobs will start and end. Entering the DSPMSG QSYSOPR command results in theerror message: SMTP job QTSMTPXXXX ended abnormally.

End the Domino Server that is configured with *DOMINO, and start SMTP again.

Enter the following command:



Work with TCP/IP Connection StatusSystem: SYSTEM01



Remote Remote LocalOpt Address Port Port Idle Time State

* * ftp-con > 013:54:54 Listen* * telnet 002:07:03 Listen* * smtp 013:54:34 Listen* * domain 013:54:54 Listen* * domain 013:54:54 *UDP* * bootps 000:01:07 *UDP* * tftp 013:54:39 *UDP* * www-http 013:54:50 Listen* * pop3 013:54:37 Listen* * 119 004:37:16 Listen* * 143 004:37:25 Listen

More...F5=Refresh F11=Display byte counts F13=Sort by columnF14=Display port numbers F22=Display entire field F24=More keys


Figure 102. SMTPJOB

11.2.9.3 The Domino router is not activeIf the Domino mail router is not active, what happens when you send mail?

If the Domino mail router is not active and you send mail from a POP3 user orOfficeVision/400 user to a Notes user, the mail stays in the/QIBM/UserData/LOTUS/SMTPMTA/INBOUND directory. See Figure 103 onpage 202.




Opt Subsystem/Job User Type CPU % Function StatusQTRTD05912 QTCP BCH .0 PGM-QTORSRVR SELWQTSMTPBRCL QTCP BCH .0 PGM-QTMSBRDG DEQWQTSMTPBRSR QTCP BCH .0 PGM-QTMSBRSR DEQWQTSMTPCLTD QTCP BCH .0 PGM-QTMSCLCD DEQWQTSMTPSRVD QTCP BCH .0 PGM-QTMSSRCD SELWQTTFT00077 QTFTP BCH .0 SELWQTTFT00084 QTFTP BCH .0 DEQWQTTFT00088 QTFTP BCH .0 DEQWQTTFT00097 QTFTP BCH .0 DEQW


You must end and restart MSF after you start SMTP.

Important


Figure 103. Inbound

If the Domino mail router is not active, and mail is sent from Notes user to anyother kind of mail user, the mail stays in the MAIL.BOX database on your Dominoserver.

The MAIL.BOX database at the server can be opened from a Notes client. Thename have to be explicitly typed in the filename field in theFile->Database->Open dialog, because it does not end in the characters NSF.Once you press Esc to leave the About this Database document, you see adisplay similar to the example in Figure 104.

Figure 104. MAIL.BOX content

Enter the Work with Domino Servers (WRKDOMSVR) command, and select theDomino server. Then choose option 8 to work with the server console, and runload router to start the router.

11.2.9.4 The Domino SMTP is not activeIf the Domino SMTP is stopped, what happens when you send mail?


Directory . . . . : /qibm/userdata/lotus/smtpmta/inbound


Opt Object link Type Attribute TextQACXQM9MQH STMFQACXQM9MQHB STMF



If Domino SMTP is not active and you send mail from a POP3 user to Notes user,the mail is not delivered, because the SMTP task is not there to receive it. SeeFigure 105.

Figure 105. Server is not responding

If the Domino SMTP is not active and you send mail from a Notes user to a POP3user, the mail will be delivered. The reason is the Domino router is SMTP awareand was able to send the mail anyway. A message at the Domino console willshow that the mail was sent out normally.

How do you stop outbound mail? Edit the Configuration document to disableSMTP (under the Router/SMTP tab and Basics sub-tab), and restart the server.

To start Domino SMTP, type load smtp at the Domino console.

11.2.10 Settings in the NOTES.INI file to obtain error reportingThe following parameters can be used to obtain some level of error messagingback from the Router task (to be added to the NOTES.INI file):

• debug_router = 0, 1, 2, 3

– Usage: 1 (to collect router messaging)

• 1: Enables additional routing logging• 2: Enables routing tables traces• 3: Both of the above

– Default: 0

• log_mailrouting = 10, 20, 30, 40 (1, 2, 3, 4 in R3)

– Usage: 40 (to collect debug information). This parameter controls theamount of logging performed during message routing.

– Default: 20


• shared_mail = 0, 1, 2

– Usage: Specifies whether the shared mail feature is used for new maildelivered to this server:

• 0: The shared mail feature is not used for new mail

• 1: The shared mail feature is used for new mail delivered to this server

• 2: The shared mail feature is used for new mail delivered to this serverand for new mail transferred through this server

– Default: 0

• smtpdebug = 0, 1, 2, 4

– Usage: To collect SMTP logging information

• 1: Logs errors• 2: Logs protocol commands

– 4: Collects SMTP messaging

– Default: 0

• SmtpClientDebug = 0, 1

– Usage: 1 (to collect inbound SMTP messaging)– Default: 0

• SmtpIODebug = 0,3

– Usage: 0 (no logging)– Usage: 3 (logs all data received by the SMTP task)– Default: 0

• SMTPSaveImportErrors = 0, 1, 2

– Usage: 0 (no messages are recorded)

– Usage: 1 (when an arriving message fails to be written as a note inMAIL.BOX, Domino writes the data stream to a temporary directory andlogs the name of the file)

– Usage: 2 (all arriving messages have their data streams written to thetemporary directory)

Before enabling any debug parameters, it is imperative that you discuss themwith a Lotus Notes/Domino Support Analyst. There may be issues surroundingtheir use or special precautions that must be considered. These debugparameters may require a large amount of disk space, depends on when theserver encounters problems. The longer the server stays up, the larger thedebug files will be. These files can grow large enough to cause disk spaceshortages.

Also, the content of the messages are accessible to anyone with the privilegesto read files in the temporary directory.

Attention


11.2.11 Helpful NOTES.INI parametersThese parameters concern the format of the date and time:

• Dateorder = DMY• Clocktype = 24_Hour• Dateseparator = .• Timeseparator= :

These parameters can be helpful when national settings are required.

11.3 Tools and traces for troubleshooting

Some of the tools and traces use CPU or disk space. It is important to rememberto turn them off when you finish with your investigation.

Remember that with any of these tools, you may have to use the AS/400 SupportCenter to interpret the data that you collect. However, using of the tools maytrigger some ideas on how to solve the problem, and it will make it easier to givethe information to the support center if you are familiar with their use.

Do not forget to use the Domino console when looking to see if mail is beingtransferred or received (WRKDOMSVR option 8). You may prefer to use the DominoAdministrator interface, either the console or the mail log:

• Console commands• TELNET to test SMTP traffic• NETSTAT command• NSLookup on the AS/400 system• NotesConnect (Nping)• Domino Message tracking for reports and tracing requests• Mail probes• Communications trace• Collecting pre-V4R4 SMTP flight recorders• Collecting V4R4 flight Recorders• AnyMail/MSF dump snap-in• Dumping Mail Server Framework (MSF)• Component journaling

11.3.1 Console commandsThe following Domino server console commands can help you to understandwhat is the actual mail activity running on the Domino server and to detect anyanomaly.

• "tell router show"

• "show task time"

• "show stat mail by dest"

• "show stat mail"

For example, the show stat mail by dest and show stat mail commands show ifthere is any problem accessing the Domain Names System, through the statisticMAIL.WaitingForDNS:


Msgs State Via Destination

Transfer Threads: Max = 25; Total = 0; Inactive = 0; Max Concurrent = 12Delivery Threads: Max = 25; Total = 0; Inactive = 0

11.3.2 Using Telnet to test SMTP trafficIf the sending mail server’s resolver successfully obtains the mail server that ishandling mail for the domain of the recipient, the sending SMTP server attemptsto open a connection to the remote (receiving) SMTP server on Port 25. Theusual TCP connectivity of SYN, SYN ACK, and ACK take place.

11.3.2.1 When to use TelnetYou can use the Telnet command to test basic SMTP connectivity, that is, to see ifSMTP is working on your server without the use of your e-mail client. You canalso use it with other TCP/IP services to confirm that you are connecting to thecorrect server. This can be done from either an AS/400 display or from your PC.

Normally, when you use TELNET to access a signon screen on a computer, youare using the well-recognized port 23, which is the default. This time, we use theSMTP port, which is port 25.

11.3.2.2 How to use TelnetIn this section, you may have to use uppercase, or you may have to uselowercase. If one format does not work, try the other. Also, try not to makemistakes in a command line because the SMTP server does not toleratebackspaces. Or you will get error code 501, Syntax error.

From an MS-DOS window, type:

TELNET dommailservxxa 25

Select Terminal->Preferences and check Local Echo.

Follow this process:

1. Type:

HELO BILLSMITH

You are telling the destination computer who you are. The destinationcomputer responds to you and tells you that it is pleased to meet you.

2. Type MAIL FROM: [email protected]

3. Type RCPT TO: [email protected]

4. Type DATA

5. Type the message you want to send, and complete it by typing a period (.) ona line by itself.

6. The message 250 Message accepted for delivery is received at the TELNETsession.

7. The Domino console shows the message: 02/12/2000 02:46:14 PM Router:

Message 0072188B delivered to userX

8. A Notes user receives mail.

9. If mail is sent to a nonexistent user, the Domino console shows the followingmessages:


02/12/2000 02:54:29 PM SMTP Server: Message 5D0034E8 received 02/12/2000

02:54:29 PM Router: Unable to deliver message 0072DA0A [email protected]

02/12/2000 02:54:29 PM User [email protected] not listed inpublic Name & Address Book

To run the TELNET command from an AS/400 command line to the Dominoserver, press F4. This will allow you to enter the port number 25. You may have touse EHLO rather than HELO to make the connection.

Note that, if you are using SMTP commands to AS/400 SMTP, you have to enterthe commands in this way (note angle brackets):

MAIL FROM: <MYNAME.MYDOMAIN>

Here is a summary of SMTP commands and return codes used with TELNET tothe SMTP server. You can find additional information in the RFC821 document at:http://info.internet.isi.edu/in-notes/rfc/files/rfc821.txt

• HELO <domain>: The HELO command is used to identify the sender-SMTPto the receiver-SMTP. The argument field contains the host name of thesender-SMTP.

• MAIL FROM:<reverse-path>: The MAIL command is used to initiate a mailtransaction in which the mail data is delivered to one or more mailboxes. Theargument field contains the reverse-path that can be used to returnnon-deliverable mail.

• RCPT TO:<forward-path>: The RECIPIENT command is used to identify anindividual recipient of the mail data. Multiple recipients are specified bymultiple use of this command.

• DATA: The receiver treats the lines following the command as mail data fromthe sender. This command causes the mail data from this command to beappended to the mail data buffer. The mail data may contain any of the 128ASCII character codes. The mail is terminated by a line containing only aperiod (.).

• QUIT: This command specifies that the receiver must send an OK reply, andthen close the transmission channel.

• RSET: The RSET command specifies that the current mail transaction is to beaborted. Any stored sender, recipients, and mail data must be discarded, andall buffers and state tables cleared. The receiver must send an OK reply.

The TELNET command can also be used to connect to other services. Fromyour DOS prompt, type:

TELNET popserver.popdomain.com 110

This will connect to the POP3 server. This is one way to confirm that the POP3server is active.

If you are trying to send SMTP mail from a Domino server through the firewall,you can eliminate the Domino SMTP server by changing your POP3 client topoint directly to the firewall port.

Tips


Here is a list of SMTP reply codes:

211 System status or system help reply

214 Help message: Information on how to use the receiver or the meaning ofa particular non-standard command; this reply is useful only to the humanuser

220 <domain> Service ready

221 <domain> Service closing transmission channel

250 Requested mail action okay, completed

251 User not local; will forward to <forward-path>

354 Start mail input; end with <CRLF>.<CRLF>

421 <domain> Service not available, closing transmission channel: This maybe a reply to any command if the service knows it must shut down

450 Requested mail action not taken: mailbox unavailable, for example,mailbox busy

451 Requested action aborted: local error in processing

452 Requested action not taken: insufficient system storage

500 Syntax error, command unrecognized: This may include errors such ascommand line too long

501 Syntax error in parameters or arguments

502 Command not implemented

503 Bad sequence of commands

504 Command parameter not implemented

550 Requested action not taken: mailbox unavailable, for example, mailboxnot found, no access

551 User not local; please try <forward-path>

552 Requested mail action aborted: exceeded storage allocation

553 Requested action not taken: mailbox name not allowed, for example,mailbox syntax incorrect

554 Transaction failed

11.3.3 NETSTATTo verify that required applications are running, you can use the Work withTCP/IP Network Status (NETSTAT) command. On any AS/400 command line,type NETSTAT *CNN. The screen shown in Figure 106 appears.


Figure 106. Work with TCIP/IP Connection Status display

For mail to be routed, SMTP needs to be in listen state. Look for SMTP in the list.Press F14 to see the port it is listening on, which is default port 25.

Press F11 to see the user that is using the job. As an alternative to pressing F11,you can select option 5 to display details.

If your Domino server is configured for native Domino SMTP (*DOMINO), theuser should be QNOTES.

If your Domino server is configured for AS/400 SMTP (*MSF), the user should beQTCP.

If no application is listening to the SMTP port, try to start the SMTP server byusing the Start TCP/IP Server (STRTCPSVR) command. On any AS/400command line, type:

STRTCPSVR *SMTP

Check again if the SMTP server is listening to port 25. If it is not, display thesystem operator message queue (use the DSPMSG QSYSOPR command) and look forerror message: SMTP job QTSMTPSRVR ended abnormally.

This display can also be useful for POP and LDAP to confirm that the correct jobsare running.

For more information about the Work with TCP/IP Network Status (NETSTAT)command, refer to 14.1, “NETSTAT command” on page 265.

11.3.4 NSLOOKUP on the AS/400 systemNSLOOKUP is an interactive query tool that can be used to retrieve informationfrom, or test the response of, a DNS server.

Work with TCP/IP Connection StatusSystem: SYSTEM01




* * ftp-con > 013:54:54 Listen* * telnet 002:07:03 Listen* * smtp 013:54:34 Listen* * domain 013:54:54 Listen* * domain 013:54:54 *UDP* * bootps 000:01:07 *UDP* * tftp 013:54:39 *UDP* * www-http 013:54:50 Listen* * pop3 013:54:37 Listen* * 119 004:37:16 Listen* * 143 004:37:25 Listen



More information about this command is available in 14.3, “NSLOOKUPcommand” on page 269.

11.3.5 NotesConnectHow can you verify that an End-to-End TCP/IP connection can be establishedwithout using the Domino Directory or Notes address resolution logic?

The NotesConnect V2.0 TCP/IP Diagnostics Tool can be used for this function.This tool is available on the Internet from Notes.net at: http://www.notes.net

More information about NotesConnect is available in 14.7, “NotesConnectdiagnostic aid” on page 274.

11.3.6 Collecting SMTP Flight RecordersSMTP Flight Recorders are tools for tracing SMTP distributions.

Each of the four SMTP jobs use a flight recorder to collect trace information asdistributions are processed by SMTP. A feature for generating extra traceinformation as a debug aid can be enabled by a user-created data area. Thevalues in the data area serve as flags to determine which jobs to generate debuginformation for and where to direct the output of the debug information. Databasefiles are created for each of the four jobs to collect permanent debug information.

Take care when using the SMTP Flight Recorders. The flight recorders must beturned off after the data is collected. If they are left on, the flight recorders couldfill all of your DASD.

More information is available in TCP/IP Configuration and Reference,SC41-5420.

There are two types of SMTP Flight Recorders that can be collected:

• Pre-V4R4 flight recorders• V4R4 flight recorders

11.3.6.1 Collecting pre-V4R4 SMTP Flight RecordersFollow these steps:

1. On the OS/400 command line, type:

ENDTCPSVR *SMTP

2. Remember to give the SMTP servers enough time to shut down. To checkthem, on the OS/400 command line, type:


The following Server Jobs should not be present:

• QTSMTPCLNT• QTSMTPSRVR• QTSMTPBRCL• QTSMTPBRSR


CALL QCMD


Note: This starts the command line screen. To exit the command line screen,press the F12 function key.


DLTF FILE(QUSRSYS/QTMSFLRCF*)


CRTDTAARA DTAARA(QUSRSYS/QTMSTRCNTL) Type(*CHAR) + VALUE('11111100')TEXT('SMTP Debug control data area')


STRTCPSVR *SMTP

7. Verify that all four SMTP Jobs are active. On the OS/400 command line, type:

WRKACTJOBSBS(QSYSWRK)

The following SMTP server jobs should be running in QSYSWRK:

• QTSMTPCLNT• QTSMTPSRVR• QTSMTPBRCL• QTSMTPBRSR

8. Run the test scenario that is causing the issue.


ENDTCPSVR *SMTP

Allow time for SMTP to shut down. Then, type:

DLTDTAARA DTAARA (QUSRSYS/QTMSTRCNTL)

10.On the OS/400 command line, type:

STRTCPSVR *SMTP

11.To view the SMTP Flight Recorders, type the following command on theOS/400 command line:

WRKF FILE(QUSRSYS/QTMSFLRCF*)

Table 13 lists the expected files.

Table 13. Spooled files generated by the SMTP Flight Recorder

11.3.7 Collecting V4R4 flight recordersIn V4R4, new SMTP servers were designed along with a new method ofcollecting SMTP data. The TRCTCPAPP tool captures SMTP inbound andoutbound connections.

Trace file Job number SMTP job

QUSRSYS/QTMSFLRCF0

QUSRSYS/QTMSFLRCF1 1 QTSMTPCLNT

QUSRSYS/QTMSFLRCF2 2 QTSMTPSRVR

QUSRSYS/QTMSFLRCF3 3 QTSMTPBRCL

QUSRSYS/QTMSFLRCF4 4 QTSMTPBRSR

QUSRSYS/QTMSFLRCF5


A PTF that is required for the V4R4 flight recorders is PTF SF58088 (APARSA81557, 5769TC1).

There are two TRCTCPAPP parameter settings for mail on the AS/400 system.They are explained in the following sections.

• The *SMTPSVR parameter specifies tracing for the SMTP server handlinginbound mail processing connections.

For SMTP inbound connections (*SMTPSVR), the trace information can befiltered by:

– Remote IP address– Port– Recipient mail address

• The *SMTPCLT parameter specifies tracing for the SMTP client handlingoutbound mail processing connections.

For SMTP outbound connections (*SMTPCLT), the trace information can befiltered by:

– Recipient host name– Recipient address– Mail exchanger information

11.3.7.1 SMTP inbound connections *SMTPSVRTo trace the SMTP server, complete these tasks:

1. On the AS/400 command line, type:

TRCTCPAPP

2. Press F4 to prompt. The screen shown in Figure 107 appears.

3. Set the TCP/IP application parameter to *SMTPSVR.

For a given application, only one trace instance can be active at a time.Therefore, the command can only be used by one user at a time.

Restrictions


Figure 107. Trace TCP/IP Application (TRCTCPAPP): SMTPSVR

Each of the parameters has a specific meaning. The descriptions are listed inTable 14.

Table 14. Parameters for TRCTCPAPP APP(*SMTPSVR)

Parameter Option Description

TCP/IP application *SMTPSVR Used for inbound mail.

*SMTPCLT Used for outbound mail.

Trace option setting *ON The collection of trace information isstarted.

*OFF The collection of trace information isstopped. The trace information is writtento the spooled printer file.

*END Tracing is ended, and all traceinformation is purged. No traceinformation output is created.

*CHK The status of tracing for the specifiedapplication is checked. Messages arereturned indicating whether tracing isactive for the specified TCP/IPapplication the command parametersspecified format the last time thatTRCTCPAPP was started for thisapplication, and other information relatedto the collection of trace information.

Maximum storagefor trace

*APP For *SMTPSVR or *SMTPCLT - 4096 KBper job.

Trace TCP/IP Application (TRCTCPAPP)


TCP/IP application . . . . . . . > *SMTPSVR *FTP, *SMTPSVR, *SMTPCLT...Trace option setting .. . . . . *ON *ON, *OFF, *END, *CHKMaximum storage for trace . . . *APP_______ 1-16000, *APPTrace full action . . . . . . . *WRAP___ *WRAP, *STOPTRCRecipient mail address . . . . . ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________Remote network address:Address family . . . . . . . . ________ *INETIP address . . . . . . . . . . _________________Subnet mask . . . . . . . . . '255.255.255.255'Port number . . . . . . . . . *ANY_______ 1-65535, *ANY



11.3.7.2 SMTP outbound connections *SMTPCLTTo trace the SMTP client, start TRCTCPAPP from the command line of the AS/400system. Press F4. The display shown in Figure 108 appears. For the availableparameters, see Table 15.

maximum KB Specify the maximum amount of storage,in KB used to store trace records (one KBequals 1024 bytes).

Trace full action *WRAP When the trace buffer is full, the tracewraps to the beginning. The oldest tracerecords are written over by new ones asthey are collected.

*STOPTRC Tracing stops when the trace buffer is fullof trace records.

User Profile The parameter is not used with*SMTPSVR or *SMTPCLT.

Recipient mailaddress

recipient-mail-address Only trace information associated with aspecific recipient mail address will becollected. This parameter is only validwhen APP(*SMTPSVR) orAPP(*SMTPCLT) is specified.Note: The recipient mail address (up to255 characters) must have the followingformat: [email protected]

Remote NetworkAddress

Address Family *INET The filter for AF_INET address family.

IP Address The remote TCP/IP address for whichtrace information is to be collected.

Subnet mask The subnet mask for which traceinformation is to be collected.

Port Number *ANY The TCP/IP port number defaults to*ANY, which implies traffic associatedwith any port on the remote system (andqualified by the IP address and subnetmask) will be traced.Note: If the user wants to specify the port,the subnet mask must also be specified.



Figure 108. Trace TCP/IP Application (TRCTCPAPP): SMTPCLT

Table 15. Parameters for TRCTCPAPP APP(*SMTPCLT)


TCP/IP application *SMTPSVR Used for inbound mail.

*SMTPCLT Used for outbound mail.

Trace option setting *ON The collection of trace information isstarted.

*OFF The collection of trace information isstopped. The trace information is written tothe spooled printer file.

*END Tracing is ended and all trace information ispurged. No trace information output iscreated.

*CHK The status of tracing for the specifiedapplication is checked. Messages arereturned, indicating whether tracing isactive for the specified TCP/IP application,the command parameters specified formatthe last time that TRCTCPAPP was startedfor this application, and other informationrelated to the collection of traceinformation.

Maximum storagefor trace

*APP For *SMTPSVR or *SMTPCLT - 4096 KBper job.

maximum KB Specify the maximum amount of storage, inKB used to store trace records (one KBequals 1024 bytes).

Trace TCP/IP Application (TRCTCPAPP)


TCP/IP application . . . . . . . > *SMTPCLT *FTP, *SMTPSVR, *SMTPCLT...Trace option setting . . . . . . *ON *ON, *OFF, *END, *CHKMaximum storage for trace . . . *APP_______ 1-16000, *APPTrace full action . . . . . . . *WRAP___ *WRAP, *STOPTRCRecipient mail address . . . . . ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________Recipient host name . . . . . . ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________Domain name service . . . . . . *NO_ *NO, *YES



Trace full action *WRAP When the trace buffer is full, the tracewraps to the beginning. The oldest tracerecords are written over by new ones asthey are collected.

*STOPTRC Tracing stops when the trace buffer is full oftrace records.

User Profile The parameter is not used with *SMTPSVRor *SMTPCLT.

Recipient mailaddress

recipient-mail-address Only trace information associated with aspecific recipient mail address will becollected. This parameter is only validwhen APP(*SMTPSVR) orAPP(*SMTPCLT) is specified.Note: The recipient mail address (up to 255characters) must have the following format:[email protected]

Recipient hostname

recipient-host-name Only trace information associated with aspecific host name will be collected. Therecipient host name (up to 255 characters)must have the following format:abc.def.comNote: This parameter is only valid whenAPP(*SMTPCLT) is specified.

Remote NetworkAddress

Address Family *INET The filter for AF_INET address family.

IP Address The remote TCP/IP address for which traceinformation is to be collected.

Subnet mask The subnet mask for which traceinformation is to be collected.

Port Number *ANY The TCP/IP port number defaults to *ANYwhich implies traffic associated with anyport on the remote system (and qualified bythe IP address and subnet mask) will betraced.Note: If the user wants to specify the port,the subnet mask must also be specified.

Domain NameService

*NO No filtering of trace information is done forDNS resolution.

*YES Trace information includes only tracepoints associated with DNS resolution andspecifies whether only trace informationassociated with domain name service(DNS) resolution will be captured. Thisparameter is only valid whenAPP(*SMTPCLT) is specified.



11.3.8 AnyMail/MSF dump snap-inThis section applies to Domino Release 4.6 or 5.0 with SMTP services *MSF.

This snap-in creates an AnyMail/MSF dump as a spooled file under user profileQMSF (WRKSPLF QMSF). One spooled file is generated each time MSF makesa pass through the piece of mail and calls the dump snap-in (for example, onespooled file for the MIME envelope and one spooled file for the attachment).Follow these steps:

1. To activate the snap-in, type the following command on the OS/400 commandline:

CALL QCMD


ADDEXITPGM EXITPNT(QIBM_QZMFMSF_ACT) FORMAT(MSFF0100) PGMNBR(200)PGM(QSYS/QZMFDUMP) TEXT('MSF Snap-in')PGMDTA(*JOB 12 SPCL01009999)

You need to press F10 if you are prompting to see all the parameters.


ENDMSF

An example of how to collect SMTP data is shown here:

1. To start the SMTP Flight Recorders for Outbound Mail, on the OS/400command line, type:

TRCTCPAPP APP(*SMTPCLT) TRCFULL(*STOPTRC)

2. To start the SMTP Flight Recorders for Inbound Mail, on the OS/400command line, type:

TRCTCPAPP APP(*SMTPSVR) TRCFULL(*STOPTRC)

3. Run the test scenario causing the issue.

4. To stop the SMTP Flight Recorders for Outbound Mail, on the OS/400command line, type:

TRCTCPAPP APP(*SMTPCLT) SET(*OFF) TITLE('Outbound E-mail')

5. To stop the SMTP Flight Recorders for Inbound Mail, on the OS/400command line, type:

TRCTCPAPP APP(*SMTPSVR) SET(*OFF) TITLE('In-Bound E-Mail')


WRKSPLF

One or more spooled files are created with the file name QTMSSTRC.

Example

The CALL QCMD command starts the command line screen. To exit thecommand line screen, press the F12 function key. If you need more lines,press F11.

Note


4. On the OS/400 command line, type

STRMSF *RESET


6. Remove the MSF snap-in by typing the following command on the OS/400command line:

ENDMSF

7. Type the following command statement:

WRKREGINF QIBM_QZMFMSF*

8. Select option 8 for QIBM_QZMFMSF_ACT.

9. Remove the snap-in by selecting option 4 (Remove) for Exit Program Number200, which is QZMFDUMP QSYS.

10.Type:

STRMSF *RESET

11.To view the MSF Dump spooled files, enter the following OS/400 command:

WRKSPLF SELECT(QMSF)

Look for the spooled file QPSRVDMP.

11.3.9 Dumping Mail Server Framework (MSF)This section applies to Domino Release 4.6 or 5.0 with SMTP services *MSF.

This tool is for Level 3 support technicians only and is only mentioned here forcompleteness. Perform the following process:


2. Wait a sufficient amount of time for the test to complete.


DMPSYSOBJ OBJ(QZMFMSF*) CONTEXT(QUSRSYS)

This command generates the QPSRVDMP spooled files.

4. To view the MSF dump, type the following command on the OS/400 commandline:

WRKSPLF SELECT(QMSF)

Look for spooled file QPSRVDMP.

11.3.10 Communications trace of a lineTo run a communications trace, perform the following steps:


STRSST

2. Select option 1 (Start a service tool).

3. Select option 3 (Work with communications trace).

4. Press F6 (Start a trace). For the parameters, see the display shown in Figure109.


Figure 109. Starting the communications trace

5. On the next screen that appears, for OS/400 V4R2 or later, select option 1 (Alldata (no filtering)).

6. To return to the OS/400 command line, press F12 two times. Then, pressEnter on the Exit System Service Tools screen.


8. To end the communications trace, complete the following tasks:

a. On an OS/400 command line, type:

STRSST

b. Select option 1 (Start a service tool).

c. Select option 3 (Work with communications trace).

d. On the Work with Communications Traces menu, select option 2 (Stoptrace).

9. To format and print the collected data, select option 6 (Format and print trace).

Ensure that the parameters are set as shown in Figure 110 on page 220.

Start Trace


Configuration object . . . . . . . tokenline

Type . . . . . . . . . . . . . . 1 1=Line, 2=Network interface3=Network server

Trace description . . . . . . . . Comm Trace 1

Buffer size . . . . . . . . . . . 6 1=128K, 2=256K, 3=2M, 4=4M5=6M, 6=8M, 7=16M, 8=32M9=64M

Stop on buffer full . . . . . . . y Y=Yes, N=No

Data direction . . . . . . . . . 3 1=Sent, 2=Received, 3=Both

Number of bytes to trace:Beginning bytes . . . . . . . . *MAX Value, *CALC, *MAXEnding bytes . . . . . . . . . *CALC Value, *CALC


Figure 110. Format Trace Data

The next screen you see (Figure 111) is useful for reducing the amount of datathat you will get in the spooled file. You may want to look at data associated withparticular IP addresses or particular ports. For mail troubleshooting, theinteresting ports are 53 (DNS) and 25 (SMTP).

Figure 111. Select IP addresses to format

10.To view the file QPCSMPRT, on the OS/400 command line, type:

WKRSPLF

Format Trace Data

Configuration object . . . . : TRNLINEType . . . . . . . . . . . . : LINE


Controller . . . . . . . . . . *ALL *ALL, name

Data representation . . . . . 1 1=ASCII, 2=EBCDIC, 3=*CALC

Format RR, RNR commands . . . N Y=Yes, N=NoFormat Broadcast data . . . . N Y=Yes, N=NoFormat MAC or SMT data only . N Y=Yes, N=NoFormat UI data only . . . . . N Y=Yes, N=NoFormat SNA data only . . . . . N Y=Yes, N=NoFormat TCP/IP data only . . . y Y=Yes, N=NoFormat IPX data only . . . . N Y=Yes, N=No


Select IP addresses to format

Configuration object . . . . : TRNLINEType . . . . . . . . . . . . : LINE


IP address . . . . . . . . . . *ALL *ALL, address

IP address . . . . . . . . . . *ALL *ALL, address

Port . . . . . . . . . . . . . 25 *ALL, 1-65535



11.3.11 Component journalingAS/400 Mail component journaling can help you understand how mail flowsthrough the major components, and lists the reasons why your mail may not bedelivered.

Journaling takes CPU cycles. Therefore, the best performance is realized whenjournaling is turned off. Journaling also takes up space on your AS/400 harddrives. However, if a mail problem occurs, journaling can offer insight into whathappened to a particular piece of mail.

Component journaling is available for V4R4, V4R3, and V4R2.

• V4R4: No PTF is required

• V4R3: Requirements include:

– PTF: SF55407 (5769TC1)– APAR: SA78237

Note: SF55407 supersedes SF52765

• V4R2: Requirements include:

– PTF: SF55403 (5769TC1)– APAR: SA78237

Note: SF55403 supersedes SF52765

For more information about component journaling, refer to the following Web site(this URL is case sensitive):http://www.as400.ibm.com/tstudio/tech_ref/tcp/Indexfr.htm

You can use AS/400 Help Text for the Select IP Addresses to format display.

If *ALL is specified for both the IP address fields and the port field, all TCP/IPframes will be formatted.

If *ALL is specified for both IP address fields and the port field is specified, onlyframes that have a source or destination port that matches the port specifiedwill be formatted.

If specific IP addresses are specified for both the IP address fields, only framesthat have source and destination, or destination and source IP addressesmatching those specified will be formatted. This is useful for formatting all theframes passing between the two systems.

If a specific IP address and *ALL are specified in the IP address fields, onlyframes that have a source or destination IP address matching the IP addressspecified will be formatted. If the port field is also specified, only frames thathave a source or destination IP address matching the IP address specified anda source or destination port number matching the port specified will beformatted.

Note


11.4 Tools for e-mail monitoring with Domino

Domino provides three tools that you can use to monitor mail. Message trackingallows you to track specific mail messages to determine if the intended recipientsreceived them. Mail usage reports provide the information you need to resolvemail problems and improve the efficiency of your mail network. Mail probes testand gather statistics on mail routes.

11.4.1 Tracking mail messagesBoth Domino administrators and end users can track mail. Administrators cantrack mail sent by any user, while end users can track only messages that theythemselves send.

When you configure mail tracking, you can specify which types of informationDomino records. For example, you can specify that Domino won’t recordmessage tracking information for certain users, or you can choose not to recordthe subject line of messages sent by specific users.

Domino records all message tracking information in the Mail Tracking Storedatabase (MTSTORE.NSF). When an administrator or user searches for aparticular message, Domino searches the Mail Tracking Store database, which iscreated automatically when you start the Mail Tracking program on the server.

As shown in Figure 112, several documents about Tools for mail monitoring areavailable to help you implement this tool in the Domino R5 Administration Helpdatabase.

Figure 112. Tools for mail monitoring help documents


11.4.2 Mail probesYou can monitor your mail network by configuring probes to test and gatherstatistics on mail routes.

11.4.2.1 Creating a mail probeUsing a mail probe, you can test and gather statistics on mail routes. To test amail route, use the ISpy task. ISpy sends a mail-trace message to the mail serverof the individual that you specify. The probe generates a statistic that indicatesthe amount of time, in seconds, that it took to deliver the message. If the probefails, the statistic has the value -1. The format of a mail probe statistic is:QOS.Mail.RecipientName.ResponseTime.

If the Collect task is running, the Statistics database (STATREP.NSF) stores themail probe statistics. In addition, ISpy generates events for probes that fail. Youcan set up an Event Notification document to notify you when an event hasoccurred. By default, ISpy monitors the local mail server. To monitor otherDomino mail servers, you must create probe documents.

11.4.2.2 Creating a mail probeComplete these steps to create a mail probe:

1. Make sure that you start the ISpy task on the server.

2. From the Domino Administrator, click the Configuration tab.

3. Click Statistics & Events->Probes->Mail.

4. Click New Mail Probe.

5. Click the Basics tab, and complete the fields shown in Table 16.

Note: Do not select All Domino servers in the domain will probe themselves.

Table 16. Fields to complete on the Basics tab for a mail probe

Field Enter

Probing servers (source) Enter the server you want the probe to start from or select theserver from the drop-down box.

Target mail address(destination)

Enter the mail recipient for which you want to check the mailroute or use the drop-down box to select a recipient from aDomino Directory or Address Book. Do not enter more thanone individual, and do not enter a group name.

ISpy is case-sensitive. Therefore, you must enter it exactly as shown below.

Start the ISpy task manually. At the console, enter the command:

load runjava ISpy

Stop the ISpy task. Enter one of the following commands:

• Tell runjava quit

• Tell runjava unload ISpy

Starting and stopping ISpy


6. Click the Probe tab, and complete the fields shown in Table 17.

Table 17. Fields to complete on the Probe tab for a mail probe

7. Click the Other tab. Complete the fields as shown in Table 18, and then clickOK.

Table 18. Fields to complete on the Other tab for a mail probe

11.5 Copying spooled files to physical filesSpooled files cannot be transferred using FTP. Only physical files can betransferred using FTP. Place the spooled files for MSF Dump, AnyMail/MSFSnap-In, and communications trace into physical file members within a physicalfile. The following examples illustrate this.

11.5.1 Example 1: MSF dump and communications trace spooled filesThe files that need to be copied in physical file members are:

• QPSRVDMP (MSF DUMP spooled file)• QPCSMPRT (communications trace spooled file)

You need to perform the following steps:

1. To get MSF dump and communications trace spooled file attributes, completethese tasks:

a. On the OS/400 command line, type:

Do a WRKSPLF

The first screen appears as shown in Figure 113.

Field Enter

Send interval Enter the probe interval. This is the frequency at whichprobes will be sent.

Time out threshold Enter the time out threshold. This is the period the probingserver (source) will wait for a response before logging afailure.

Field Enter

Event Select the severity of the event you want to be generated ifthe probe fails.

Create a new notificationprofile for this event.

You can set up notification for a custom event. If you click thisbutton, you will be guided through the process by the EventNotification Wizard.

Enablement Select the “Disable the probe” field if you want to disable thisprobe. You can re-enable it at any time.


Figure 113. Work with Spooled File Attributes (MSF Dump)

b. Enter option 8 next to QPSRVDMP. The display shown in Figure 114appears.

Figure 114. Work with Spooled File Attributes (Communications Trace)

c. Enter option 8 next to QPCSMPRT. The screen shown in Figure 115 onpage 226 appears.



Device or Total CurOpt File User Queue User Data Sts Pages Page Copy8 QPSRVDMP MYUSERPRF QEZDEBUG RDY 81 18 QPCSMPRT MYUSRPRF DFTPRT RDY 9 1


Work with Spooled File Attributes

Job . . . . . . . . : QPADEV0053 File . . . . . . . . : QPSRVDMPUser . . . . . . . : MYUSRPRF Number . . . . . . : 2Number . . . . . . : 520693

Status . . . . . . . . . . . . . . . . : READYOutput queue . . . . . . . . . . . . . : QPRINTLibrary . . . . . . . . . . . . . . : QGPL

ASP file resides on . . . . . . . . . : 1Form type . . . . . . . . . . . . . . : *STDOutput priority . . . . . . . . . . . : 5Copies left to produce . . . . . . . . : 1Total copies . . . . . . . . . . . . . : 1Maximum records . . . . . . . . . . . : *NOMAXNumber of separators . . . . . . . . . : 0File becomes available . . . . . . . . : *FILEENDHold file before written . . . . . . . : *NOSave file after written . . . . . . . : *NO


F3=Exit F5=Refresh F12=Cancel F13=Change


Figure 115. Work with Spooled File Attributes (Communications Trace)

d. Write down the spooled file attributes for Job, File, User, and Number forthe spooled files QPSRVDMP and QPCSMPRT.

2. Create the Physical File P12345B678 by using the following command:

CRTPF FILE(QUSRSYS/P12345B678) RCDLEN(133) MAXMBRS(*NOMAX) SIZE(*NOMAX)

Here, 12345 is the PMR number, and 678 is the branch. Substitute your PMRnumber and branch for these numbers.

3. Copy the spooled file QPSRVDMP to the physical file member MSFDMPOBJ.To do so, on the OS/400 command line, type:

CPYSPLF FILE(QPSRVDMP) TOFILE(QUSRSYS/P12345b678) +JOB(520693/MYUSRPRF/QPADEV0053) SPLNBR(2) TOMBR(MSFDMPOBJ) MBROPT(*ADD)

Note: The spooled file attributes for QPSRVDMP are placed in the parametersabove.

4. To copy the spooled file QPCSMPRT to the physical file memberCOMMTRACE, on the OS/400 command line, type:

CPYSPLF FILE(QPCSMPRT) TOFILE(QUSRSYS/P12345b678) +JOB(520693/MYUSRPRF/QPADEV0053) SPLNBR(1) TOMBR(COMMTRACE) MBROPT(*ADD)

11.5.2 Example 2: AnyMail/MSF Dump Snap-In spooled filesOne or more spooled files are generated for each piece of e-mail processed byMSF.

QPSRVDMP is the AnyMail/MSF Dump Snap-In spooled file. Perform thefollowing steps:

1. To obtain the spooled file attributes for the Any-Mail/MSF Dump Snap-In,complete these tasks:


Job . . . . . . . . : QPADEV0053 File . . . . . . . . : QPCSMPRTUser . . . . . . . : MYUSRPRF Number . . . . . . : 1Number . . . . . . : 520693






a. On the OS/400 command line, type the command:

Do a WRKSPLF SELECT(QMSF)


Figure 116. Work with All Spooled Files

b. Enter option 8 next to QPSRVDMP. Then, enter option 8 next to any of theother required QPSRVDMP spooled files that must be sent by FTP to IBM.You will see the attributes for each file as shown in Figure 117 and Figure118 on page 228.

Figure 117. Work with Spooled File Attributes (AnyMail snap-in): File 1



Device or Total CurOpt File User Queue User Data Sts Pages Page Copy8 QPSRVDMP QMSF QEZDEBUG RDY 2 18 QPSRVDMP QMSF QEZDEBUG RDY 2 1



Job . . . . . . . . : QMSF File . . . . . . . . : QPSRVDMPUser . . . . . . . : QMSF Number . . . . . . : 1Number . . . . . . : 520696






Figure 118. Work with Spooled File Attributes (AnyMail snap-in): File 2

c. Document the spooled file attributes for Job, File, User, and Number forQPSRVDMP spooled files.

2. Place each AnyMail/MSF Dump Snap-in spooled file that is to be sent to IBMinto its own physical file member. In this example, two spooled files werecreated by the Snap-in. Both of the spooled files are copied to Physical FileMember Snapin1 and Physical File Member Snapin2 within Physical FileP12345B678.

Note: In this example, the PMR is 12345 and the Branch is 678. Substituteyour PMR number and branch.


CPYSPLF FILE(QPSRVDMP) TOFILE(QUSRSYS/P12345b678) JOB(520696/QMSF/QMSF) +SPLNBR(1) TOMBR(SNAPIN1) MBROPT(*ADD)


CPYSPLF FILE(QPSRVDMP) TOFILE(QUSRSYS/P12345b678) JOB(520696/QMSF/QMSF) +SPLNBR(2) TOMBR(SNAPIN2) MBROPT(*ADD)


Job . . . . . . . . : QMSF File . . . . . . . . : QPSRVDMPUser . . . . . . . : QMSF Number . . . . . . : 2Number . . . . . . : 520696






Chapter 12. Database integration problems

When you install Domino and configure a Domino server, you have access to anumber of standard APIs or tools for integration with enterprise applications, alsocalled line of business applications. These APIs or tools are part of the LotusDomino for AS/400 product, except Lotus Enterprise Integrator that is distributedas a separate product.

This chapter focuses on what is happening on the server, which is the access toDB2 UDB for AS/400 through the DB2 Call Level Interface.

12.1 DB2 Call Level InterfaceAll data access or program calls that happens from the Domino for AS/400 serverto the DB2 UDB for AS/400 relational database go through the DB2 Call LevelInterface.

12.1.1 DB2 UDB for AS/400The major characteristic of the AS/400 database manager, named DB2 UDB forAS/400, is its being part of the operating system. In practice, this means that thelarge majority of your AS/400 data is stored in the relational database. Althoughthe AS/400 system also implements other file systems in its design, the relationaldatabase on the AS/400 system is the most commonly used by the customers. Itstores your relational data as well as typical non-relational information such asthe source of your application programs or binary files.

12.1.1.1 Physical files and tablesData on the AS/400 system is stored in objects called physical files. Physical filesconsist of a set of records with a predefined layout. Defining the record layoutmeans that you define the data structure of the physical file in terms of the lengthand the type of data fields that participate in that particular layout.

These definitions can be made through the native data definition language of DB2UDB for AS/400, called Data Description Specifications (DDS). If you are familiarwith other relational database platforms, you are aware that the most commonway to define the structure of a relational database is by using the data definitionstatements provided by the Structured Query Language. This is also possible onthe AS/400 system. The SQL terminology can be mapped to the native DB2 UDBfor AS/400 terminology for relational objects. An SQL table is equivalent to a DDSdefined physical file. We use both terms interchangeably in this redbook.Similarly, table rows equate to physical file records for DB2 UDB for AS/400 andSQL columns are a synonym of record fields.

12.1.1.2 Logical files, SQL views, and SQL indexesBy using DDS, you can define logical files on your physical files or tables. Logicalfiles provide a different view of the physical data, allowing column subsetting,record selection, the joining of multiple database files, and so on. They can alsoprovide the physical files with an access path when you define a keyed logicalfile. Access paths can be used by application programs to access records directlyby key or for ensuring uniqueness.


On the SQL side, there are similar concepts. An SQL view is almost equivalent toa native logical file. The selection criteria that you can apply in an SQL view ismuch more sophisticated than in a native logical file. An SQL index provides akeyed access path for the physical data exactly the same way as a keyed logicalfile does. Still, SQL views and indexes are treated differently from native logicalfiles by DB2 UDB for AS/400, and they cannot be considered to exactly coincide.

By database file, we mean any DB2 UDB for AS/400 file such as a logical orphysical file, an SQL table, or view. Any database file can be used by applicationsfor accessing DB2 UDB for AS/400 data.

12.1.2 Call Level Interface (CLI)The DB2 Call Level Interface (CLI) is the IBM callable SQL interface to the DB2family of database servers. It is an application programming interface forrelational database access and uses function calls to pass dynamic SQLstatements as function arguments. DB2 CLI is based on Microsoft ODBCspecification and the International Standard for SQL/CLI. Using this interface, anapplication developer can code, compile, and ship an application withouttargeting a specific DBMS.

The DB2 Call Level Interface is an alternative to an embedded dynamic SQL. Onthe AS/400 system, this interface is available to any of the ILE languages.

For the most part, the SQL CLI is syntactically and semantically equivalent toODBC. This is not to say that everything in the latest level of ODBC is supported,but an application that uses the most common routines will likely run against theCLI with few changes.

The SQL CLI is a standard created by the X/Open standards group, and theAS/400 DB2 CLI was built in V3R6 according to that standard. ODBC is theMicrosoft implementation of the X/Open CLI standard, and has veered off fromthe X/Open standard in a few areas. Even though the CLI has been updated inseveral releases since V3R6, it is never likely to match ODBC exactly, since thatis technically not the standard that the AS/400 DB2 CLI is built with which tocomply.

More information about the DB2 Call Level Interface is available in the “Call LevelInterface Programming” chapter of the redbook DB2/400 Advanced DatabaseFunctions, SG24-4249.

12.2 Tools for database integration

APIs or tools for integration with enterprise applications available in Domino forAS/400 are:

• @DB functions• LotusScript Data Object (LS:DO)• Lotus Domino Connector APIs (LSX LC, LC Java classes)• Domino Enterprise Connections Services (DECS)

Using JDBC or Enterprise JavaBeans (available with IBM WebSphere AdvancedEdition V3) can be another solution to access DB2 UDB for AS/400 data or storedprocedures from Domino for AS/400 applications.


Most of these options are already described in more detail in Lotus Domino forAS/400: Integration with Enterprise Applications, SG24-5345.

12.2.1 @DB functionsUsed within Domino formulas, @DB functions offer the simplest programmaticoption for including DB2 UDB for AS/400 data in a Domino application. Thesefunctions enable access to relational data with some limitations. Availablefunctions are @DBColumn, @DBLookup, and @DBCommand.

12.2.2 LotusScript Data Object (LS:DO)LS:DO provides powerful and flexible programmatic access to a relationaldatabase, such as DB2 UDB for AS/400, from a Domino application. LS:DO canbe used wherever LotusScript is available: Forms, Navigators, Agents, and Views(Actions only).

12.2.3 Lotus Domino Connector APIsA LotusScript Extension, called LotusScript Extension for Lotus DominoConnectors (LSX LC), allows programmatic access to data external to Dominoapplications. It works with the Lotus Domino Connectors, a new technologyannounced by Lotus in 1998, which extends the capabilities of Dominoapplications to access external data while making the coding more independentof the type of data source.

The equivalent Java API (LC Java classes) is also available.

12.2.4 Domino Enterprise Connections Services (DECS)Domino Enterprise Connection Services (DECS), a component of the Domino Serverstarting with Release 4.6.3, allows you to create Domino applications that providelive, real-time access to enterprise data and applications.

Using DECS administration, identify the fields in a Domino application form thatmap to corresponding entities in an external database. Document events thatoccur with this form, such as open or update, automatically query or updateexternal connector data.

12.2.5 Lotus Enterprise Integrator (LEI)Lotus Enterprise Integrator is a separately acquired Lotus product for the Dominoserver, available at an additional charge, which is used for exchanging databetween data sources either on schedule or on demand. Data sources can beDB2 UDB for AS/400 tables, Lotus Notes databases, flat files stored in theAS/400 Integrated File System, or Zmerge defined text files.

The activities that Lotus Enterprise Integrator/Lotus NotesPump can performinclude data transfer, replication, and the execution of LotusScript and Java. It isalso capable of real-time data access. Activities may be scheduled, on-demand,or triggered by an event.

Chapter 12. Database integration problems 231

12.3 Options for database integration

This section presents an overview of the options that are available for databaseintegration. It also covers stored procedures calls, since they are part of databaseintegration.

The options are:

• Access to DB2 UDB for AS/400 data from a Domino application• Data replication between the AS/400 system and Domino• Access to an AS/400 application from a Domino application (stored procedure

call)

12.3.1 Access to DB2 UDB for AS/400 data from a Domino applicationFigure 119 shows the options available to allow a Domino application to accessAS/400 DB2 UDB for AS/400.

Figure 119. Interactive access to DB2 UDB for AS/400 data from a Domino application

Inside the Domino server, the following options are available:

• @DB functions can be used inside a formula language agent or in a computedforms field.

• LS:DO can be used inside a LotusScript agent or in a computed forms field.

• LSX LC can be used inside a LotusScript agent or in a computed forms field.

• JDBC can be used inside a Java agent or Java servlet.

• LC Java classes can be used inside a Java agent or Java servlet.

• DECS Real-time activity allows access to DB2 UDB for AS/400 data for reador update purposes while reading or updating a Domino document, withcollapse and expand capability since Domino for AS/400 Release 4.6.5.


• Lotus Enterprise Integrator real-time activity allows access to DB2 UDB forAS/400 data for read or update purposes while reading or updating a Dominodocument, with Collapse/Expand capability.

Inside the Notes client, the following options are available:

• @DB functions can be used inside a form, a navigator, a view (Actions only),or a Formula agent.

• LS:DO can be used inside a form, a navigator, a view (Actions only), or aLotusScript agent.

• LSX LC can be used inside a form, a navigator, a view (Actions only), or aLotusScript agent.

• JDBC can be used inside a Java agent or in an applet embedded in a form ora document.

• LC Java classes can be used inside a Java agent or in an applet embedded ina form or a document.

• DB2 Import allows you to import the result of an SQL Query inside a rich textfield of a Domino document.

For a Web client, the following options are available:

• @DB functions, LS:DO, LSX LC, DECS Real-time and Lotus EnterpriseIntegrator/Lotus NotesPump all run on the Domino server.

• JDBC can be used inside an applet or inside a servlet on the Domino servertriggered by an applet.

• LC Java classes can be used inside a Java agent or in an applet embedded ina form or a document.

In addition, @DB Functions, LS:DO, LSX LC, JDBC, LC Java classes, LEI/NotesPump, and DECS can all access remote data sources (DB2) through the use ofDRDA.

12.3.2 Data replication between the AS/400 system and DominoFigure 120 on page 234 displays the options available to allow a data propagationbetween AS/400 data and Domino applications.


Figure 120. Data propagation between AS/400 system data and Domino applications

Data replication, as presented in Figure 120, shows the transfer of data betweenAS/400 data (stored in DB2 UDB for AS/400 tables or files stored in theintegrated file system and Domino databases (.nsf)).

Domino for AS/400 introduces new enablement for connectors beyond theexisting base connectors (such as the DB2 connector or file connectors). ERPconnectors and Transaction Systems connectors extend the scope of enterprisedata integration.

In most cases, only modifications are replicated between the two sources of data.It allows users to benefit from the same freshness of data as soon as thereplication has taken place. The replication can be planned on a short schedule,for example, every 60 minutes or every day, week, or month.

Replication happens at the server level. The transfer activities can be scheduled,on-demand, or triggered by an event.

Domino replication further extends the replication capabilities proposed by LotusNotesPump and its newest version, Lotus Enterprise Integrator.

Replication is one of the most powerful aspects of the Domino environment.When replication of a Domino database takes place between two Dominoservers, or between a Notes client and a Domino server, only changes at a fieldlevel are copied from one Domino database to another. This means that, if onlyone field is modified in a document, only the value in that field is replicated.

As shown in Figure 121, a Domino database on the Domino for AS/400 servercan be replicated on other Domino servers (on AS/400 servers or other serverplatforms, such as the PC servers).


Figure 121. Data propagation and Domino replication

A Domino database can also be replicated on a regular basis by the user on theworkstation. This is especially true for mobile workers who need the most recentdata, but work disconnected.

For example, a Domino database can be dedicated to product descriptions. Itwould contain one document per product, with unstructured data, such as textdescribing the product characteristics, an image displayed, or an attacheddocument stored in a rich field text. This database will be of greater benefit to theuser if information, such as price and available stock quantity, are refreshedevery day from the DB2 UDB for AS/400 data source.

Replication happens here in two steps: first through Lotus Enterprise Integrator,and then from the Domino server to the user’s laptop through the Lotus Notesclient replication capability.

The following conditions are for mobile users:

• They must connect to the Domino server on a regular basis, for exampleonce a day.

• A network architecture study may be necessary if replication volumes areexpected to be significantly large. Connection time may prove too long orcosts too high if the users connect to the Domino server or their enterprisenetwork through a telephone line.

Note


12.3.3 Access to an AS/400 application from a Domino applicationFigure 122 shows the options that are available for a Domino application to callan AS/400 application.

Figure 122. Access to the AS/400 application from a Domino application

The first option is to call an AS/400 program through a procedure call. Input andoutput parameters can be passed to the program. This can be done from LotusScript and Java.

The following options are available if you are using Java:

• JDBC, which includes a Procedure Call interface

• AS/400 ToolBox for Java, which allows you to issue commands and callprograms on the AS/400 system and return the results

Another option is to let the Domino application communicate with an AS/400application using the following methods:

• An AS/400 data queue: The AS/400 ToolBox for Java can then be used.

• Messages exchanged between the Domino application and the AS/400application.

12.4 Database integration and OS/400 DB2 CLI

Domino for AS/400 server-side database integration consists of these abovecomponents:

• @DB functions• LotusScript Data Object (LS:DO)• Lotus Domino Connector APIs (LSX LC, LC Java classes)• Domino Enterprise Connections Services (DECS)• Lotus Enterprise Integrator


All those components are designed to use ODBC. However, in the Domino forAS/400 implementation, they use DB2 SQL Call Level Interface (CLI).

12.4.1 V4R2 changes in CLIEach of the Lotus database integration components were initially designed to beused with ODBC. The Domino for AS/400 port converted the ODBC calls to CLI.Because Domino is multi-threaded and R420 SQL is not thread safe, CLI wasmodified in R420.

CLI now supports an SQL_ATTR_SERVER_MODE environment attribute. Whenthis attribute is set (SQLSetEnvAttr), a SQLConnect request “connect” the threadissuing the request to a QSQSRVR prestart job that runs in the QSYSWRKsubsystem.

The SQL calls are made in the context of the QSQSRVR prestart job, not theDomino thread. To enforce security, the “server mode” connection requires a validAS/400 user profile and password to be passed on the connection. Similar toother OS/400 server jobs, the QSQSRVR job does a user profile switch fromQUSER to the user profile passed on the connection and runs with this authority.

12.4.2 Determining if a problem is with CLI or DominoNeither Domino or CLI have a trace as detailed as an ODBC spy. If you are ableto run the Domino application from a PC (over Client Access ODBC), then anODBC spy can be used. This can help in determining if the Domino applicationand AS/400 database are working correctly.

AS/400 job logs and Domino logs can be used to screen for obvious problems oneither side (error messages that are APAR conditions or an invalid SQLstatement). If the problem is isolated to either the Domino port code (ODBC toCLI conversion code) or CLI code, more detailed diagnostics such as the LS:DOtrace and the AS/400 Database monitor may be needed. Finally, if the problemstill cannot be isolated, recreating material may be needed.

12.4.3 Locating the job and the job logServer database requests can be made from three different Domino jobs:

• AMGR: Any agents that execute on the server will run as a thread under oneof the AMGR jobs in the appropriate Domino subsystem. This job log recordsany errors initializing CLI or connecting to the QSQSRVR job (such asSQL0901 errors). It also records the fully qualified job name of the QSQSRVRjob that was used.

• HTTP requests: Any Database requests made from forms or agents that areexecuted from an HTTP request run on the server as a thread under one ofthe HTTP jobs in the appropriate Domino subsystem. This job log records anyerrors initializing CLI or connecting to the QSQSRVR job. It records the fullyqualified job name of the QSQSRVR job that is used.

• QSQSRVR job: First, verify that at least one QSQSRVR job is available toattach to, and that no error appears in the job log. The QSQSRVR prestart jobdoes not need to be running. CLI starts the job if needed (PTF required).However, certain errors (such as the QUSER user profile being disabled) mayprevent the prestart job from starting correctly. To verify that the prestart job isactive and in the proper state, use the following command:



Press F14 (Include key). You should see at least one QSQSRVR job in thePSRW state, as shown in Figure 123 (you can also use the WRKSBSJOB QSYSWRK

command).

Figure 123. Displaying QSQSRVR jobs

Check the job log before any application attaches to the job and verify that noerrors have occurred.

Any SQL errors that occur after the application issues a SQLConnect willappear in this job log. The DBMON and STRDBG command should be run onthis job, not the requesting job.

Note: The job log is erased after the thread issues a CLI disconnect. Youcannot force the job log to print out with the debug information. This serviceaid is being considered for V4R3 CLI.

To force a thread to connect to a specific prestart job (one that may have DBmonitor or debug running), follow these steps:

1. End the prestart jobs by issuing the command:

ENDPJ SBS(QSYSWRK) PGM(QSQSRVR)

2. Change the prestart job so that only one prestart job is active (this allows youto predict which prestart job will be used):

CHGPJE SBSD(QSYSWRK) PGM(QSQSRVR) INLJOBS(1) THRESHOLD(1) ADLJOBS(1)MAXJOBS(2) MAXUSE(1)

Note: Set the MAXUSE parameter to 1 if you want one job log written for eachCLI connect (keeping in mind that debug information is still erased from thejob log).

3. Restart the prestart jobs by entering this command:

ENDPJ SBS(QSYSWRK) PGM(QSQSRVR)




Opt Subsystem/Job User Type CPU % Function StatusQRWTLSTN QUSER BCH .0 SELWQRWTSRVR QUSER PJ .0 PSRWQSNMPSA QTCP BCH .0 PGM-QNMSARTR DEQWQSQSRVR QUSER PJ .0 PSRWQSQSRVR QUSER PJ .0 PSRWQSQSRVR QUSER PJ .0 PSRWQSQSRVR QUSER PJ .0 PSRWQSQSRVR QUSER PJ .0 PSRWQTBOOTP QTCP BCH .0 PGM-QTODBOOT SELW



4. Change the logging level to *SECLVL if you want to force a job log to be written.

5. Start a service job, debug session, or DB monitor, if desired.

6. Recreate the problem.

7. Change the prestart job settings back to their original settings.

12.4.3.1 Tracing DB errorsWhere can you find some information for database access debugging? Table 19describes different tools.

Table 19. DB errors: Where to find information

12.4.3.2 Example of lsxodbc_gdf_level=2 informationThe following example shows the commands that are passed and the informationmessages displayed at the Domino console:

> SET CONFIG lsxodbc_gdf_level=2

03/17/98 11:33:44 AM AMgr: Start executing agent 'LSDO Test Shared' in'DxdLSDOtest.nsf' by Executive '1'QTnewhEnv: SQLALLOCENV RC=0QTnewhEnv: SQLSETENVATTR RC=0_QTEstablishConnection: SQLALLOCCONNECT_QTEstablishConnection: SQLALLOCCONNECT, lpConnection-> hDBC = 2QTSetAutoCommit: SQLSETCONNECTOPTION, SQL_AUTOCOMMIT = SQL_AUTOCOMMIT_ON_QTEstablishConnection: SQLCONNECT_QTEstablishConnection: SQLCONNECT: RC= 0QTQueryPrepSQL: calling SQLPREPARE for <Select * from qiws.qcustcdt><SqlSetStmtOption> succeeded OKQTExecute: SQLEXECDIRECT RC = 0<SqlNumResultCols> succeeded OK

Error or tool Procedure (or comments)

LS:DO errors@DB errors

- Job logs for jobs AMGR, HTTP, or QSQSRVR- LS:DO GetError, GetErrorMessage andGetExtendedErrorMessage methods- LS:DO log setting. This setting will log additional informationto the Notes console. A sample is attached below.

To enable the LS:DO log:1. From the Domino console, enter the following command:SET CONFIG lsxodbc_gdf_level=2 (use a level of 0 to turn offtracing)

2. End the server task that is processing the LS:DO or @Dbrequests:tell AMgr quit (Agent manager)tell HTTP quit (HTTP)

3. Restart the server task:load AMgr (Agent manager)load HTTP (HTTP)

LEI 3.0Notes Pump 2.5a

In the DB2 link document, select the check box Output SQLStatements to Log. This records the SQL statement beingexecuted in the Activity Log.

DB Monitor This is a low-level detailed trace of DB2 UDB for AS/400.

STRDBG The STRDBG command logs useful SQL information in theAS/400 job log. This should be run against the QSQSRVR job.You must view and save the information manually since the joblog is erased after the application issues a disconnect.


QTExecute: SQLNUMRESULTCOLS RC=0, nColumns=11_QTFetchRowData: calling SQLFETCH_QTFetchRowData: after SQLFETCH, retcode = 0_QTFetchRowData: calling SQLGETDATA_QTFetchRowData: after SQLGETDATA, retcode = 0, lLength = 0_QTFetchRowData: calling SQLGETDATA_QTFetchRowData: after SQLGETDATA, retcode = 0, lLength = 8_QTFetchRowData: calling SQLGETDATA_QTFetchRowData: after SQLGETDATA, retcode = 0, lLength = 3_QTFetchRowData: calling SQLGETDATA_QTFetchRowData: after SQLGETDATA, retcode = 0, lLength = 13_QTFetchRowData: calling SQLGETDATA_QTFetchRowData: after SQLGETDATA, retcode = 0, lLength = 6_QTFetchRowData: calling SQLGETDATA_QTFetchRowData: after SQLGETDATA, retcode = 0, lLength = 2_QTFetchRowData: calling SQLGETDATA_QTFetchRowData: after SQLGETDATA, retcode = 0, lLength = 0_QTFetchRowData: calling SQLGETDATA_QTFetchRowData: after SQLGETDATA, retcode = 0, lLength = 0_QTFetchRowData: calling SQLGETDATA_QTFetchRowData: after SQLGETDATA, retcode = 0, lLength = 0_QTFetchRowData: calling SQLGETDATA_QTFetchRowData: after SQLGETDATA, retcode = 0, lLength = 0_QTFetchRowData: calling SQLGETDATA_QTFetchRowData: after SQLGETDATA, retcode = 0, lLength = 0_QTFetchRowData: calling SQLFETCH_QTFetchRowData: after SQLFETCH, retcode = 100QTMakeDisconnection: SQLTRANSACT = SQL_ROLLBACKQTMakeDisconnection: SQLDISCONNECT, lpConnection-> hDBC = 2QTMakeDisconnection: SQLFREECONNECT, lpConnection-> hDBC = 2QTMakeDisconnection: returning QTstsSUCCESS

12.5 Additional information on DB2 CLI problem determinationThe DB2 Call Level Interface (or CLI) is at the heart of any access to thedatabase. If there is a problem here, database or operating system problemdetermination tools should be used.

If the problem is in Domino, you can expect it to be:

• An application coding error• A bug in the function that accesses the DB2 Call Level Interface

Some important information about accessing the DB2 CLI follows in this section.

12.5.1 Journaling required at OS/400 V4R4After upgrading to V4R4, DECS, LEI, NotesPump or LS:DO may fail because theDB2 UDB for AS/400 database file is not journaled. This works on previousversions of the operating system. This is due to a change in the operating system.DB2 UDB for AS/400 now requires all files to be journaled where commitmentcontrol is not set to *NONE. *NONE is a proprietary switch. Lotus adheres to thestandards, which allow autocommit to be set to *YES or *NO. In either case,journaling is required.

Before OS/400 V4R4, the AS/400 system did not support a true autocommit.Most databases (ODBC, OLEDB, JDBC, etc.) treat the isolation levelindependent of autocommit. They allow an isolation level, such as readcommitted, to be used with autocommit on. The commit is automatically done bythe database after an insert, update, or delete operation. Because DB2 UDB forAS/400 did not support this function, most IBM data access methods used anisolation level of *NONE to emulate autocommit. Due to limitations of this method,OS/400 V4R4 is phasing in support for a new autocommit mode. Theselimitations are explained in the next section.


12.5.1.1 Why this change is being madeSeveral problems can result from using *NONE in place of autocommit.

Changes made under commitment control via a trigger program, storedprocedure, or SQL function do not get committed. These changes are lost whenthe connection ends. This is the case where the isolation level is set to somethingother than *NONE when the program is compiled or when the SETTRANSACTION statement is executed.

• “Atomic” stored procedures cannot be called.

• Commits and rollbacks cannot be issued through the Run SQL Scripts featureof Operations Navigator.

• BLOB and CLOB locators require that the cursor be opened using an isolationlevel greater than *NONE. The old behavior requires that the job be run withautocommit off and an isolation level of at least read uncommitted.

12.5.1.2 How ‘autocommit’ mode worksAutocommit mode functions in the ways that are presented here.

ISOLATION *NONE with autocommit onWith the new autocommit mode, SQL now upgrades the isolation level of thecursor from *NONE to *CHG when the file opened contains a BLOB and CLOBfield and locators will be used. Any changes made under commitment control by astored procedure, function, or trigger will be committed upon completion of thestatement that fired the trigger, function or stored procedure.

Only the files being accessed under commitment control (such as the table withthe BLOB or those accessed by the stored procedure) need to be journaled. Thischange resolves the problems listed above with little or no impact to existingapplications or to performance.

ISOLATION *NONE with autocommit offThere is no change in behavior. Changes made under commitment control bytriggers, stored procedures, and SQL functions will not be committed.

ISOLATION not *NONE with autocommit onDB2 UDB for AS/400 will perform a COMMIT HOLD after any operations that alterdata: UPDATE, INSERT, DELETE, ALTER TABLE, stored procedure calls, triggerexecution, and so on.

All files accessed by the application must be journaled. Applications that performa large number of operations requiring commits (such as populating tables withinserts) may see a significant performance impact. These applications shouldconsider using isolation *NONE or handling the commit themselves.

ISOLATION not *NONE with autocommit offNo changes in behavior. Applications control the commit.

12.5.1.3 Autocommit support PTFsBase OS/400 support for the new autocommit mode was supplied by thefollowing V4R4 PTFs: SF58482, SF58629, SF58465, SF58567, SF58560. FutureOS/400 releases will have the code built into the base release. These PTFs arenot optional. The PTFs (or their supersedes) were included in the R440 01December 1999. DB group PTF and may be included on future CUM PTFs.


12.5.1.4 DB2 Call Level Interface (CLI)No additional PTFs are needed. Autocommit mode cannot be disabled. Note thatany technology using CLI may be affected. This includes such products as LotusNotes Pump/Lotus Enterprise Integrator (LEI), the host JDBC classes, SQLJ, andso on. Applications that use the AS/400 specific extension ofSQL_TX_NO_COMMIT will typically default toSQL_TXN_READ_UNCOMMITTED with autocommit turned on. This requiresthat files used by the application be journaled.

12.5.1.5 OS/400 SQLJThe same discussion presented in 12.5.1.4, “DB2 Call Level Interface (CLI)” onpage 242, applies to AS/400 SQLJ.

12.5.1.6 OS/400 Host JDBCOS/400 HostJDBC uses APAR SA84673 and PTF SF60735. It requires the SQLPTFs listed in 12.5.1.4, “DB2 Call Level Interface (CLI)” on page 242.

12.5.1.7 Autocommit supportThe PTF to enable autocommit support for the database server and its clients isnot yet available.

Java Toolbox JDBCThe Java Toolbox sends an isolation level of *NONE when autocommit is set on.If the database server has autocommit support enabled, Java toolbox users willget the new behavior. Support for autocommit on and isolation levels greater than*NONE are being considered for a future release.

Java Toolbox Record Level AccessThis feature does not support autocommit.

Client Access ODBCClient Access ODBC sends an isolation level of *NONE when autocommit is seton. If the database server has autocommit support enabled, ODBC users will getthe new behavior.

Support for autocommit on and isolation levels greater than *NONE are beingconsidered for a future release.

Note that the ODBC specification does not support an isolation level of *NONE.Client Access ODBC defaults to the isolation level specified in the datasource ifthe SQLSetConnectOption SQL _TXN_ISOLATION API is not called. Becausemost third-party applications that use autocommit also callSQL_TXN_ISOLATION (usually with the lowest isolation level of readuncommitted), enabling autocommit on with the specified ODBC isolation levelwould require journaling (and have a significant performance impact) on theseapplications.

Client Access OLEDBIn all versions of Client Access, the OLE DB Record level access dialect (RLA)implements autocommit by sending a commit request after every insert, update,or delete if the isolation level is other than *NONE. There are no plans to supportthe “server side” autocommit mode for OLE DB RLA or for the AS/400 DDMserver that handles the requests.


Note that the OLE DB RLA autocommit implementation does not send a commit ifthe file is opened under isolation level *NONE (connection level isolation *NONEis not supported). The restriction involving trigger programs still applies to any filebeing accessed through OLE DB RLA and a file-level isolation level of *NONE.

The SQL dialect always uses a *NONE isolation level. Support for SQLautocommit mode (and transaction SQL transaction support in general) is beingconsidered for a future release.

Client Access cwbDB APIClient Access cwbDB API has no support for autocommit. An isolation level is notset unless specified by the programmer. Autocommit mode will be used if it isenabled by the database server. Support for autocommit is being considered for afuture release.

Other Clients of the Database server (QZDAINIT)Autocommit mode will be used if it is enabled by the database server.

12.5.2 Tracing CLIThe persistent trace objects (which contain the raw trace data) are namedQUSRSYS/QP0Zxxxxxx, where xxxxxx is the job number of the job that did thetracepoints. The objects are type *USRSPC and can be saved and restored. TheDisplay Trace command creates a formatted version of the data in a file namedQTEMP/QAP0ZDMP.

12.5.2.1 Running User Level Tracing (formerly known as CPA Trace)There are multiple versions of the trace tool, each one requiring different PTFs orOS/400 release levels. The most straightforward method to use is theCHGCPATRC command. Using this command may require that 5769SS1 option15, OS/400 - Common Programming APIs Toolkit, be installed on the system.

The user trace commands are intended to replace the CPA commands. However,they are more difficult to use, particularly when attempting to trace a batch job.

In V4R3 and later, new trace commands are built into the operating system anddo not require the CPA Toolkit.

These commands are:

• DMPUSRTRC: Dump the contents of a specified job's trace

• CHGUSRTRC: Change attributes (size, wrapping, clear) of a specified job'strace

• DLTUSRTRC: Delete the persistent trace object associated with a job's trace

To use this method, you must set the appropriate environment variable for the jobto be traced. You can do this by setting the scope as *SYS (the job must be startedafter the change). Or, set the environment variable for your emulation job and callCHGUSRTRC on the job to be traced to update its environment variable.

For example in V4R3, use:

ADDENVVAR QIBM_USRTRC_LEVEL 'INFO'


In V4R4 (with proper PTFs applied), use:

ADDENVVAR QIBM_COMPONENT_TRACE_LEVEL 'SQRT,ERROR:SQCLI,INFO'

In V4R2 and later, the trace can be dumped (again without needing the CPAtoolkit) using the following command:

CALL PGM(QSYS/QP0WUSRT) PARM('000000')

Here, 000000 is the job number. Press the Enter key.

Access to further information on running the User level Trace (formerly calledCPA Trace) is for IBM internal use only.

12.5.2.2 Tracing CLI callsThe following information was taken from the CLI Frequently Asked Questionsdocument available at the Web site:http://www.as400.ibm.com/db2/clifaq.htm#header_16

Note that the job to trace is the job making the CLI calls rather than theQSQSRVR job.

The CLI has a trace function built into the code, starting with V4R3. To turn on thetrace, use the following command:

CHGCPATRC JOBNUMBER(*CURRENT) SEV(*INFO)

This example turns on *INFO level tracing for the current job. The *INFO level iswhat is needed for the CLI to start writing out trace records. For a long-runningprogram that uses many CLI calls, this trace can fill up quickly.

The CHGCPATRC command also has parameters for specifying the buffer sizefor the trace, if the trace should wrap when full, and if the buffer should beresized. Also, note that trace can be turned on for jobs other than *CURRENT byspecifying the job number. This allows you to run the CLI trace against batchjobs, prestart jobs, and so on.

After turning on the trace and running a CLI program, use the DSPCPATRC commandto see the trace records. You will see every CLI function call and a display of theparameters. This may be helpful in debugging problems or in identifying howmany times a certain function is called.

All the CLI trace records start with SQL CLI: in the text. Because there may betrace records from non-CLI functions (such as threads APIs), you can use thatvalue to quickly locate CLI trace records. Another option is to query thetemporary file that contains the data, and use search criteria to just display theCLI trace records.

The DSPCPATRC output displays the information shown in Figure 124.


Figure 124. DSPCPATRC output

You can filter out everything except the CLI trace records. Run the following SQLstatement using Interactive SQL or any query tool:

>SELECT * FROM FROM QTEMP/QAP0ZDMP WHERE SUBSTR(QAP0ZDMP,17,8) = 'SQL CLI:'

12.5.3 Prestart Job QSQSRVRWhen an agent makes a request to DB2 UDB for AS/400 on the same system,the SQLCLI interface is used. This interface requires a Prestart Job, QSQSRVR,to be running in the QSYSWRK subsystem.

If this job is not running, the agent will hang. IBM ships the subsystem to auto startthis job. If the job is not running, it may be because the subsystem description wasaltered.

Display Physical File MemberFile . . . . . . : QAP0ZDMP Library . . . . : QTEMPMember . . . . . : QP0Z886434 Record . . . . . : 1Control . . . . . Column . . . . . : 1Find . . . . . . .*...+....1....+....2....+....3....+....4....+....5....+....6....+....7....+...

886434:222784 SQL CLI: SQLAllocHandle ( 1 , 0 , SQLINTEGER* )886434:246288 Qp0zSGetEnv(QIBM_USE_DESCRIPTOR_STDIO)886434:246976 Qp0zSGetEnv(QIBM_USE_DESCRIPTOR_STDIO): Could not find string886434:513040 SQL CLI: SQLSetEnvAttr ( 1 , 10004 , 1 , 0 )886434:513352 SQL CLI: SQLAllocHandle ( 2 , 1 , SQLINTEGER* )


Chapter 13. Performance troubleshooting

Many factors can impact the overall performance in the AS/400 Dominoenvironment:

• High resources utilization: CPU, memory, and disk• Object contention• Speed of communication links• Congestion of network resources• Processor speed of the client system• Domino NOTES.INI base parameters

All the above mentioned examples can impact the performance and requiretuning.

When talking about performance, people often have different perceptions of whatperformance is. There are two ways to look at it:

• Response time: Performance is the measure of time spent on an individualoperation or a job.

• Capacity planning or throughput: Performance can be measured in terms oftransactions per hour. It can be measured in terms of data transfer rates orresource utilizations, such as CPU utilization or disk storage utilization.

A partial list of the most common symptoms that may occur, which would indicatethat the system needs to be tuned again, are:

• An excessive number of Domino clients time out or are dropped.

• The number of SERVER jobs in the Domino subsystem keeps growing.

• An excessive number of threads in a SERVER job have a status of MTXW andthat status rarely seems to change for a thread.

• On the WRKSYSSTS screen, the Wait->Inel column is non zero, and theremany more examples.

Many aspects determine the overall performance of a Domino server. However,care should be given to the performance aspects of the different applicationsrunning in that specific server, and to the various elements that will influence theperformance.

Often the main factor in a long response time or missing throughput is found insome of the layers below the application. However, to the user, it is theapplication that doesn’t perform as expected, and the problem is the applicationowners until proven otherwise. It is important to identify the elements withinvarious applications that can affect the performance of a Domino server. Somespecific functions are I/O and/or CPU intensive. For example, there is a hugedifference between a regular mail Domino database, and the same Dominodatabase being indexed by the user.

You can have a sales tracking database, and you can have the same end userlooking-like database, but the latter one containing sophisticated views, formulas,agents and such features that require much more CPU power.

Figure 125 on page 248 shows how certain Domino applications can need a lot ofCPU, where other applications would need mainly I/O capacity.


Figure 125. Domino application characteristics

For information on how to optimize your Domino applications performance, referto the Lotus white paper “Maximizing Application and Server Performance inDomino (PDF file)”. This white paper is available at:http://www.as400.ibm.com/developer/domino/perform/maxperform.pdf

Domino provides many ways to build dynamic Web sites and applications.Navigators, layout regions, programmable fields, and hide-when formulas givedevelopers a wealth of options for building complex Web pages that aredynamically generated.

However, as the complexity of a Web application increases, the processing andcalculation required by the server increases, which can affect performance. Inaddition, Domino provides many services and functions as a Web applicationserver, from replication to e-mail, to clustering. As the demands on a serverincrease, the increased processing load can further compromise overallperformance.

13.1 PerformanceA lot of information about Domino performance and Domino for AS/400performance has been already published. Our purpose is not to describe all thatinformation again. In the following sections, different sources of information arelisted.

13.1.1 Domino performanceThe following sources of information are available about Domino performance:

• Domino R5 Administration Help• Domino R5 Designer Help• Redbooks• White papers• Web sites

Mail and C&S

Discussion Database

Workflow

Applications that access DB2/400

Web-based

Applications with many views

Applications with Full-Text Search

IntensityI/O___________________________CPU


13.1.1.1 Domino R5 Administration HelpDomino R5 Administration Help contains performance information, which can befound by selecting Performance. Look for the help5_admin.nsf file on the Dominoserver.

Domino R5 Administration Help has a performance section, where you can findhelpful information about Domino and Notes. This database can be found underthe “help” directory in your installed server. See Figure 126 for an example of theDomino R5 Administration Help.

Figure 126. Domino Release 5 Administration Help performance information

Another section that can be very helpful is the System Monitoring section. A goodmonitoring of the system can be considered a pillar for mastering theperformance of a Domino for AS/400 server.

13.1.1.2 Domino R5 Designer HelpDomino R5 Designer Help (Figure 127 on page 250) details development tips andtechniques in numerous documents. Browse through the database with searcharguments such as problem or performance, and you will get very usefulinformation. Look for the help5_designer.nsf file on the Domino server.

Chapter 13. Performance troubleshooting 249

Figure 127. Domino R5 Designer Help

13.1.1.3 RedbooksThe following redbooks contain valuable information on Domino performance:

• Lotus Domino for AS/400: Performance, Tuning, and Capacity Planning,SG24-5162

• Lotus Domino for AS/400 R5: Implementation, SG24-5592 (refer to the“Tailoring for system performance” section, in the “Basic server administration”chapter.)

• Lotus Domino for AS/400: Integration with Enterprise Applications, SG24-5345(refer to the “Performance aspects” section, in the “Options for Integration withEnterprise Applications” chapter.)

• TCP/IP for AS/400: More Cool Things Than Ever, SG24-5190

• Performance Considerations for Domino, SG24-5602

13.1.1.4 White papersThe following white papers are available:

• “Evaluating Appropriate Workloads for the AS/400e Dedicated Server forDomino” white paper is available at: http://www.as400.ibm.com/whpapr/dsd.htm

• Lotus “Maximizing Application and Server Performance in Domino” whitepaper is available at:http://www.as400.ibm.com/developer/domino/perform/maxperform.pdf

13.1.1.5 Web sitesThe following Web sites can be helpful when looking for additional or recentinformation on performance:

• IBM Domino for AS/400 dedicated Web site at:http://www.as400.ibm.com/domino


Select the Performance section.

• Domino for AS/400 performance information is available at:http://www.as400.ibm.com/developer/comm/survey.html

• IBM AS/400 Performance Tool Information at:http://www.as400.ibm.com/sftsol/pt400

• Lotus Iris Web site at: http://www.notes.net

13.1.2 Performance considerationsBefore going further in this chapter, keep in mind that some basic principlesshould be evaluated first, such as:

• Dedicating Domino servers to specific tasks reduces the number of tasks thata server performs and improves response time.

• Limiting the number of users per partition (at the time this redbook waswritten, 1,000 to 2,000 active users was the recommended number of usersper partition with Domino Release 5).

• Using full text indexes, mainly on the mail files.

This can have a large impact on the CPU processing time.

• Replication parameters.

Replication schedules, frequencies, and priorities are among the importantfactors in performance concerns.

13.1.2.1 Partitioning and clustering considerationsRunning multiple Domino partitions on a single AS/400 system has severalpotential performance benefits. First, you should limit the number of active usersper Domino partition. This will limit the number of total active threads and thusreduce contention among the open threads in a given Domino server. Therecommendation if you are running Domino version 4.6.x is 600 to 800 activeusers. In R5, this may be raised to 1,000 to 2,000 active users. Therefore, if youhave more than the maximum number of active users, you should divide themamong multiple Domino partitions.

Another performance benefit of having multiple partitions is the ability to allocatedifferent types of Domino activity to different servers. For example, you can haveyour mail users in a mail Domino partition, so they are not competing forresources with users in another Domino partition who are using an applicationthat requires heavy database accesses or doing Web serving. By splitting yourNotes users into different subsystems, you also increase the availability of yourAS/400 system. If one of your Domino servers goes down, only that subsystem isaffected. All of your other subsystems stay up and running.

Multiple Domino for AS/400 servers are supported on one system via partitioning,one of the advanced functions included in the Domino Enterprise Server license.Up to 30 partitions are supported with Domino for AS/400 4.6.2 and above. Thisincreased from 16 in Release 4.6b.

Furthermore, multiple Domino servers (up to six) can be clustered for failoversupport and load balancing, using another advanced function included in theDomino Enterprise Server license. A cluster can include multiple partitioned


servers on the same AS/400 system, Domino servers on more than one system(like or unlike platforms), or a combination of the two.

If you are using partitioned servers on the same AS/400 system and areclustering them on OS/400 Release V4R2 or V4R3, we recommend that you setthese servers up to use the *loopback option. This saves CPU resources andcommunication time when the clustered servers communicate with each other.This is accomplished by specifying the Loopback IP address (127.0.0.1) insteadof an external IP address. For details on setting up *loopback, see Chapter 6 inLotus Domino for AS/400: Installation, Customization and Administration,SG24-5181.

Note: For OS/400 V4R4, it is better to not configure *loopback manually. TheTCP/IP communications stack has been modified in V4R4 to detect this setupand will do the *loopback for you automatically.

13.2 Handling performance issues

Performance issues may occur even on a server that has been working well for along time. Starting a new application, adding a partition, or starting clustering cancreate sudden and unexpected performance problems. Sizing adequately isessential.

Tools, such as the AS/400 Workload Estimator (available athttp://as400service.ibm.com/estimator) or BEST/1, help in sizing and provideassistance in selecting the right processor.

13.2.1 AS/400 sizing considerationsSoftware performance is directly related to the speed of the hardware on which itruns. As discussed in the details on the AS/400 Domino Web site(http://www.as400.ibm.com), a newer processor offers better performance foradvanced applications such as Domino.Doc or Web serving.

Adequate hardware sizing is the key to a good performance for Domino on theAS/400 system.

Let us take the example of how disk access may impact the e-mail function. Toofew disk arms will have a direct influence on the speed of the mail databaseopening and views. We recommend that you use the sizing tools to determine thenumber of disk arms necessary to ensure good performance.

Processors affect:

• Indexer speed• Replicator speed• Number of maximum possible database transactions• Number of add-ins that can run in parallel

Sizing the number of disk arms is essential. Disk access affects:

• Number of maximum possible database connections• Speed of database and view opening

The memory size is often the bottleneck. Memory affects:


• Number of maximum possible simultaneous Notes client connections(sessions)


• Size of caches

• Server add-in task performance (less paging to disk)

Figure 128 shows sizing considerations about the AS/400 Dedicated Server forDomino, including:

• How the number of users supported by a system largely depends on thecomplexity of the applications used.

• How real-world figures can be different from benchmark figures.

Note that this is true not only for the AS/400 Dedicated Server for Domino, butalso for all AS/400 models, and any other platform running Lotus Domino.

Figure 128. AS/400e Dedicated Server for Domino sizing considerations

13.2.2 Using the statistics informationPrior to solving performance problems and tuning Domino for AS/400, someDomino for AS/400 statistics can prove to be excellent sources of information withuseful data, such as:

You may see certain Lotus documents that discuss the impact of using RAID-5and mirroring on the CPU. These papers refer to CPUs other than the AS/400system. The results they report cannot be applied to the AS/400 platform,whose technology and architecture make these considerations irrelevant.

Sizing Domino on AS/400, with Workload Estimator, for example, will end upwith the same CPU utilization forecast, whether you are using RAID-5,mirroring, or none of these.

Note

Estimated concurrent users for typical Dominoworkloads...

240725 75 430215

x

240840 130 380 760

x2300

2409 //75 715 1430

x4300250

1300

*

Heavy Domino Applications

Typical Domino Applications

80% Typical Domino Mail, 20% Typical Domino Applications

x Typical Mail and/or Light Domino Applications

Simple Mail Users*

*

*

* Estimates; performance in customer environments may vary


• Peak transactions per minute• Time of last peak transactions per minute• Total transactions processed• Number of current users• Peak number of users• Time of last peak number of users• Number of sessions dropped in mid-transaction

These statistic reports are particularly interesting when tuning a Web server,because they provide information that would be hard to find elsewhere, such asthe number of transactions processed. Statistic reports contain the mostfrequently used statistics. To view additional statistics, use the SHOW STAT

command at the server console.

Statistic reports (Figure 129) are found in the Domino database STATREP.NSFon the server disk. The Statistics Reporting template is a special template usedto automatically create a database that records and stores information about oneor more Notes servers. The database serves as a repository for information aboutserver usage, resources, alarms, and events. You can view this informationgraphically in the database and even export it to a spreadsheet for creating moreformal reports.

You can create a Statistics Reporting database for each Domino server or for allof the servers within a particular domain. Each server automatically records itsown statistics in the database, based on a frequency that you determine. Theinterval can be as small as 15 minutes or as large as you want.

Figure 129. Statistics report

For more information on this function, refer to the Domino Administration Helpdatabase on the Domino server.


13.2.3 High activity for CPU, paging, or diskWhen the CPU utilization rate is high (above 7%0 or 80%), or when pagingbegins to rise, or when some disks give a bad average response, the server mayexperience slowdowns.

Several tools are available to determine the CPU utilization rate, the diskresponse time, or the paging activity. These tools include such AS/400 ControlLanguage commands as:

• Work with System Status (WRKSYSSTS) command• Work with Active Jobs (WRKACTJOB) command• Work with Disk Status (WRKDSKSTS) command

They also collect and analyze performance data through Performance Tools forAS/400 LPP (5769-PT1).

A detailed description of those tools is available in the “Monitoring Performancewith AS/400 Tools” chapter in Lotus Domino for AS/400: Performance, Tuning,and Capacity Planning, SG24-5162.

Controlling the AS/400 system values is also good practice. System values areused to control or change overall operation. For AS/400 tuning, pay specialattention to:

• QTOTJOB: Initial total number of jobs• QADLTOTJ: Additional number of total jobs• QACTJOB: Initial number of active jobs• QPFRADJ: Performance adjustment• QMAXACTLVL: Maximum activity level of system• QMCHPOOL: Machine storage pool size• QDYNPTYADJ: Dynamic priority adjustment• QDYNPTYSCD: Dynamic priority scheduler

These system values are also described in the “Monitoring Performance withAS/400 Tools” chapter in Lotus Domino for AS/400: Performance, Tuning, andCapacity Planning, SG24-5162.

Platform statistics are planned to be introduced in Release 5.0.3 of Domino forAS/400. Domino will track performance metrics of the operating system andoutput the results to the server. You will be able to type show stat platform atthe server console to display them. This feature is disabled by default inRelease 5.0.3 and later versions. You can enable it by setting the followingparameter in the NOTES.INI file and restarting your server:

PLATFORM_STATISTICS_ENABLED=1

Metrics are collected continuously, unless disabled or reset (see thecommands in the following section) or unless the server is brought down. Theserver resets this information each time it is started. This feature also expectscertain Notes processes to be present. Performance counters for absent Notesprocesses will have zero values. This is normal behavior.

Note


13.2.3.1 Looking back at the sizingYou should observe and balance the overall system-wide performance beforefocusing on a Domino performance problem. Domino performance is only arelatively small part of the overall performance when working on non-DSDmodels.

If the entire system is functioning poorly, even if no Domino server is started,there is no reason to focus on Domino performance the first time.

13.2.3.2 Jobs consuming CPUIf Domino uses a lot of CPU time at certain times, it may be because of someDomino agents or server tasks.The Domino administrator should be able to tellwhat tasks were running at a given time. If you are experiencing performanceproblems, list the Domino tasks that were running at those times to help with youranalysis.

A performance problem can be caused by one task that requires a lot ofresources (for example CPU) during a long time. An example is the Updatefunction.

Poor performance can also be caused by an excessive number of tasks runningat the time, competing for the same resources.

In both cases, changing the run attributes of a job can be a solution. Job runattributes are discussed in the “Monitoring Performance with AS/400 Tools”chapter in Lotus Domino for AS/400: Performance, Tuning, and CapacityPlanning, SG24-5162.

Installing Domino on an AS/400 system which is already used to run a line ofbusiness applications should be done with a special care. If the average CPUutilization rate is already high (for example over 50%) or disk usage is high (forexample 40%), the additional load created by Domino for AS/400 may lowerthe performance of existing applications.

All AS/400 RISC models can technically run Domino. However, older modelsperform certain Domino functions slowly. Recent models, such as recent 170and 7xx models, are recommended for Domino.

Notes

The Update task updates all active views in a database when a user or a servertask, such as the Replicator, updates any documents in the database. TheUpdate task also updates the full text index if one was created for thedatabase.

The resources required by the Update task depend on the size and thecomplexity of the views in the database and on the size of the database thatneeds to be updated.

Notes


A job entering the AS/400 system receives its run-time attributes from the classdescription. Each of the attributes contributes to how a thread is processed. Therun priority is a value ranging from 0 (highest priority) through 99 (lowest priority).It represents the importance of a job when it competes with other jobs formachine resources. The value is the highest run priority allowed for any threadwithin the job. Individual threads within the job may have a lower priority.Changing the run priority for the job affects all threads within the job.

By default, all Domino-related threads run on a priority of 20, which is on thesame priority level as the interactive threads.

When you lower a run priority, the threads within that job stay in a queue longer.When you raise the priority, you decrease the time the threads are in a queue.Therefore, it gains control of the CPU more often, and completes the job faster.The priority for jobs that run in batch environments should be lower than thepriority for jobs in interactive environments.

You can use the Work with Active Jobs (WRKACTJOB) command to change therun attributes of a specific job. However, this operation requires caution asmentioned earlier because some tasks running in the DOMINOxx subsystemneed a run priority equal to interactive jobs (SERVER or HTTP for example).Some of the tasks are more like traditional batch jobs, where a job is submittedand processed in the background (BILLING for example).

To change the attributes of a job, such as the run priority for the HTTP job,position the cursor in front of the HTTP job, and enter 2 for the Change in theAction prompt. Press Enter, you can now update the value to your requirements.

With Domino 5.0, you can change the individual job priority permanently byaffecting specific classes to some tasks of a Domino server.

For example, you want to create a class on Domino server DOMSVR1 with a runpriority that equals 30, and place some tasks there that can run in a lower priority,such as BILLING and STATS. Follow these steps:

1. Create a class in library QUSRNOTES:

CRTCLS CLS(QUSRNOTES/SPECIALCLASS) RUNPTY(30)

2. Edit the /qibm/userdata/lotus/notes/domino_classes file to tell Domino whichtask (or tasks) should use this class:

EDTF ('/qibm/userdata/lotus/notes/domino_classes')

For V4R4, enhancements for the performance of threaded applications havebeen made. If you perform manual performance tuning on your system (thesystem value QPFRADJ is set to 0 or 1), you may have to tune your systemagain after V4R4 has been installed to avoid performance degradation whenrunning Domino for AS/400.

If the system value for QPFRADJ is set to 2 or 3, the system will make theappropriate adjustments to keep Domino running smoothly on the new releaseof the AS/400 system.

Note


For more information on the Edit File (EDTF) command, refer to 7.1,“Displaying or editing a stream file from a 5250 display” on page 117.

3. Type the following input:

SERVER=DOMSVR1CLASS=specialclassTASKS=BILLING,STATS

Note that several servers can be mentioned in the/qibm/userdata/lotus/notes/domino_classes file.

For example, the content of the /qibm/userdata/lotus/notes/domino_classes filecan be:

SERVER=DOMSVR1CLASS=specialclass1TASKS=BILLING,STATSSERVER=DOMSVR5CLASS=specialclass1TASKS=BILLING,STATSCLASS=specialclass2TASKS=HTTP

In the preceding example, the file contains task class settings for:

• Tasks BILLING and STATS running with the specialclass1 class in the Dominoserver DOMSVR1

• Tasks BILLING and STATS running with the specialclass1 class in the Dominoserver DOMSVR5

• Task HTTP running with the specialclass2 class in the Domino serverDOMSVR5

All tasks can be changed, except SERVER and QNNINSTS.

13.2.3.3 Don’t exclude a malfunctionA performance problem may also be the consequence of a malfunction (programloop, for example). A job may use a lot of resources without any sensible reason,and you will need to look for the cause of this problem.

13.2.4 Determining network problemsWhile observing the performance, it is important not to forget to verify that thenetwork is not the bottleneck. If you had all the CPU power in the world, but a badnetwork, your users would suffer from poor response times or low transactionthroughput.

Performance problems can be introduced by network configuration changesmade without the awareness of the Domino administrator.

This procedure works only for Domino Version 5 running on OS/400 V4R3 orlater.

Note


13.2.4.1 Using the PING commandThe PING command sends an echo to an IP host to determine whether the hostis accessible. The response time shows how long the round trip message takes.This is the first tool to use to identify connectivity problems. The PING commandcan be issued on any workstation that supports TCP/IP, for example:

• Workstation client command line• AS/400 workstation• UNIX client

Figure 130 illustrates using the ping command from a 5250 display.

Figure 130. PING command submitted from a 5250 display

In the following example, the PING command sends a data packet of 256 bytesfive times in a row to remote system DOMSVR1:

PING RMTSYS('DOMSVR1')Verifying connection to host system 1.2.3.4.PING reply 1 from 1.2.3.4. took 36 ms. 256 bytes. TTL 62.PING reply 2 from 1.2.3.4. took 8 ms. 256 bytes. TTL 62.PING reply 3 from 1.2.3.4. took 12 ms. 256 bytes. TTL 62.PING reply 4 from 1.2.3.4. took 8 ms. 256 bytes. TTL 62.PING reply 5 from 1.2.3.4. took 13 ms. 256 bytes. TTL 62.Round-trip (in milliseconds) min/avg/max = 8/15/36Connection verification statistics: 5 of 5 successful (100 %)

The minimum, maximum, and average response times are displayed, and theaverage response time is 15 milliseconds.

If you experience response times between a hundred and a thousandmilliseconds or even time-outs, use the PC TRACERT command to obtain additionalinformation.

The TRACERT command is described in 14.4, “Tracing routes with TRACERT” onpage 271.

Verify TCP/IP Connection (PING)


Remote system . . . . . . . . . DOMSVR1

Remote internet address . . . .

Additional Parameters

Message mode:Response message detail . . . *VERBOSE *VERBOSE, *QUIETSummary, if response errors . *COMP *COMP, *ESCAPE

Packet length (in bytes) . . . . 256 8-512Number of packets . . . . . . . 5 1-999Wait time (in seconds) . . . . . 1 1-120Local internet address . . . . . *ANYType of service . . . . . . . . *NORMAL *MINDELAY, *MAXTHRPUT...

More...F3=Exit F4=Prompt F5=Refresh F12=Cancel F13=How to use this displayF24=More keys


13.2.4.2 Network settingsObserving the Network Performance is mandatory to determine whether thenetwork is not the bottleneck. If you had all the CPU power in the world but a poornetwork, your users would suffer from poor response time or low transactionthroughput anyway.

These communication tuning recommendations apply to all Domino environmentssince the Notes Object store is rich and can contain attachments that can alsouse these tuning recommendations. However, these guidelines are important forDomino Web serving as Web users are stateless. This means that when a Webuser requests information from the Domino server, the server not only needs tosend back the requested data, it also needs to send back the design element anda response document for every page that is served.

The AS/400 Configure TCP/IP (CFGTCP) command is used to display a menuthat allows a user to define or change the Transmission Control Protocol/InternetProtocol (TCP/IP) configuration. Issue the CFGTCP command on the AS/400command line and press Enter.

At the Configure TCP/IP screen, select option 3 (Change TCP/IP Attributes). Thisbrings you to the Change TCP/IP Attributes screen.

Consider the following network settings carefully:

• MTU size• Maximum Frame (MAXFRAME) size• Type of Service (TOS)• Buffer size• DNS configuration• SNA support• Filtering• Duplex

Maximum Transmission Unit (MTU) sizeThe MTU size specifies the maximum size (in bytes) of IP datagrams that can betransmitted through this interface. A datagram is a basic unit of informationpassed over an Internet network.

We recommend setting this value to *LIND, which sets the MTU to the largestamount of data that can be transmitted on the line based on the line descriptioninformation. By increasing the value of this parameter, you can reduce the overallnumber of transmissions and, therefore, increase the potential capacity of theCPU and the input/output processor (IOP).

Similar parameters also exist on the Web browser. The negotiated value will bethe minimum of the server and browser (and perhaps any bridges or routers), soincrease them all. This parameter can be changed by using the CFGTCPcommand.


Maximum frame (MAXFRAME) sizeThe Maximum frame size is a parameter on the CHGLINTRN command thatcontrols the size of the frame that is sent and received. The size of this frame canhave quite an impact on the performance of the AS/400 communication controller.We recommend that you change this value to the maximum for the linedescription used for your Domino Web server.

To change the Maximum Frame Size, enter the CFGTCP command. At theConfigure TCP/IP screen, select option 1 (Work with TCP/IP Interfaces). Thisbrings you to the Work with TCP/IP Interfaces screen. Write down the LineDescription of the IP address that is being used by your Domino server.

Type of Service (TOS)This parameter specifies the type of service to be used. The type of servicedefines how the Internet hosts and routers should make trade-offs betweenthroughput, delay, reliability, and cost. The default value is set to *NORMAL, butwe recommend that you set the value *MAXTHRPUT. This means that a high datarate is important for data on this connection.

If you change the Type of Service setting in the TCP/IP configuration, you shouldalso modify it accordingly in the routes configuration. To change the routeconfiguration, issue the CFGTCP command on the AS/400 command line and pressEnter. At the Configure TCP/IP screen, select option 2 (Change Routes).

TCP/IP buffer sizeWe recommend that you maximize the size of send (TCPSNDBUF) and receive(TCPRCVBUF) buffers. Increasing buffer size can increase Web servingperformance.

Our recommendations include:

• Increasing from the default of 8192 to 64000 KB• If there is enough memory, set it to a maximum of 8388608 KB

The size of both buffers can be changed using the Configure TCP/IP (CFGTCP)command or Configure TCP/IP Attributes (CHGTCPA) command.

The TCP Receive Buffer Size specifies what to allocate for the default receivebuffer size. The TCP receive window size is based on this value. Decreasing thisvalue decreases the amount of data that the remote system can send beforebeing read by the local application. Decreasing this value may improveperformance in situations where many retransmissions occur due to theoverrunning of a network adapter. Increasing the buffer size that is used byTCP/IP can increase the Web serving performances.

The TCP Send Buffer Size specifies the TCP send buffer size. This parameterinforms TCP what to use for the default send buffer size.

You have to be careful here because you do not want to set this value tosomething larger than what the router can handle. If you have a slower router,you will only be able to set this value and the others we discuss later in thischapter to the maximum that the router can support.

Note


The TCP send buffer size provides a limit on the number of outgoing bytes thatare buffered by TCP. Once this limit is reached, attempts to send additional bytesmay result in the application blocking until the number of outgoing bytes buffereddrops below this limit. The number of outgoing bytes buffered is decrementedwhen the remote system acknowledges the data sent.

Setting these parameters does not guarantee the size of the TCP receive or sendbuffer. This is the default buffer size that is used for initial TCP connectionnegotiations. An individual application can override this value by using theSO_RCVBUF socket option. For more information, see the publication SocketsProgramming, SC41-5422.

Domain Name SystemThe DNS configuration should be turned off if possible. This is done through theparameter CHGTCPDMN (CHGTCPDMN = *LOCAL). Entries should appear in list ofhost table entries (CFGTCP option 10).

The Domain Name System translates domain names to IP addresses. DNS is adistributed database with a structure that’s similar to the structure of a filesystem. The complete database or file system can be described as an invertedtree with the root at the top. Each node in this tree represents a partition of thedatabase. Each domain or directory can be further divided into partitions, calledsubdomains (such as the file system's subdirectories). The domain name spaceis “tree” structured. The top-level domains are dividing the Internet domain namespace organizationally.

For AS/400 DNS support, you need to install 5769-SS1 OS/400 option 31,Domain Name System, on the AS/400 system. You can use GO LICPGM option11, Install licensed programs, to uninstall or install the DNS OS/400 option.

SNA over TCP/IPYou can configure SNA over TCP/IP. However, check whether SNA support isactually needed. It often can happen that SNA has been configured and activatedbut is not used at all. You can disconnect it using one of the following twomethods:

• If using TCP/IP only, set the TCPONLY parameter on Ethernet LineDescription = *YES

• Set AUTOCRTCTL = *NO on Ethernet Line Description

IP packet filteringIP packet filtering support lets you explicitly control what IP traffic is allowed inyour network. IP Packet Security protects the AS/400 system on the Internet withIP packet security. It contains two components, IP packet filtering and NetworkAddress Translation, that act as “firewall” protection for your system. Packetfilters are set rules that limit IP packet flow into or out of a network. You candefine policies that determine which packets are allowed access into or out of thenetwork. If there is no matching rule, the default rules are used to deny accessand discard the packets.

If using Ethernet cards, consider using the 2838 Ethernet card:


• If the part number is 21H5458, this card has built-in filtering. It should not havean external filter because double filtering can cause problems.

• If the part number is anything else, this card requires an external filter. Usepart number 97H7385, filter cable.

DuplexThe Duplex setting determines whether the request-to-send (RTS) is permanentlyset on (for duplex modems) or turned on only when transmission is required (forhalf duplex modems). For Ethernet lines, consider the following settings in theline description, obtained by the Work with Line Description (WRKLIND)command:

• Recommended Line Speed is 100 Mb

• Connected directly to the 8271 Ethernet switch

If the switch's port is set to FULL duplex, the Duplex setting on Ethernet lineshould be FULL duplex.

• Connected to a “stackable” hub

Ensure that the duplex setting on the line is set to *HALF (matches the settingon the hub).

For additional information on network settings, refer to the redbook TCP/IP forAS/400: More Cool Things Than Ever, SG24-5190.

Note


Chapter 14. Network troubleshooting

When connectivity is lost, but all other components are working properly, youneed to investigate the network. These problems are usually routed to a separatenetwork organization to isolate the problem to a router, common carrier service,or other failing component. Some useful tools include:

• NETSTAT: An OS/400 command for local host TCP/IP information• PING: Used in checking connectivity• NSLOOKUP: Perform DNS name validation• TRACERT: For tracing routes over TCP/IP networks• Domino Administrator Display: The graphical console• Network Trace: From Notes Client Desktop• NotesCONNECT or NPING: An extension to PING that tests for the presence

of the Domino server

14.1 NETSTAT command

You can use the Work with TCP/IP Network Status (NETSTAT) command. On anyAS/400 command line, enter NETSTAT *CNN. Figure 131 shows the list obtainedafter scrolling through the display several times.

Figure 131. Work with TCP/IP Network Status (NETSTAT) example

In our example, shown in Figure 131, port 1352 is used three times. The Dominoserver usually listens on the standard port 1352. In our example, there are twopartitioned Domino servers running on the same AS/400 system, each of themusing a single IP address.

Work with TCP/IP Connection StatusSystem: AS25




* * 10400 000:28:39 *UDP* * 1352 001:06:16 Listen* * 500 055:06:02 *UDP* * 500 055:06:02 *UDP* * 500 055:06:02 *UDP* * 5148 055:06:02 *UDP10.8.6.148 1027 telnet 000:00:00 Established10.8.62.125 1047 1352 000:02:40 Established10.8.62.208 1079 telnet 000:01:07 Established10.8.62.220 1544 1352 000:00:11 Established10.7.92.37 as-mgtc > 6613 005:30:07 Established



The first time port 1352 appears, it corresponds to a first running partitionedserver, without any connected Lotus Notes client. The second and thirdoccurrences of port 1352 correspond to the connections for two different users,working at IP addresses 10.8.62.125 and 10.8.62.220.

Figure 132 shows the detailed connection information for the second user. Noticethat the name of the local host name is the same as the name of the partitionedDomino server. The observed state is Established.

Figure 132. Display TCP Connection Status details

The Domino server listens on the standard port 1352 of all IP addressesdefined to the AS/400 system, unless a specific address has been specified. Ifusing R4.6.3 or earlier, you must manually edit the NOTES.INI file to bind theserver to a specific IP address. Later releases allow you to choose a specific IPaddress while using the Configure Domino Server (CFGDOMSVR) command.

Note

Instead of looking at the connections for all IP addresses defined on theAS/400 system, it is possible to limit the list of connections to one IP address.

After entering NETSTAT *CNN, press F24 (More keys) and then F15 (Subset bylocal address). This allows you to select one local Internet address among allIP addresses defined on the AS/400 server.

If your Domino server is using a specific IP address, you need to choose itthere. Only connections with a local Internet address equal to this Internetaddress are displayed in the list.

Tip

Display TCP Connection StatusSystem: AS25

Connection identification:Remote host name . . . . . . . . . . . . . . :Remote internet address . . . . . . . . . . : 10.8.62.220Remote port . . . . . . . . . . . . . . . . : 1544

Local host name . . . . . . . . . . . . . . . : DOMSVR3Local internet address . . . . . . . . . . : 10.8.62.33Local port . . . . . . . . . . . . . . . . : 1352

Associated user profile . . . . . . . . . . . : QNOTESTCP programming interface information:State . . . . . . . . . . . . . . . . . . . . : EstablishedConnection open type . . . . . . . . . . . . : Passive

Timing information:Idle time . . . . . . . . . . . . . . . . . . : 000:00:11.937Last activity date/time . . . . . . . . . . : 03/01/00 16:20:16

Round-trip time . . . . . . . . . . . . . . . : .079Round-trip variance . . . . . . . . . . . . . : .052

More...Press Enter to continue.F3=Exit F5=Refresh F6=Print F9=Command line F10=Display IP optionsF12=Cancel F14=Display port numbers F22=Display entire field


When connections have been made with the Domino server, the Domino serverSHOW USERS command should show the same number as the number ofconnections shown in a NETSTAT command. If the numbers do not match, moreinvestigation is required.

Performance problems and slowdowns can mislead people into believing theyhave a network problem. One specific problem where a “Close-Wait” state hasbeen encountered is referred to as the “hex session” or “ghost session” problem.That name comes from the fact that a Domino show users debug command listsmany lines of hexadecimal session IDs without any user identification. If theDomino server cannot authenticate the users quickly enough as they try toconnect, the user may try again several times and create many more pendingconnections. Therefore, the Domino server does more processing because thesesessions are tying up resources and create longer task queues through which theserver must search.

14.2 Checking network connectivity

When users describe problems with symptoms such as “can’t get connected,”“server not responding,” or “request timed out,” there may be a network problem.If the difficulty is isolated to only one user, the problem is in the client’s machineor in their local network environment. If many users report problems, theinvestigation should begin from the server and go out to the network. Mail routingproblems may be an indication of network problems between servers.

14.2.1 Using PINGWhen a user reports no connectivity or a loss of connectivity to the Dominoserver, the first test is for the end user to issue a ping command pointing to thehost in question by numeric IP address from an MS-DOS or PC session.

The specific options used in this example are to send a 1024-byte test frame 100times (see Figure 133). If the number of bytes returned is seen to decreaseduring this test, it indicates a network congestion problem. This approach cannotbe used from the server because the bytes returned by the target are not shown,as shown in Figure 135.

Figure 133. Ping from a Windows 95 client

C:\>ping domsvr3

Pinging DOMSVR3 [10.8.62.33] with 32 bytes of data:

Reply from 10.8.62.33: bytes=32 time=4ms TTL=62Reply from 10.8.62.33: bytes=32 time=4ms TTL=62Reply from 10.8.62.33: bytes=32 time=5ms TTL=62Reply from 10.8.62.33: bytes=32 time=3ms TTL=62

Ping statistics for 10.8.62.33:Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:Minimum = 3ms, Maximum = 5ms, Average = 4ms

C:\>

Chapter 14. Network troubleshooting 267

If this test succeeds, try to ping the remote client from the server. First, obtain theTCP/IP address of the client machine by having the user issue the appropriatecommand for their system. For example, from Windows NT, issue the ipconfig

command, as shown in Figure 134. The WINIPCFG command can be used on aWindows 95 or Windows 98 workstation.

Figure 134. Results of the ipconfig command from a Windows NT client

Then, you can use the ping command on the AS/400 system. See Figure 135 foran example.

Figure 135. Results of the ping command from a server

Success with the ping command verifies that a path to the client is clear and thatyou can communicate with IP through network routers. At this point, we suggesthaving the workstation ping its own numeric IP address. If this test fails, TCP/IP isnot properly configured on the client’s machine and needs to be fixed.

If the response times displayed in the ping command are not consistent, thenetwork may be congested.

The final two verifications of the TCP/IP and Domain Names Server setup are:

1. Ping the server from itself using the DNS alias name, such as DOMSVR3.2. Ping the server from the workstation the same way.

C:\>ipconfig

Windows NT IP Configuration

Token Ring adapter IBMTRP1:

IP Address. . . . . . . . . : 10.8.62.150Subnet Mask . . . . . . . . : 255.255.255.0Default Gateway . . . . . . : 10.8.62.2

Display All MessagesSystem: AS25

Job . . : QPADEV000Q User . . : RBOUTAUL Number . . . : 154241

3>> PING RMTSYS('10.8.62.220')Verifying connection to host system 10.8.62.220.PING reply 1 from 10.8.62.220 took 52 ms. 256 bytes. TTL 126.PING reply 2 from 10.8.62.220 took 5 ms. 256 bytes. TTL 126.PING reply 3 from 10.8.62.220 took 5 ms. 256 bytes. TTL 126.PING reply 4 from 10.8.62.220 took 4 ms. 256 bytes. TTL 126.PING reply 5 from 10.8.62.220 took 4 ms. 256 bytes. TTL 126.Round-trip (in milliseconds) min/avg/max = 4/14/52Connection verification statistics: 5 of 5 successful (100 %).


F3=Exit F5=Refresh F12=Cancel F17=Top F18=Bottom


Failures here should be referred to the network support group. We recommendthat the server host alias names should be the same as the Domino Notes servernames.

14.2.2 Resolving PING command problemsA host may fail to respond even after several ping commands for any of thefollowing reasons:

• The host is not listening to the network.

• The host is inoperative, or some network or gateway leading from the user tothe host is inoperative.

• The host is slow because of activity.

• The packet is too large for the host.

The echo request sent by the ping command does not guarantee delivery. Morethan one ping command should be sent before you assume that a communicationfailure has occurred.

Use additional ping commands to communicate with other hosts in the network todetermine the condition that is causing the communication failure. However, youshould know the network topology to determine the location of the failure.

Issue the ping commands in the following order until the failure is located.

1. Send a ping command to your local host. A successful ping command sent toa different host on the same network as the original host suggests that theoriginal host is down or is not listening to the network.

2. Send a ping command to a host other than your local host on your localnetwork.

3. Send a ping command to each intermediate node that leads from your localhost to the remote host, starting with the node closest to your local host.

If you cannot get echoes from any host on that network, the trouble is usuallysomewhere along the path to the remote hosts.

Direct a ping command to the gateway leading to the network in question. If theping command fails, continue to test along the network from the target until youfind the point of the communication breakdown.

14.3 NSLOOKUP command

TCP/IP name resolution problems are commonly reported by such messages asNotes server is not a known host or Server not responding.

If you are using a hosts file, check for illegal characters. Make sure there are noillegal characters such as a space or a letter in the numeric IP address. Eachsection of the dotted decimal numeric IP address is no longer than three numbersand there are four sections to an address such as 10.8.62.33.

Make sure there are no illegal characters in the Names Fields. Only alphabeticcharacters, numbers, and dashes should appear. Spaces and underscores arenot allowed according to DNS standards.


Check for any changes made to the hosts file recently and confirm that theinformation in the hosts file is correct. All the target machines that a computermay contact must be defined in the local hosts file.

The Start DNS Query (NSLOOKUP) command, on the AS/400 system, uses theAS/400 TCP/IP settings to determine the search order and source of DNSinformation.

You can verify that a DNS server is responding correctly before you configureyour system to use it. You can also retrieve DNS information about hosts,domains, and DNS servers.

NSLOOKUP displays the name and IP address of the default name server. This istaken from option 12 of the Configure TCP/IP (CFGTCP) command.

From the Domino point of view, to verify that the Domino server has the correctnames configured, check the server document in the Domino directory (formerlyknown as the Notes Name and Address Book). Make sure the Domino server’scommon name in the TCP port’s Net Address Field is the same as the TCP hostname. If this is not the case, a name resolution alias is required in either theHOSTS file or in the DNS table.

If changes are required to the server document, restart the Domino server andverify that the changes have taken effect.

As shown in Figure 136, we had defined an alias in the DNS because the Dominoserver’s common name was not the same as the TCP host name. Our host nameas defined in the hosts table was DOMINO3, while the Domino common name wasdomsvr3.site1.company.com.

It is important to check option 12 settings of the Configure TCP/IP (CFGTCP)command.

The value of the Domain Name Server Retry (DMNSVRRTY) parameter shouldalso be checked. It consists of two elements: number-retries and time-interval.The first element, number-retries, specifies the number of additional attemptsmade to establish communication with each Domain Name Server (DNS)specified in the INTNETADR parameter, in the event that the first attempt fails.

Our recommendation is to have a value for the number of retries lower than 5.

Note


Figure 136. Net address in a Domino server document

14.3.1 Client error: ‘Not a known TCP/IP host’Some Notes client name resolution errors occur because a server has beenmoved and the client is resolving the name using out-of-date information cachedin hidden fields within their local location document. The error is reported with Not

a known TCP/IP Host message.

The hidden fields are in the location document referred to as the Notes NameServer Cache (NNSC). They provide fast execution using the Notes NamedNetwork (NNN) that the client and server share.

To force the client to resolve the name to a new address, you can perform one ofthe following tasks:

• Change the client NOTES.INI file and restart the client. The setting to changeis Dont_Use_Remembered_Addresses=1. Reverse the setting aftersuccessfully resolving to the correct address.

• Create a new location document. This does not have the incorrect address inthe hidden fields.

14.4 Tracing routes with TRACERT

Tracing a route through the network can be done from the client with theTRACERT command. This can help you understand the slow response time youmay experience.

In the example shown in Figure 137 on page 272, the path taken to this serverwent through seven hops. Performance problems can be introduced by networkconfiguration changes made without the awareness of the Domino administrator.


Figure 137. Tracing the network route from a Windows 95 workstation

14.5 Domino Administrator display

The administrative client provides the Domino server’s point of view of who isconnected and what resources are used. Figure 138 shows the results of thefollowing commands:

• show port tcpip: Shows current sessions on the port

• show users debug: Displays authenticated users and their associatedTCP/IP connection

• show dbs: Shows database stats, number of file descriptors, maximumwaiters, current waiters, and more

These commands are all standard Domino operator commands and can beentered from the Domino server console on a 5250 display.

This information helps the administrator validate network data. TCP/IPconfigurations can be compared to other display commands such as onetstat. Auser’s logon status can be verified when troubleshooting a connectivity issue.

C:\>tracert 10.24.1.154

Tracing route to system2.site5.company.com [10.24.1.154]over a maximum of 30 hops:

1 2 ms 1 ms 1 ms system5.company.com [10.8.62.1]2 2 ms 1 ms 1 ms system10.site3.company.com [10.8.101.25]3 58 ms 63 ms 57 ms 10.2.105.454 100 ms 102 ms 102 ms 10.2.1.455 210 ms 214 ms 209 ms 10.2.74.546 231 ms 230 ms 228 ms 10.1.255.747 235 ms 238 ms 237 ms system2.site5.company.com [10.24.1.154]

Trace complete.


Figure 138. Administrative Client: ‘sh port tcpip’, ‘sh user debug’, and ‘sh dbs’

14.6 Connection trace from a Notes client

A Notes user can trace the connection to the Domino server from the Notesdesktop. This verifies that a complete TCP/IP connection can be made using thesymbolic names of the Domino server. This is a good test for the Help Desk torequest when a user cannot access a server.

As shown in Figure 139, to perform this connection trace, the user clicksFile->Preferences->User Preferences->Ports->Trace. The user can select froma pull-down list of servers that the client has accessed.

Figure 139. Client Trace Connections function


14.7 NotesConnect diagnostic aid

Ping only uses IP/ICMP protocol to test a network connection. There is noapplication-specific protocol or data exchanged during the connection.

An extension to ping called NotesCONNECT (or NPING) is available and issupported on both Windows NT and Windows 95 and 98. Both a graphicalinterface and a command-line version of the tool are available.

The NotesConnect V2.0 TCP/IP Diagnostics Tool is available on the Internet fromNotes.net (http://www.notes.net). NotesCONNECT is also described in a LotusSupport Knowledge Base document “NotesConnect Utility for Use inTroubleshooting IP Connectivity” Issues (Knowledge Base document 2755).

NotesCONNECT (NPing) is a TCP/IP diagnostics tool designed to verify that aservice on a given machine is available. This is accomplished establishing anend-to-end TCP/IP connection with the target Host/application without using theNotes NAB or address resolution logic. This tool requires that Notes be installedon the local machine.

Nping establishes the connection with TCP/IP rather than IP/ICMP, which pinguses. There is no application-specific protocol or data exchanged during thenping test. It simply connects and disconnects. This permits the tool to go into theNotes TCP/IP Interface (NTI) layer.

This tool is ideal for determining if an TCP/IP connectivity problem is Notesrelated or an IP infrastructure problem. This connection that is established is aTCP/IP connection, as opposed to ping, which uses an IP connection over ICMP.No application or service specific protocol or data is exchanged during the pingconnection.

New features in Version 2.0 include:

• A command-line version of NotesConnect called npingcl.exe• Target Server History List• Target Server Type-ahead

For more information, see the IRIS Today article “Testing TCP/IP connections withNotesCONNECT” by Barbara Burch. This article can be viewed online at:http://www.notes.net/today.nsf

In the Archives section, click Perform a full-text search of all Iris Todayarticles, and search for “Testing TCP/IP connections with NotesCONNECT”. Thearticle was last updated 16 June 1997.

Download nconnect.exe, and run it in your Notes executable directory. Thenconnect.exe is a self-extracting zip file. Next run the nping.exe. NotesConnectuses Notes DLLs and needs access to your NOTES.INI file.


For the example shown in Figure 140, nping was started without any parameters,so the GUI interface prompted the user for the information. The address or DNSname of the target server is specified, the number of times to perform the test isselected with a slider bar, and the specific service is selected from a drop-downmenu. The results of the test appear in the message box.

Figure 140. NotesCONNECT GUI with results of the test

NotesCONNECT does not connect through proxies (only to the proxy port) anddoes not connect to UDP ports.

NotesCONNECT is not an officially marketed Lotus product. It is presented asa troubleshooting utility only. Therefore, its use is not directly supported byLotus Support analysts.

Note


Part 3. Appendices

This part contains appendices that cover:

• Domino job names and corresponding functions

• Domino problem determination tools

• A list of debugging NOTES.INI settings that can be used for problemdetermination


Appendix A. Domino job names and corresponding functions

All of the Domino server jobs listed in Table 20 run in a subsystem (usually calledDOMINOxx, unless you decide to choose a specific name while configuring theDomino server), under the user profile QNOTES.

Table 20. Domino server jobs

Program Load programname

Description Default startup setin NOTES.INI

AdministrationProcess

Adminp Automates a varietyof administrativetasks.

ServerTasks

Agent manager AMgr Runs agents on oneor more databases.

ServerTasks

Billing Billing Collects allgenerated billinginformation.

ServerTasks(Advanced Serviceslicense only)

Calendar Connector Calconn Part of the FreeTime Manager thathandles requests forfree-timeinformation fromanother server.

ServerTasks

Cataloger Catalog Updates theDatabase Catalog.

ServerTasksAt1

Chronos Chronos Updates full textindexes that aremarked to beupdated hourly,daily, or weekly.

None

ClusterAdministrationProcess

Cladmin Oversees thecorrect operation ofall components of acluster.

None

Cluster DatabaseDirectory Manager

Cldbdir Updates the ClusterDatabase Directoryand managesdatabases withcluster-specificattributes.


Cluster Replicator Clrepl Performs databasereplication with acluster.


Collector Collect Collects statistics formultiple servers.

None

Databasecompactor

Compact Compacts alldatabases on theserver to removewhite space and freeup disk space.

None


Database fixup Fixup Locates and fixescorrupteddatabases.

None

Designer Design Updates alldatabases to reflectchanges totemplates.

ServerTasksAt1

Event Event Monitors events ona server.

None

HTTP Server HTTP Enables a Dominoserver to act as aWeb server sobrowser clients canaccess databaseson the server.

None

IMAP Server Enables a Dominoserver to act as amaildrop for IMAPclients.

None

Indexer Updall Updates all changedviews or full-textindexes for alldatabases.

ServerTasksAt2

LDAP Server LDAP Enables a Dominoserver to provideLDAP directoryservices to LDAPclients.

None

NNTP Server NNTP Enables a Dominoserver to act as anews server forNNTP clients.

None

Object storemanager

Object Performsmaintenanceactivities ondatabases and mailfiles that use sharedmail.

ServerTasksAt2*

POP3 Server POP3 Enables a Dominoserver to act as amaildrop for POP3clients.

None

Replicator Replica Replicatesdatabases withother servers.

ServerTasks

Reporter Report Reports statistics fora server.

None




A.1 SDD/NAB synchronization jobs

The System Distribution Directory (SDD)/Notes Address Book (NAB)synchronization jobs are listed in Table 21.

Table 21. System Distribution Directory (SDD)

Router Router Routes mail to otherservers.

ServerTasks

Schedule manager Sched Part of the FreeTime Manager thatreturns meetingtimes and dates andavailable invitees.

ServerTasks

Statistics Statlog Records databaseactivity in the log file.

ServerTasksAt5

Stats Stats Generates statisticsfor a remote serveron demand.

ServerTasks

Web Retriever

Server Startup Job

Web

QNNINSTS

Implements theHTTP protocol toretrieve Web pagesand convert theminto Notesdocuments.

LaunchesServerTask. Thejoblog stores allerror messageslogged on theconsole at startup.

None

Job Subsystem User profile Function

QNNDI2NA QSYSWRK QNOTES Synchronizes SDDto NAB.

QNNDI2SD QSYSWRK QNOTES Synchronizes NABto SDD.

QNNDIMSG QSYSWRK QNOTES Logs messages inNNDILOG.NSF.

QNNDILCK Allows only oneactive configuration.Called byQNNINADD whenthe Domino serverstarts and by the DBHook Driver when aconfiguration isenabled.



Appendix A. Domino job names and corresponding functions 281

A.2 SMTP/MIME MTA program objects and jobsThe SMTP/MIME MTA program objects/jobs are listed in Table 22.

Table 22. SMTP/MIME MTA program objects/jobs

QNNDIRQS QSYSWRK QNOTES QNNDIRQS is oneof the jobs that startswhen option 1 isinstalled on yourAS/400 system. It isrelated to directorysynchronization, butis not under thecontrol of theDomino server. Youcan end it using anENDJOB command.

QNNDISTJ QSYSWRK QNOTES ASJ that submitsQNNDI2NA,QNNDI2SD, andQNNDIMSG jobs.

QNNDIEND QSYSWRK QNOTES Ends QNNDI2NA,QNNDI2SD, andQNNDIMSG.


SMTPMTA DOMINOxx QNOTES Main SMTPMTAtask.

DRT DOMINOxx QNOTES Delivery Reporttask.

OMSGCNV DOMINOxx QNOTES Outbound MessageConversion task.

OSESCTL DOMINOxx QNOTES Outbound SessionController Task.

IMSGCNV DOMINOxx QNOTES Outbound SessionHandler task.

ISESCTL DOMINOxx QNOTES Inbound MessageConversion task.

ISESHLR DOMINOxx QNOTES Inbound SessionController task.

MTA DOMINOxx QNOTES MTA PlatformIndependent andPlatform dependentOS Interfaces.

MTS DOMINOxx QNOTES Mapping TableService.

MTAWQ DOMINOxx QNOTES MTA Work QueueInterfaces.



CCCVS DOMINOxx QNOTES ConversionServices.

SMTPLDQ DOMINOxx QNOTES SMTP Local DeliverRequest Queue.


Appendix A. Domino job names and corresponding functions 283

Appendix B. Domino problem determination tools

Table 23 shows an overview of tools described in different chapters of thisredbook.

Table 23. Domino tools

Tool Problems that the tool resolves How to access the tool

Server log file (LOG.NSF) All problems From the Server - Analysis tab in theDomino Administrator

Domino Web server log file(DOMLOG.NSF)

Web server problems From the Server - Analysis tab in theDomino Administrator

Server's MAIL.BOX Mail routing problems From the Messaging - Mail tab in theDomino Administrator

Mail trace Mail routing problems From the Messaging - Mail tab in theDomino Administrator

ISpy Slow mail; server problems Configured in the Statistics and Eventsdatabase on the Configuration tab in theDomino Administrator

Mail reports Mail user activity From the Messaging - Mail tab in theDomino Administrator

Mail tracking Lost mail From the Messaging - Tracking Center tabin the Domino Administrator

Mail routing status Undelivered mail From the Messaging - Mail tab in theDomino Administrator

Mail routing topology maps Mail routing problems between servers From the Messaging - Mail tab in theDomino Administrator

Mail routing events view inthe log file (LOG.NSF)

Undelivered mail From the Messaging - Mail tab in theDomino Administrator

Shared Mail view in the logfile (LOG.NSF)

Disk space usage From the Messaging - Mail tab in theDomino Administrator

Network trace Connection problems In User Preferences. Choose File->Preferences->Notes Preferences

Replication events in the logfile (LOG.NSF)

Replication problems for a particularserver


Replication history Replication problems with a specificdatabase

Under Database Properties. ChooseFile->Database->Properties; or chooseFile->Replication->History

Replication schedule Replication problems for a particularserver


Replication topology maps Replication problems between servers From the Replication tab in the DominoAdministrator

Statistics and events Server statistics and events youspecifically monitor

Configured from the Configuration tab ofthe Domino Administrator; view statisticsfrom the Server- Analysis tab in the DominoAdministrator


Database analysis Database problems From the Files tab in the DominoAdministrator

Administration Requestsdatabase

Administration Process errors From the Servers - Analysis tab in theDomino Administrator

Server commands Various From the Servers - Status tab in the DominoAdministrator

Tool Problems that the tool resolves How to access the tool


Appendix C. Debugging settings

This appendix contains a list of some NOTES.INI settings that can be used forproblem determination. You should be careful not using them in regular times,because generated files can take a lot of place. Some files can grow largeenough to cause disk space shortages. There may also be a performance impacton the system.

Before attempting the debug parameters shown in Table 24, it is imperative thatyou discuss them with a Lotus support specialist beforehand. There may beissues surrounding their use, or special precautions that must be considered priorto debug parameter usage.

Table 24. General debugging settings

Debugging setting Described in Explanations

debug_outfile=(filename).txt Lotus NotesKnowledge BaseTechnote 162400

This allows to put data from the server console to adesignated file. The debug file is placed in the Dominoserver data directory by default. If you want the file to beplaced elsewhere, the full path can be specified before thefilename.Notes:The Domino server must be shut down and restarted tohave these parameters take effect.If no log (text) file is created, consider the following points:- If the Domino server does not come down cleanly, theremay still be some Notes processes left in memory.- Even though the Domino server is restarted, the outfile willnot be written if the server's Notes client is left running.- If the debug_outfile is the last entry in the NOTES.INI file,make sure that there is a hard carriage return after it.

debug_capture_timeout=1debug_show_timeout=1

Lotus NotesKnowledge BaseTechnote 112710

To capture semaphore information, you can add thefollowing appropriate NOTES.INI parameters:

For Domino 4.6.1 (4.6.2 on the AS/400 system) or higher:debug_capture_timeout=1debug_show_timeout=1

The semaphore timeout messages will be written tosemdebug.txt, which is created under the Notes programdirectory.

Other parameters that are not useful include:- debug_sem_timeout=1 should not be used at all. It needsto be removed from any NOTES.INI file. This puts a heavyperformance load on the Domino servers when turning onsemaphore debugging.- debug_outfile= outfile is not useful for semaphoredebugging, and does not capture anything. And even whenit does, the information in it is not useful.

Statements to be defined are:debug_capture_timeout=1debug_show_timeout=1debug_sem_timeout=xx in millisecondsdebug_threadid=1


debug_threadid=1 Lotus NotesKnowledge BaseTechnote 175161

The debug_threadid parameter can be useful for serverpanic debugging.

This parameter is dynamic. When you issue the set configcommand, the process and thread IDs for each messagestart appearing in the server console window and in thedebug_outfile if it was already active when the server wasstarted:set config debug_threadid=1set config debug_outfile=(filename).txt

Typing the above commands at the console adds thefollowing lines:debug_threadid=1debug_outfile=(filename).txt

Similarly, you can issue one of the following server consolecommands to turn off Debug_Threadid dynamically:- set config debug_threadid=0 (leaves it in theNOTES.INI file)- set config debug_threadid= (takes it out of theNOTES.INI file)

debug_tcp_all = 1 Lotus NotesKnowledge BaseTechnote 176401

This parameter can be used to track IP Addresses.

There are currently two ways to find an IP address that hasan open session on a Domino server:1.You can use the console command Show User Debug.2.You can implement the following debug parameters:Debug_Outfile = (filename).txtdebug_tcp_all = 1

There is no NOTES.INI parameter at this time that will trackIP addresses and write them to the log.An enhancement request for this new functionality has beensubmitted to Lotus Quality Engineering. A request has beenmade to add a NOTES.INI parameter to allow tracking IPaddresses in the log file.

debug_directory_assistance=1debug_name_lookup=1

These parameters can be used to troubleshoot directoryassistance problems.These parameters populate the NOTES.LOG with theadditional information. If you want this information to beplaced within a text file, add:debug_outfile=(filename).txtThe file is placed in the Domino server data directory bydefault. If you want the file to be placed elsewhere, the fullpath can be specified before the filename.Note: If these parameters are placed within the NOTES.INIfile for the server, the Domino server must be shut down andrestarted, for these parameters take effect.

These parameters can also be added to the serverdynamically, by typing:set config debug_directory_assistance=1set config debug_name_lookup=1set config debug_outfile=(filename).txt

Debugging setting Described in Explanations


Appendix D. Special notices

This publication is intended to help Domino for AS/400 administrators understandbetter the Domino for AS/400 technical environment and which tools andtechniques can help them in Domino for AS/400 problem determination. Theinformation in this publication is not intended as the specification of anyprogramming interfaces that are provided by IBM AS/400 software, includingOS/400 or Lotus Domino. See the PUBLICATIONS section of the IBMProgramming Announcement for AS/400 software, including OS/400, for moreinformation about what publications are considered to be product documentation.

References in this publication to IBM products, programs or services do not implythat IBM intends to make these available in all countries in which IBM operates.Any reference to an IBM product, program, or service is not intended to state orimply that only IBM's product, program, or service may be used. Any functionallyequivalent program that does not infringe any of IBM's intellectual property rightsmay be used instead of the IBM product, program or service.

Information in this book was developed in conjunction with use of the equipmentspecified, and is limited in application to those specific hardware and softwareproducts and levels.

IBM may have patents or pending patent applications covering subject matter inthis document. The furnishing of this document does not give you any license tothese patents. You can send license inquiries, in writing, to the IBM Director ofLicensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785.

Licensees of this program who wish to have information about it for the purposeof enabling: (i) the exchange of information between independently createdprograms and other programs (including this one) and (ii) the mutual use of theinformation which has been exchanged, should contact IBM Corporation, Dept.600A, Mail Drop 1329, Somers, NY 10589 USA.

Such information may be available, subject to appropriate terms and conditions,including in some cases, payment of a fee.

The information contained in this document has not been submitted to any formalIBM test and is distributed AS IS. The information about non-IBM ("vendor")products in this manual has been supplied by the vendor and IBM assumes noresponsibility for its accuracy or completeness. The use of this information or theimplementation of any of these techniques is a customer responsibility anddepends on the customer's ability to evaluate and integrate them into thecustomer's operational environment. While each item may have been reviewed byIBM for accuracy in a specific situation, there is no guarantee that the same orsimilar results will be obtained elsewhere. Customers attempting to adapt thesetechniques to their own environments do so at their own risk.

Any pointers in this publication to external Web sites are provided forconvenience only and do not in any manner serve as an endorsement of theseWeb sites.

Any performance data contained in this document was determined in a controlledenvironment, and therefore, the results that may be obtained in other operating


environments may vary significantly. Users of this document should verify theapplicable data for their specific environment.

This document contains examples of data and reports used in daily businessoperations. To illustrate them as completely as possible, the examples containthe names of individuals, companies, brands, and products. All of these namesare fictitious and any similarity to the names and addresses used by an actualbusiness enterprise is entirely coincidental.

Reference to PTF numbers that have not been released through the normaldistribution process does not imply general availability. The purpose of includingthese reference numbers is to alert IBM customers to specific information relativeto the implementation of the PTF when it becomes available to each customeraccording to the normal IBM PTF distribution process.

The following terms are trademarks of the International Business MachinesCorporation in the United States and/or other countries:

The following terms are trademarks of the Lotus Development Corporation in theUnited States and/or other countries:

The following terms are trademarks of other companies:

Tivoli, Manage. Anything. Anywhere.,The Power To Manage., Anything.Anywhere.,TME, NetView, Cross-Site, Tivoli Ready, Tivoli Certified, Planet Tivoli,and Tivoli Enterprise are trademarks or registered trademarks of Tivoli SystemsInc., an IBM company, in the United States, other countries, or both. In Denmark,Tivoli is a trademark licensed from Kjøbenhavns Sommer - Tivoli A/S.

C-bus is a trademark of Corollary, Inc. in the United States and/or other countries.

Java and all Java-based trademarks and logos are trademarks or registeredtrademarks of Sun Microsystems, Inc. in the United States and/or other countries.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks ofMicrosoft Corporation in the United States and/or other countries.

PC Direct is a trademark of Ziff Communications Company in the United Statesand/or other countries and is used by IBM Corporation under license.

AS/400 � eNetworkES/9000 GDDMIBM � IMSMQ MQSeriesNetwork Station OS/400 �Operating System/400 OS/2OS/390 RS/6000S/390 SPSystem/390 VisualAgeWebSphere XT

Lotus DominoLotus Notes NotesNotesPump Lotus Enterprise IntegratorLotusScript


ActionMedia, LANDesk, MMX, Pentium and ProShare are trademarks of IntelCorporation in the United States and/or other countries.

UNIX is a registered trademark in the United States and other countries licensedexclusively through The Open Group.

SET and the SET logo are trademarks owned by SET Secure ElectronicTransaction LLC.

Other company, product, and service names may be trademarks or service marksof others.

Appendix D. Special notices 291

Appendix E. Related publications

The publications listed in this section are considered particularly suitable for amore detailed discussion of the topics covered in this redbook.

E.1 IBM Redbooks

For information on ordering these publications, see “How to get IBM Redbooks”on page 297.

• DB2/400 Advanced Database Functions, SG24-4249

• AS/400 Performance Management, SG24-4735

• Lotus Domino for AS/400: Performance, Tuning, and Capacity Planning,SG24-5162

• Lotus Domino for AS/400: Installation, Customization, Administration,SG24-5181

• TCP/IP for AS/400: More Cool Things Than Ever, SG24-5190

• Lotus Domino for AS/400: Integration with Enterprise Applications, SG24-5345

• Lotus Domino for AS/400 R5: Implementation, SG24-5592

• Performance Considerations for Domino, SG24-5602

• AS/400 Mail: Multiple SMTP Domain Names Behind a Firewall, SG24-5643

• Managing AS/400 V4R4 with Operations Navigator, SG24-5646

• AS/400 Performance Explorer Tips and Techniques for V3R6 and V3R7,SG24-4781

This publication can only be accessed online at the redbooks home page at:http://www.redbooks.ibm.com

At the site, click Redbooks Online! Then, in the redbooks search field, enterthe title or publication number of the book you seek, and click Submit Search.When the search results appear, select the appropriate title.

E.2 IBM Redbooks collections

Redbooks are also available on the following CD-ROMs. Click the CD-ROMsbutton at http://www.redbooks.ibm.com/ for information about all the CD-ROMsoffered, updates and formats.

CD-ROM Title Collection KitNumber

System/390 Redbooks Collection SK2T-2177Networking and Systems Management Redbooks Collection SK2T-6022Transaction Processing and Data Management Redbooks Collection SK2T-8038Lotus Redbooks Collection SK2T-8039Tivoli Redbooks Collection SK2T-8044AS/400 Redbooks Collection SK2T-2849Netfinity Hardware and Software Redbooks Collection SK2T-8046RS/6000 Redbooks Collection (BkMgr Format) SK2T-8040RS/6000 Redbooks Collection (PDF Format) SK2T-8043Application Development Redbooks Collection SK2T-8037IBM Enterprise Storage and Systems Management Solutions SK3T-3694



E.3 Other resources

These publications are also relevant as further information sources:

• AS/400 System Handbook, GA19-5486

• DB2 for OS/400 SQL Call Level Interface, SC41-4806

• Basic System Operation, Administration, and Problem Handling, SC41-5206

• OS/400 Backup and Recovery, SC41-5304

• AS/400 Work Management, SC41-5306

• Performance Tools/400 V4R2, SC41-5340

• BEST/1 Capacity Planning Tool for V4R1, SC41-5341

• AnyMail/400 Mail Server Framework Support V4, SC41-5411

• TCP/IP Configuration and Reference, SC41-5420

• AS/400 Sockets Programming, SC41-5422

• Integrated File System Introduction, SC41-5711

• AS/400e Series CL Programming Version 4, SC41-5721

• AS/400e Series CL Reference, SC41-5722

• OS/400 CL Reference V4R4, SC41-5725

This publication can be accessed online at the AS/400 Information Centerlibrary at: http://as400bks.rochester.ibm.com/pubs/html/as400/onlinelib.htm

At the site, select your language and click GO! Select V4R4 and then clickSearch or view all V4R4 books. In the search field that appears, enter thetitle or publication number of the book you seek, and click Find. Then, whenthe search results appear, click the title of the book you want to view ordownload.

E.4 Referenced Web sites

These Web sites are also relevant as further information sources:

• Visit the IBM Domino Web site at: http://www.as400.ibm.com/domino

• The AS/400 Information home page can be accessed at:http://www.as400.ibm.com/infocenter

• The Domino 5 Administration Help (help\help5_admin.nsf) database containsdetailed Domino 5 System Administration information. It can be downloadedat: http://notes.net/notesua.nsf

• An updated version of Domino for AS/400 Release Notes can be accessed atthe Lotus Domino and Notes User Assistance Web site at:http://notes.net/doc

• The Domino 5 Administration Help database is also available in hardcopy asthe R5 Domino Administration Doc Pack. It can be purchased through theLotus Store at: http://www.lotus.com/store

• Visit the Lotus Web page on Domino for AS/400 at:http://www.lotus.com/dominoas400


http://www.as400.ibm.com/domino

• The IBM Global Software Solutions Guide can ce accessed online at the IBMGlobal Solutions Directory at: http://www.ibm.com/software/solutions/isv

• The AS/400 PartnerWorld for Developers online Domino solutions catalog canbe accessed at: http://www.as400.ibm.com/developer/ebiz/index.html

• The AS/400 Technical Studio, which can help you find online workshops,wizards, other tools, and information on a smaller set of topics, can be foundat: http://www.as400.ibm.com/tstudio

• To find information about product fixes, new technology, technical supportservices, and more, visit the AS/400 Worldwide Technical Support home pageat: http://as400service.ibm.com

• The AS/400 Technical Support home page can be accessed at:http://as400service.rochester.ibm.com

• AS/400-specific functions new with a Domino for AS/400 QMR and PTFsrequired for that QMR are described at:http://www.as400.ibm.com/domino/DomSupport.htm

• Information on a new Domino release can be found in customer supporttechnical papers available at: http://support.lotus.com/lshome.nsf

• You can find a list of the group PTFs available for ordering by linking to theWeb site at: http://www.as400service.ibm.com

• AS/400 Preventive Service Planning information Group PTFs can beaccessed at: http://as400service.rochester.ibm.com/s_dir/sline003.NSF/GroupPTFs?OpenView&view=GroupPTFs

• You can find service pack information at the Web site at:http://www.as400.ibm.com/clientaccess

• You can search for solutions to common problems on the Lotus CustomerSupport Web site at: http://www.support.lotus.com

• For Backup Recovery and Media Services implementation instructions, seethe BRMS external Web site at: http://www.as400.ibm.com/service/brms.htm

• For a technical resource for Notes and Domino, visit the Web site at:http://www.notes.net

• Check for the Domino for AS/400 release and its proper QMU level at:http://www.as400.ibm.com/domino/qmulist.htm

• Information about tools for analyzing text log files can be found in an article inIRIS Today, entitled “Logging your Web Site traffic”. You can find this article at:http://www.notes.net/today.nsf

• To confirm that the ISP has the correct MX record for your mail domain, visitthe site at: http://www.samspade.org

• To confirm that the ISP has the correct MX record for your mail domain, visitthe site at: http://www.baboon.ch

• For information about component journaling, refer to the following Web site(this URL is case sensitive):http://www.as400.ibm.com/tstudio/tech_ref/tcp/Indexfr.htm

• You can find information regarding SMTP commands in the RFC821 documentat: http://info.internet.isi.edu/in-notes/rfc/files/rfc821.txt

Appendix E. Related publications 295

• The CLI Frequently Asked Questions document is available on the Web siteat: http://www.as400.ibm.com/db2/clifaq.htm#header_16

• For assistance in selecting the right processor, access the AS/400 WorkloadEstimator at: http://as400service.ibm.com/estimator

• The Lotus “Maximizing Application and Server Performance in Domino” whitepaper is available at:http://www.as400.ibm.com/developer/domino/perform/maxperform.pdf

• The “Evaluating Appropriate Workloads for the AS/400e Dedicated Server forDomino” white paper is available at: http://www.as400.ibm.com/whpapr/dsd.htm

• Domino for AS/400 performance information is available at:http://www.as400.ibm.com/developer/comm/survey.html

• Access the AS/400 Domino Web site at: http://www.as400.ibm.com.domain

• IBM AS/400 Performance Tool Information can be accessed at:http://www.as400.ibm.com/sftsol/pt400


How to get IBM Redbooks

This section explains how both customers and IBM employees can find out about IBM Redbooks, redpieces, andCD-ROMs. A form for ordering books and CD-ROMs by fax or e-mail is also provided.

• Redbooks Web Site http://www.redbooks.ibm.com/

Search for, view, download, or order hardcopy/CD-ROM Redbooks from the Redbooks Web site. Also readredpieces and download additional materials (code samples or diskette/CD-ROM images) from this Redbookssite.

Redpieces are Redbooks in progress; not all Redbooks become redpieces and sometimes just a few chapters willbe published this way. The intent is to get the information out much quicker than the formal publishing processallows.

• E-mail Orders

Send orders by e-mail including information from the IBM Redbooks fax order form to:

• Telephone Orders

• Fax Orders

This information was current at the time of publication, but is continually subject to change. The latest informationmay be found at the Redbooks Web site.

In United StatesOutside North America

e-mail [email protected] information is in the “How to Order” section at this site:http://www.elink.ibmlink.ibm.com/pbl/pbl

United States (toll free)Canada (toll free)Outside North America

1-800-879-27551-800-IBM-4YOUCountry coordinator phone number is in the “How to Order” section atthis site:http://www.elink.ibmlink.ibm.com/pbl/pbl

United States (toll free)CanadaOutside North America

1-800-445-92691-403-267-4455Fax phone number is in the “How to Order” section at this site:http://www.elink.ibmlink.ibm.com/pbl/pbl

IBM employees may register for information on workshops, residencies, and Redbooks by accessing the IBMIntranet Web site at http://w3.itso.ibm.com/ and clicking the ITSO Mailing List button. Look in the Materialsrepository for workshops, presentations, papers, and Web pages developed and written by the ITSO technicalprofessionals; click the Additional Materials button. Employees may access MyNews at http://w3.ibm.com/ forredbook, residency, and workshop announcements.

IBM Intranet for Employees



mailto:[email protected]

http://www.elink.ibmlink.ibm.com/pbl/pbl






http://w3.itso.ibm.com/

http://w3.ibm.com/

IBM Redbooks fax order formPlease send me the following:

We accept American Express, Diners, Eurocard, Master Card, and Visa. Payment by credit card notavailable in all countries. Signature mandatory for credit card payment.

Title Order Number Quantity

First name Last name

Company

Address

City Postal code

Telephone number Telefax number VAT number

Invoice to customer number

Country

Credit card number

Credit card expiration date SignatureCard issued to


Glossary

@DbFunction A fast and easy-to-use read-onlyaccess method to ODBC-compliant DBMSs.

access control list (1) A collection of all access rightsfor one object. (2) A list associated with an object thatidentifies all subjects that can access the object andtheir access rights. For example, a list associated witha file that identifies users who can access the file andtheir access rights to that file.

agent A Lotus Notes routine that automates tasks.Agents generally perform routine Lotus Notes tasks inthe background but can also be started interactively bya user. You can program agents, using Java,LotusScript, or the Lotus Notes formula language.

APAR See Authorized Problem Analysis Report.

API See application programming interface.

application programming interface (API) A set ofcalling conventions that define how a service isinvoked through a software package.

application unit of work A set of actions within anapplication that the designer chooses to regard as anentity. It is up to the designer to decide how, if at all, anapplication should be subdivided into application unitsof work, and whether any application unit of work willconsist of one, or many, logical units of work (LUWs).

authentication (1) Verification of the identity of a useror the user's eligibility to access an object. (2)Verification that a message has not been altered orcorrupted. (3) A process used to verify the user of aninformation system or protected resources.

authorization (1) The right granted to a user tocommunicate with or make use of a computer system.(2) An access right. (3) The process of granting a usereither complete or restricted access to an object,resource, or function.

Authorized Problem Analysis Report (APAR) Arequest for the correction of a defect in a currentrelease of an IBM-supplied program.

call level interface (CLI) A callable API for databaseaccess, which is an alternative to the embedded SQLAPI. In contrast to embedded SQL, CLI does notrequire precompiling or binding by the user. Instead, itprovides a standard set of functions to process SQLstatements and related services at runtime.

certificate A message signed with a public key digitalsignature stating that a specified public key belongs tosomeone or something with a specified name.

certification authority An entity, typically a company,that issues digital certificates to other entities(organizations or individuals) to allow them to provetheir identity to others.

© Copyright IBM Corp. 2000

certification In Lotus Notes, the process of having anauthorized (the certifier) person authenticate theidentity of a user or server.

CLI See call level interface.

client As in client/server computing, the applicationthat makes requests to the server and, often, dealswith the interaction necessary with the user.

client/server computing A form of distributedprocessing, in which the task required to be processedis accomplished by a client portion that requestsservices and a server portion that fulfills thoserequests. The client and server remain transparent toeach other in terms of location and platform. Seeclient, distributed processing, and server.

commit An operation that applies all changes madeduring the current unit of work.

database (1) A collection of interrelated data storedtogether with controlled redundancy according to ascheme to serve one or more applications. (2) All datafiles stored in the system. (3) A set of data storedtogether and managed by a database managementsystem. (4) In Lotus Notes, a group of documents andtheir forms and views, stored under one file.

DECS See Domino Enterprise Connection Services.

distributed processing An application or systemsmodel in which function and data can be distributedacross multiple computing resources connected on alocal area network or wide area network. Seeclient/server computing.

Distributed Relational Database Architecture(DRDA) Architecture proposed by IBM, that definesthe rules, protocols, and the semantics for writingprograms implementing distributed data access. All theplatforms participating in this architecture must complywith these rules and definitions.

DNS See Domain Name System.

document In Lotus Notes, an object containing text,graphics, video, or audio data or any kind of rich textdata.

domain A Domino domain is a collection of Dominoservers and users that share a common DominoDirectory. The primary function is mail routing. Userdomains are determined by the location of theirserver-based mail files.

Domain Name System (DNS) A function to associatenames and addresses on Internet domain servers.

Domino Lotus Domino is an applications andmessaging server with an integrated set of servicesthat enable you to easily create secure, interactivebusiness solutions for the Internet and corporateIntranets.

299

Domino Directory A directory of users, servers,groups, and other entities, for example printers. It is atool that administrators use to manage the Dominosystem, for example, to connect servers for replicationor mail routing, to schedule server tasks, and so on. Inprevious releases of Domino, the Domino Directorywas called the Name and Address Book or the PublicAddress Book.

Domino Enterprise Connection Services (DECS) Aforms-based interface that allows integration toexternal data into Domino applications.

DRDA See Distributed Relational DatabaseArchitecture.

electronic messaging The creation, transfer, storage,and retrieval of text, graphics, image, or voice data byelectronic means.

environment The collective hardware and softwareconfiguration of a system.

form In Lotus Notes, forms are used to author, read,and edit documents. Application developers designforms as templates for document creation or as a wayto view document data. Forms are comprised of otherdesign elements such as text, fields, buttons, andprogramming code.

formula A Lotus Notes programming language thatcontains a set of built-in macros, functions, andcommands.

graphical user interface (GUI) A style of userinterface that replaces the character-based screenwith an all-points-addressable, high-resolutiongraphics screen. Windows display multiple applicationsat the same time and allow user input by means of akeyboard or a pointing device such as a mouse, pen,or trackball.

GUI See graphical user interface.

host (1) In a computer network, a computer providingservices such as computation, database access, andnetwork control functions. (2) The primary orcontrolling computer in a multiple computerinstallation.

Hyper Text Transmission Protocol (HTTP) AnInternet protocol used to transfer files from onecomputer to another.

HTTP See Hyper Text Transmission Protocol.

ICM See Internet Cluster Manager.

IIOP See Internet Inter-ORB Protocol.

IMAP See Internet Message Access Protocol.

index An ordered set of pointers to the data of aDBMS table. An index allows more efficient access torows in a table by creating a direct path to the datathrough the pointers. Each index is stored separatelyfrom the table it refers to and is based on the values ofdata in one or more columns of the table.

initial program load (IPL) The initialization procedurethat causes an operating system to start operation.

Internet Large international, national, and regionalbackbone networks that allow local and campusnetworks and individuals access to global resources.

Internet Cluster Manager (ICM) Domino componentfor failover and load balancing of HTTP clustering.

Internet Inter-ORB Protocol (IIOP) An Internetprotocol that implements CORBA solutions over theWeb. IIOP lets browsers and servers exchangecomplex objects.

Internet Message Access Protocol (IMAP) A mailprotocol that allows clients running it to retrieve mailfrom a host mail server also running the protocol.IMAP is similar to POP3 but has additional features.For example, it supports three modes of mailboxaccess. You can enable IMAP on a Domino server.

Internet Service Provider (ISP) A company thatprovides access to the Internet.

intranet A TCP/IP network that is entirely under thecontrol of a private authority or company. The intranetmay have connections to other independent intranets(which would then be referred to as extranets) or theInternet. It may be fully or partially visible to theoutside, depending on the implementation.

IPL See initial program load.

ISP See Internet Service Provider.

LDAP See Lightweight Directory Access Protocol.

Lightweight Directory Access Protocol (LDAP) Aset of protocols for accessing information directories.LDAP is based on the X.500 protocol, but supportsTCP/IP, which is necessary for Internet access.Because it's a simpler version of X.500, LDAP issometimes called X.500-lite. You can enable LDAP ona Domino server to allow LDAP clients to accessinformation in Domino Directory, for example, e-mailaddresses.

link A defined connection from the NotesPump serverto a specific database product with specific accessparameters, such as user IDs and passwords.

LMBCS See Lotus Multibyte Character Set.

logical unit of work (LUW) (1) An update that durablytransforms a resource from one consistent state toanother consistent state. (2) A sequence of processingactions (for example, database changes) that must becompleted before any of the individual actions can beregarded as committed. When changes are committed(by successful completion of the LUW and recording ofthe syncpoint on the system log), they do not have tobe backed out after a subsequent error within the taskor region.

Lotus A wholly-owned subsidiary of IBM and theowner of the Domino/Notes product.


Lotus Multibyte Character Set (LMBCS) The formatin which Notes stores all internal text, except fileattachments and objects. As a result, any user canedit, forward, and mail documents and work withdatabases in any language. All text leaving the system(displayed, printed, and exported) is translated fromLMBCS to the appropriate character set. LMBCSsupports Western and Eastern European, NorthAmerican, and Asian languages.

LotusScript A version of BASIC that offers not onlystandard capabilities of structured programminglanguages, but a powerful set of language extensionsthat enable object-oriented development within andacross products. Its interface to Notes and Domino isthrough predefined object classes.

LotusScript extension (LSX) A set of specializedLotusScript classes that extends the standard objectmodel to meet a specific need (for example: access toa DBMS or to an ERP).

LSX See LotusScript extension.

LUW See logical unit of work.

mail transfer agent (MTA) A message transfer agent,also called a gateway. A program that translatesmessages between mail formats.

metadata A defining unit of data such as a databasetable, data file, or Notes form.

metalink A special kind of NotesPump link thatprovides preprocessing operations on link data beforeit is transferred within a defined Activity form. Twometalinks, Order and Collapse/Expand, are suppliedwith NotesPump 2.5.

middleware A set of services that allow distributedapplications to interoperate on a local area network orwide area network. It shields the developer or end userfrom the system complexity and enables delivery ofservice requests or responses transparently acrosscomputing resources.

MIME See Multipurpose Internet Mail Extensions.

MTA See mail transfer agent.

Multipurpose Internet Mail Extensions (MIME)Software that allows you to attach nontext files toInternet mail messages. Non-text files includegraphics, spreadsheets, formatted word-processordocuments, and sound files.

mutex Also known as, mutual exclusion object. Aprogram object that is created so that multiple programthreads can take turns sharing the same resource,such as the access to a file.

NAB See Notes Name and Address Book.

Network News Transfer Protocol (NNTP) Protocolthat supports reading newsgroups, posting newarticles, and transferring articles between newsservers. When enabled on a Domino server, it allowsNNTP clients to access newsgroups on the server and

the Domino server to exchange news with other NNTPservers.

NNTP See Network News Transfer Protocol.

Notes Name and Address Book (NAB) Now calledthe Domino Directory or Personal Address Book.

Notes RPC See Notes remote procedure call.

Notes remote procedure call (Notes RPC) This isthe architectural layer of Notes used for allNotes-to-Domino processor unit resources that isdefined to support the communication. You can set upeither the HTTP or the SOCKS proxy to work withRPC.

Notes storage file (NSF) The file extension for aNotes database file (.NSF). A database file containsthe data for an application. Its structure is composed offorms, fields, pages, folders, views, and otherpresentation features, such as a navigator and adatabase icon object. Notes databases are now calledDomino databases.

Notes template file (NTF) A Notes template filecontains the structure for the database, that is, forms,folders, and views, but does not contain documents.Domino Designer comes with a collection of templatesthat you can use to create system and applicationdatabases.

NSF See Notes storage file.

NTF See Notes template file.

ODBC See Open Database Connectivity.

ODS See On Disk Structure.

On Disk Structure (ODS) The format in which a Notesdatabase is physically stored on disk since DominoRelease 5. Also called a database format. InRelease 5, the ODS version of a database is listed onthe Info tab of the Database Properties box.

Open Database Connectivity (ODBC) A Microsoftdeveloped C database API that allows access toDBMSs through a callable SQL, which does notrequire the use of an SQL preprocessor.

POP3 See Post Office Protocol level 3.

Post Office Protocol level 3 (POP3) A mail protocolthat allows clients running it to retrieve mail from ahost mail server also running the protocol. You canenable POP3 on a Domino server.

program temporary fix (PTF) A temporary solution orby-pass for a problem diagnosed by IBM as a resultfrom an error in a current unaltered release of theprogram.

PTF See program temporary fix.

QMR See Quarterly Maintenance Release.

QMU See Quarterly Maintenance Update.

301

Quarterly Maintenance Release (QMR) Anintermediate Lotus product release. QMRs arepublished four times a year.

Quarterly Maintenance Update (QMU) A set ofcorrections that fix specific issues that occur betweenreleases of a Lotus product.

recovery The use of archived copies to reconstructfiles, databases, or complete disk images after theyare lost or destroyed.

replication A Lotus Notes procedure that updates anddistributes copies (replicas) of the same Lotus Notesdatabase that are stored on different servers.

rich text A Lotus Notes field capable of storing avariety of type styles, graphics, and multimedia.

server Any computing resource dedicated toresponding to client requests. Servers can be linked toclients through local area networks or wide areanetworks to perform services, such as printing,database access, fax, and image processing, onbehalf of multiple clients at the same time.

server command Command that lets you perform atask, such as shutting down or restarting a server. Youcan enter commands manually at the console orremote console or use a Program document in theDomino Directory to run commands automatically.

server connection A document in the DominoDirectory or your Personal Address Book that definesa connection to a server. There are four types of serverconnection documents: dialup, network, passthru, andremote LAN.

server program Program that automates anadministration task, such as compacting all databaseson a server. You can schedule server programs to runat a particular time, or you can run them as the needarises.

server task Program provided with the Domino serverthat runs only when specifically loaded. Server tasksserve various purposes. The Administration Process,HHTP Server, and Reporter are just a few examples ofserver tasks.

Secure/MIME (S/MIME) A secure version of the MIMEprotocol that allows users to send encrypted andelectronically signed mail messages, even if usershave different mail programs.

Simple Mail Transfer Protocol (SMTP) The Internet'sstandard host-to-host mail transport protocol. Ittraditionally operates over TCP, using port 25. SMTPdoes not provide any mailbox facility, nor any specialfeatures beyond basic mail transport.

Secure Sockets Layer (SSL). Security protocol forthe Internet and intranets that providescommunications privacy and authentication for Dominoserver tasks that operate over TCP.

S/MIME See Secure/MIME.

socks-enabled Pertaining to TCP/IP software, or to aspecific TCP/IP application that understands the socksprotocol. “Socksified” is a slang term forsocks-enabled.

socks protocol A protocol that enables an applicationin a secure network to communicate through a firewallvia a socks server.

socks server A circuit-level gateway that provides asecure one-way connection through a firewall to serverapplications in a nonsecure network.

SQL See Structured Query Language.

Structured Query Language (SQL) A standard set ofstatements used to manage information stored in adatabase. By using these statements, users can add,delete, or update information in a table, requestinformation through a query, and display the result in areport.

stored procedures A facility to execute proceduresthat are stored at the server. Stored procedures allowan application program to be run in two parts. One partruns on the client and the other on the server.Therefore, one call to a remote database can executea procedure that may represent several repetitiveaccesses to the database. The server procedure at thedatabase runs within the same transaction as theclient application.

table A relational database presents data as acollection of tables. A table consists of data logicallyarranged in columns and rows. The data in the table islogically related, and relationships can be definedbetween tables. Data can be viewed and manipulatedon the basis of mathematical principles and operationscalled relations. Table data is accessed through SQL.

TCP/IP See Transmission Control Protocol/InternetProtocol.

thread Placeholder information associated with asingle use of a program that can handle multipleconcurrent users.

transaction A unit of processing (consisting of one ormore application programs) initiated by a singlerequest. A transaction can require the initiation of oneor more tasks for its execution.

Transmission Control Protocol/Internet Protocol(TCP/IP) A set of communication protocols thatsupport peer-to-peer connectivity functions for bothlocal and wide area networks, and used on theInternet. Originally designed for UNIX, TCP/IPsoftware is available for every major computeroperating system.

transaction processing A style of computing thatsupports interactive applications in which requestssubmitted by users are processed as soon as they arereceived. Results are returned to the requester in arelatively short period of time. A transactionprocessing system supervises the sharing of


resources for processing multiple transactions at thesame time.

unit of work A recoverable sequence of operationsperformed by an application between two points ofconsistency.

URL Uniform Resource Locator.

view (1) In Lotus Notes, views are used to sort, list,filter, and categorize documents. Applicationdevelopers design views with particular selectioncriteria so that they display lists of pertinentdocuments in a given order. (2) In a relational DBMS, aview is an efficient way of representing data withouthaving to maintain it. It can include all or some of thecolumns or rows of the tables on which it is defined.

Web See World Wide Web.

Web browser The client component of the World WideWeb. The Web browser is responsible for formattingand displaying information, interacting with the user,and invoking external viewers for data types that itdoes not support directly. Examples of Web browsersare Mosaic and Netscape.

Web server The server component of the World WideWeb. It is responsible for servicing requests forinformation from Web browsers. The information canbe a file retrieved from the server’s local disk orgenerated by a program called by the server toperform a specific application function.

World Wide Web Also referred to as WWW, W3, orthe Web. An Internet client/server distributedinformation and retrieval system based on HTTP thattransfers hypertext documents across a varied array ofcomputer systems.

workstation (1) A configuration of input and outputequipment at which an operator works. (2) A terminal,a microcomputer, or a network computer. It is usuallyconnected to a server, a mainframe, or a network atwhich a user can perform applications.

X.500 The directory services standard of ITU, ISO, andIEC.

303

Index

Symbols*DOMINO 169*MSF 169@DB functions 231

AACCESS-LOG 143active jobs 132activity log parameters 103, 104Add Environment Variable (ADDENVVAR) command 138ADMIN4.NSF 92, 94AGENT-LOG 144AnyMail/400 169, 188APAR 67Apply Program Temporary Fix (APYPTF) command 75AS/400

control language 8DB2 for AS/400 UDB 5ease of management 4e-business 5integrated file system 8reliability 4security 9single-level storage 4support 16system architecture 4system concepts 6

AS/400 mail component journaling 221AS/400 Operations Navigator 72, 73AS/400 technical support Web page 66asynchronous I/O 28authorities issues 128Authorized Problem Analysis Reports (APARs) database67Authorized Program Analysis Report

see APARautocommit 240, 241automatic recovery 59automatic restart 137automatic server restart 59auxiliary storage pool 95

BBEST/1 252BRMS 97broadcast 50business partner

applications 16services 16

CCall Level Interface

see CLIcall stack 36call stack information 58, 153

© Copyright IBM Corp. 2000

CGI-ERROR-LOG 144Change Owner (CHGOWN) command 93CL 8CLI 243, 244Collect task 116command

Add Environment Variable (ADDENVVAR) 138Apply Program Temporary Fix (APYPTF) 75Change Owner (CHGOWN) 93Configure TCP/IP (CFGTCP) 261Display Domino Console (DSPDOMCSL) 47Display File (DSPF) 58, 117Display Program Temporary Fix (DSPPTF) 73Display Stream File (DSPSTMF) 58, 117Edit File (EDTF) 117End Domino Server (ENDDOMSVR) 40, 121Load Program Temporary Fix (LODPTF) 75Power Down System (PWRDWNSYS) 122Remove Program Temporary Fix (RMVPTF) 78, 80Start DNS Query (NSLOOKUP) 187, 270Start Domino Server (STRDOMSVR) 40Start Mail Server Framework (STRMSF) 189Submit Domino Command (SBMDOMCMD) 47Work with Active Jobs (WRKACTJOB) 256, 258Work with Disk Status (WRKDSKSTS) 256Work with Domino Console (WRKDOMCSL) 47Work with Environment Variable (WRKENVVAR) 138Work with System Status (WRKSYSSTS) 256Work with TCP/IP Network Status (NETSTAT) 208,265

communications trace 218COMPACT 86, 87Configure TCP/IP (CFGTCP) 261connection trace from Notes client 273console 47, 48, 58

dump of console entries 137control language 8corequisite PTFs 74cumulative PTF 80cumulative PTF package 71, 74

DData Description Specifications (DDS) 229database instance ID 94database replicator 21DB2 Call Level Interface 229, 230, 236, 240DB2 for AS/400 UDB 5DB2 UDB for AS/400 70, 229DBIID 94debug parameters 103, 104, 136, 203, 204debug settings 137debugging settings 287DECS 240directory synchronization 41, 128Display Domino Console (DSPDOMCSL) command 47Display File (DSPF) command 58, 117Display Program Temporary Fix (DSPPTF) 73

305

Display Stream File (DSPSTMF) command 58, 117DNS 26documentation

Domino 12Domino for AS/400 13

Domain Name Server 26Domain Name System (DNS) 263Domino

documentation 12running on the AS/400 system 5server family 3

Domino administration toolCOMPACT 86, 87FIXUP 81, 87, 100UPDALL 81, 87

Domino Administrator 102, 272Domino architecture 19

asynchronous I/O 28database file structure 23database replicator 21Domino database 23indexer 21IOCP 29kernel design 19mail router 21on disk structure (ODS) 30replicator 24shared memory management 25structure of a Domino server 19thread pooling 27threads 22transaction logging 30unified buffer manager 29usage of semaphores 23

Domino console 47Domino database

index 25maximum database size limit 30

Domino Directory 25Domino Enterprise Connections Services

see DECSDomino for AS/400

directory synchronization 41documentation 13overview 3partitioned servers 4releases 11support structure 16technical support 10

Domino for AS/400 support information Web page 65Domino log database 56Domino plug-in 7, 72Domino release 130Domino server command

broadcast 50drop 50load 50quit 50replicate 51set configuration 51

show port 52show stat 52TELL 94tell 55

Domino server environment variables 45Domino Web server log database 56, 58, 142Domino Web server log text files 56, 58, 143DOMINO_SERVERS file 89, 91, 118, 120DOMLOG.NSF 56, 58, 142, 146, 148drop 50dump files 39, 58Duplex setting 264dynamic thread logging 145

Ee-business 3, 5Edit File (EDTF) command 117e-mail monitoring 222

mail probes 223e-mail troubleshooting 169End Domino Server (ENDDOMSVR) command 40, 121environment variables 45, 131ERROR-LOG 144Event task 116exit point 171

Ffirewall 183FIXUP 81, 87, 100

options 82

Ggroup PTF 71

Hhot fix 69, 79, 80, 127HTTP crash 58, 141HTTP hang 147HTTP settings 141HTTP Tell commands 149HTTPD.CNF 120

II/O completion ports 29indexer 21integrated file system 8Inter Process Communication Status 43Internet PTF download 75Internet Service Provider 183invocation stack trace 130IOCP 29IP address 26IP packet filtering 263iPTF 75ISpy 102


JJava 70JDBC 242job call stack 36job log 34, 58, 131

LLEI 240, 242LMBCS 26load 50Load Program Temporary Fix (LODPTF) 75LOG.NSF 56, 58, 81, 102, 128, 148logical files 229looking at damaged semaphores 128looking at damaged shared memory 128Lotus Domino Connector APIs 231Lotus Enterprise Integrator

see LEILotus Multi-Byte Character Set 26Lotus Notes Knowledge base 61Lotus Passport Advantage 16Lotus support 12LOTUS_SERVERS file 89, 91, 119LotusScript Data Object

see LS:DOLS:DO 240

Mmail probes 223mail router 21, 201Mail Server Framework

see MSFMAIL.BOX 81, 102Management Central 8Maximum Frame (MAXFRAME) 262Message Transfer Agent

see MTAmessages 39, 58mirroring 254monitoring e-mail 222monitoring tasks

Collect task 116Event task 116

monitoring the Domino server 114MSF 169, 175, 193, 196

MSF dump 217MTA 174, 175, 193MTU (Maximum Transmission Unit) 261mutex 135

NName and Address Book 25NAMES.NSF 81, 87, 92, 94NETSTAT 208, 265NOS 19Notes C API 19Notes Object Services 19Notes Remote Procedure Call 19, 25

NOTES.INI file 102, 104, 117, 128, 136, 148, 203, 205debugging settings 287editing the NOTES.INI file 107

NOTES_AS400_CONSOLE_ENTRIES 46NOTES_AS400_STSDEBUG 46NotesConnect 210, 274NOTESHST stream file 47NotesPump 240NRPC 19, 25NSD 57NSD diagnostic tool 57NSD file 59, 127, 141, 142, 146

listing NSD files 57reading an NSD file 58

NSF_BUFFER_POOL_SIZE 136NSLOOKUP 209, 270

OODSOfficeVision/400 169, 193on disk structure

see ODSordering PTFs 74OS/400 70OS/400 release 130ownership issues 128

PPerformance Tools for AS/400 256physical files 229PING 267plug-in 7, 72POP3 172, 188, 191, 193Post Office Protocol

see POP3Power Down System (PWRDWNSYS) command 122prerequisite PTFs 74preventive service planning 72problem determination

problem determination phase 17problem fixing 17process 16

product news 61program temporary fix

see PTFProgram Temporary Fixes (PTFs) database 67PTF 67, 69, 70, 127

Internet download 75ordering 74

QQHST 41, 128QMR 69

how to obtain a QMR 77installing a QMR 77

QMSF 198QMU 69, 80, 127

how to obtain 78

307

QNNINSTS 94, 127QPFRADJ 258QSNADS 193QSQSRVR 245QSYS 128QSYS2 128QSYSOPR message queue 39, 127QSYSWRK 193Quarterly Maintenance Release

see QMRQuarterly Maintenance Update

see QMUquit 50

RRAID-5 254REFERER-LOG 144release notes 61release of the OS/400 operating system 127Remove Program Temporary Fix (RMVPTF) 78, 80replicate 51replicator 24REQ###.LOG 145, 146, 148RIP file 57router 21

Ssecondary threads 135security

access to DB2 UDB for AS/400 9access to Domino databases 9

semaphores 23, 42, 135, 153, 162finding a damaged semaphore 43looking at damaged semaphores 58semaphore timeouts 162

server commands 48server crash 59, 127

helping to diagnose the problem 127service pack 72set configuration 51shared memory 25, 42, 43, 135

finding damaged shared memory 43looking at damaged shared memory 58

show port 52show stat 52SMTP 169, 177, 188, 190, 191, 193, 194, 198, 202

MSF 178SMTP Flight Recorders 210SMTP/MIME 174SNA over TCP/IP 263SNADS 193snap-in 171Software Problem Report

See SPRspooled files 38, 58, 127SPR 62SQL Call Level Interface 230SQL indexes 229SQL tables 229

SQL views 229SQLJ 242Start DNS Query (NSLOOKUP) 187, 270Start Domino Server (STRDOMSVR) 40Start Mail Server Framework (STRMSF) 189STATREP.NSF 128, 148, 255Submit Domino Command (SBMDOMCMD) 47superseded PTF 73support for the AS/400 systen 16support structure for Domino for AS/400 16system history log 41

TTCP host name 26TCP/IP 26, 70TCP/IP buffer size 262TELL 94tell 55TELNET 206thread pooling 27threads 22, 135, 153, 258

displaying job threads 36TRACERT 271transaction logging 30, 94

how to set up 95Type of Service (TOS) 262

UUNICODE 120unified buffer manager 29UPDALL 81, 87

options 84UPDATE 83Update task 257upgrading OS/400 78, 80user error 69

Vviruses 8VLIC Trace 58

WWebQueryOpen 148WebQuerySave 148Work with Active Jobs (WRKACTJOB) command 256,258Work with Disk Status (WRKDSKSTS) command 256Work with Domino Console (WRKDOMCSL) 47Work with Environment Variable (WRKENVVAR) 138Work with System Status (WRKSYSSTS) command 256Work with TCP/IP Network Status (NETSTAT) 208, 265Workload Estimator 252, 254, 296

XX.500 25



IBM Redbooks review

Your feedback is valued by the Redbook authors. In particular we are interested in situations where a Redbook"made the difference" in a task or problem you encountered. Using one of the following methods, please review theRedbook, addressing value, subject matter, structure, depth and quality as appropriate.

• Use the online Contact us review redbook form found at http://www.redbooks.ibm.com/• Fax this form to: USA International Access Code + 1 914 432 8264• Send your comments in an Internet note to [email protected]

Document NumberRedbook Title

SG24-6051-00Lotus Domino for AS/400: Problem Determination Guide

Review

What other subjects would youlike to see IBM Redbooksaddress?

Please rate your overallsatisfaction:

O Very Good O Good O Average O Poor

Please identify yourself asbelonging to one of the followinggroups:

O CustomerO Business PartnerO Solution DeveloperO IBM, Lotus or Tivoli EmployeeO None of the above

Your email address:The data you provide here may beused to provide you with informationfrom IBM or our business partnersabout our products, services oractivities.

O Please do not use the information collected here for future marketing orpromotional contacts or other communications beyond the scope of thistransaction.

Questions about IBM’s privacypolicy?

The following link explains how we protect your personal information.http://www.ibm.com/privacy/yourprivacy/



http://www.ibm.com/privacy/yourprivacy/

SG24-6051-00

Printed in the U.S.A.

Lotus

Dom

inofor

AS/400:

Problem

Determ

inationG

uideSG

24-6051-00

®

Lotus Domino for AS/400: Problem Determination Guide

Documents

Transcript of Lotus Domino for AS/400: Problem Determination Guide