T5S3-ibm-domino-logging.pdf - AdminCamp '22
-
Upload
khangminh22 -
Category
Documents
-
view
3 -
download
0
Transcript of T5S3-ibm-domino-logging.pdf - AdminCamp '22
Head of Product at Ytria
IBM Notes Administration & Development for
various enterprise IBM customers
Certified Advanced Domino Developer & Admin
and Security Professional v7+
Speaker at ICS / ESS technology conferences around the globe
IBM Champion for Collaboration Solutions 2014 & 2015
ca.linkedin.com/in/benedekmenesi
Speaker: Ben Menesi
@BenMenesi
Founder, CEO & CTO at Ytria
15+ years IBM Notes & API development
Speaker at ICS conferences & published author
IBM Design Partner
ca.linkedin.com/pub/eric-houvenaghel/0/37/941
Co-AuthorEric Houvenaghel
1.• Events take place
2. • Events are logged
3.• Some events captured
4• Few events addressed
5
• Lots of events missed, angry users on the phone
Everything about LOG.NSF
– Understanding different log events
– Extracting additional data
– Log Retention, Searching the log and more
User Activity Logging
– Understanding what user activities are logged
– Utilizing Recorded user activity
Replication Logs
– Replication History tips & tricks
– Understanding what / where / why
– Replication Triangulation
Agenda
LOG.NSF
What is (or isn’t) logged in log.nsf?
How can we extract more data?
Best Practices for retention, searching and more
Almost everything from your Domino server (HTTP Web Server is another topic!)
– Generally 90%+ of what’s in your Domino console will be in here
• Except for Traveler and some DAOS command logs
Log.nsfWhat is being logged?
Log.nsf
replica
amgrrouter
MAIN Log.nsf setting in your server’s Notes.ini file
Logfilename: typically log.nsf
Log_option: 1 = Log to the console, 2 = Force DB fixup when opening log file, 4 = Full document scan
Days: # of days logs will be kept*
Size: Size of log text in event documents*
Days2: Optional # of days to keep activity trends data documents
Log.nsfLog file structure
LOG=logfilename, log_option, cant_touch_this, days, size, days2
Understanding your log views
Log.nsfLog file structure
SELECT Form=“Events”
SELECT Form=“Mail Routing”
SELECT Form=“Replication”
SELECT Form=“Security”
SELECT Form=“NNTPEvent”
SELECT Form=“PassthruConnection”
Server tasks simultaneously log into various log.nsf documents
Miscellaneous Events (form=Events)
– Size & content controlled by the Log= server notes.ini parameter (later)
Mail Routing Events (form=Mail Routing)
– Mainly updated by the SMTP Server and router tasks
Security Events (form=Security)
– E.g. ID Vault messages & errors
Replication Events (form=Replication)
– Replica & CLREPL tasks
Log.nsfLog file structure
Understanding your log views
Log.nsfLog file structure
Select Form=“Session”
Select Form=“Session”
SELECT Form=“Activity”
SELECT Form=“Activity” | Form=“Size”
User Session Activity (form=Session)
– Sessions this server had with users or other servers
Activity Data (form=Activity)
– Populated by the nightly Statistics Log task
Log.nsfLog file structure
Server tasks can log events to multiple types of documents!
– But no duplicate logs: different information!
– Example: Server 1 replicator opens session to Server 2 as per Connection Document settings
Log.nsfLog file structure
Misc.
Replication
Miscellaneous events: what do we see?
– Some high level info (Server, Start & Finish Time) and
– UI only shows ONE text type item with limited information
Log.nsf1.) Miscellaneous events
What the EventsR6 item tells us us
– Used to be a Rich Text item Prior to R6 (whew!)
– @If(@IsAvailable(Eventlist);EventList;@IsAvailable(Eventslist);EventsList;Events)
– EventsList item: only contains the Event time & description for each event
What is an event?
Log.nsf1.) Miscellaneous events
Time Status Code Severity Type
Addin Name Target Server Target DB Target User
Miscellaneous events – event attributes logged in multi-value Textitems
Log.nsf1.) Miscellaneous events
EventListTime &
Description
EventTime
EventSeverity
EventStatus
EventType
Event 1
Time
Severity
HEX Status
Type
Time &
Description
Event 2
Time
Severity
HEX Status
Type
Time &
Description
Event 3
Time
Severity
HEX Status
Type
Time &
Description
Event x
Time
Severity
HEX Status
Type
Miscellaneous events – EventSeverity
– Severities are recorded via numbers 0 – 5 where
• 0: Unknown
• 1: Fatal
• 2: Failure
• 3: Warning High
• 4: Warning Low
• 5: Normal
Log.nsf1.) Miscellaneous events
Miscellaneous events – EventType
– Multiple type identifiers in use
• Best not to mess with this stuff
Log.nsf1.) Miscellaneous events
Can we get more data?
– We can use formulas and the list of Severities, Events & Status Codes to get a better idea of each Misc. log document
Total # of events / document [Eventlist item]
– @Elements(Eventlist)
Total # of unique events [EventStatus item]
– @Elements(@Unique(EventStatus))
Highest severity / document [EventSeverity item]
– _sev:=@If(@Sort(@Unique(@Text(EventSeverity));[Ascending])[1]="0";@Sort(@Unique(@Text(EventSeverity));[Ascending])[2];@Sort(@Unique(@Text(EventSeverity));[Ascending])[1]);
Log.nsf1.) Miscellaneous events
Customization options (Cont’d)
Once we have the _sev variable, use icons to display (from DDM resources)
– @If(_sev="0";"iconNormal";_sev="1";"iconFatal";_sev="2";"iconFailure";_sev="3";"iconHigh";_sev="4";"iconLow";_sev="5";"iconNormal";"")+".gif“
# of Fatal severities / document [EventSeverity item]
– @Elements(@Trim(@Replace(@Text(EventSeverity);"5":"4":"3":"2":"0";"")))
# of Failure severities [EventSeverity item]
– @Elements(@Trim(@Replace(@Text(EventSeverity);"5":"4":"3":“1":"0";"")))
Log.nsf1.) Miscellaneous events
# of High Warning severities / document
– @Elements(@Trim(@Replace(@Text(EventSeverity);"5":"4":"1":"2":"0";"")))
# of Low Warning severities / document
– @Elements(@Trim(@Replace(@Text(EventSeverity);"5":"1":"3":"2":"0";"")))
# of Normal severities / document
– @Elements(@Trim(@Replace(@Text(EventSeverity);"1":"4":"3":"2":"0";"")))
# of Unknown severities / document
– @Elements(@Trim(@Replace(@Text(EventSeverity);"5":"4":"3":"2":"1";"")))
Log.nsf1.) Miscellaneous events
Customization options: this is what you’ll get
– Tip: we can make the Highest and severity columns sortable but careful with your indexes!
Log.nsf1.) Miscellaneous events
Mail Routing Events
– Same Structure as Miscellaneous events
– Incrementally fills documents (considering 40KB / Log= limit)
Log.nsf2.) Mail Routing Events
Time EventList EventStatus EventSeverity
Is there any additional data we can mine?
– Use same custom views (Severities, unique events, etc…)
• Tip: Messages can be traced based on the last 8 characters of their Universal ID
• Extract list of Messages transferred per Mail Routing Log, or their (unique) numbers.
Log.nsf2.) Mail Routing Events
Unique # of Messages transferred [EventList item]
– Could be used as a sortable column formula
• @Elements(@Unique(@Explode(@Implode(@Trim(@Word(@ReplaceSubstring(EventList;"Router: Message ":"Router: No messages":" transferred to ":"NOT transferred ":" delivered to ";"§§§":"§§§":"§§§":"§§§":"§§§");"§§§";2));", ");", ")))
Log.nsf2.) Mail Routing Events
List of unique message codes [EventList item]
– Could be used as a categorized column formula – help finding emails
• @Unique(@Explode(@Implode(@Trim(@Word(@ReplaceSubstring(EventList;"Router: Message ":"Router: No messages":" transferred to ":"NOT transferred ":" delivered to ";"§§§":"§§§":"§§§":"§§§":"§§§");"§§§";2));", ");", "))
Log.nsf2.) Mail Routing Events
Security Event documents
– Same Structure as Miscellaneous events
– Incrementally fills documents (considering 40KB / Log= limit)
– Typically you won’t even need a special view to understand unless you have a very high volume.
Log.nsf3.) Security Events
Replication Events <> Miscellaneous events
– Important: Logs documents PER SESSION
– Only Replication Sessions where the current server did the work!
• Search all participating server log files to get the big picture!
– What is stored in those replication events?
Log.nsf4.) Replication Events
InitiatedBy Server SourceServer
Body BytesIn BytesOut Pathname
EventList EventSeverity EventStatus EventType
What additional data can we extract and use?
– EventList & EventSeverity items ONLY created on error
– # of entries in the Pathname item tells us how many events
– Body item contains: Access, Additions, Updates, Deletes, Kbytes sent & received for each DB
• Body item is NOT multi-value (can’t really work with formulas)
• Body item is NOT part of the summary if more than 1 event is logged = can NOT be displayed in views
• Cluster Replication events are NOT properly logged unless you use the RTR_Logging parameter
• http://www-01.ibm.com/support/docview.wss?uid=swg21214739
Log.nsf4.) Replication Events
Interesting stuff: even when DB1 only has to SEND data to DB2, we can see data being received by DB1
– BytesIN & BytesOUT: NOT reliable:
Log.nsf4.) Replication Events
Creating a custom replication view
Replicator / Cluster Replicator? [Initiatedby item]
Number of events [Pathname item]
– @Elements(Pathname)
Any Errors? [EventSeverity item]
– @If(@Elements(EventSeverity)>0;150;"")
Direction [Body item]
– _bl1:=@Contains(Body;"PULL");
– _bl2:=@Contains(Body;"PUSH");
– @If(_bl1=1&_bl2=1;"PULL-PUSH";_bl1=1&_bl2=0;"PULL";_bl2=1&_bl1=0;"PUSH";"-")
Log.nsf4.) Replication Events
Usage Session logs – what do we (not) see?
– IMPORTANT: Logging on a /USER/SESSION base!
– Contains a TON of raw data
Log.nsf5.) Usage Session Documents
What is an event?
Log.nsf5.) Usage Session Documents
Database Reads Writes Transactions
Bytes Read Bytes Written DB Open time
UserName Pathname Reads Writes
Transactions SentFromServer SentToServer Body
What additional data can we extract and use?
– PathName item (multi-value) contains list of databases accessed
• @Elements(PathName) gives us the # of DBs accessed for each session
– Rearrange columns in the Usage \ By User view
Log.nsf5.) Usage Session Documents
Searching your log file is painful
– Reason: you’ll be presented with any matching documents but you won’t be able to read between the lines
– Looks familiar?
– … and the sad part: 6 results would actually make us happy (RIGHT?)
Log.nsfHow to search?
Option #1: search using the Admin client
– TONS of parameters to specify – slow setup. Better for periodical search requirements
Log.nsfHow to search?
Option #1: search using the Admin client (Cont’d)
– Tip: do NOT use the Event Type tab!
• You can miss results because events are logged across types
Log.nsfHow to search?
Option #1: search using the Admin client (Cont’d)
– Tip: You can save queries in domadmin.nsf to reuse later
– BEWARE: Search results populate new documents!
• Avg. 180 events / document: single search with 1800 matches creates 10 documents
• Results can not be analyzed nor copied and are stored across multiple documents
– Conclusion: Useful for very specific queries that you may need periodically
Log.nsfHow to search?
Option #2: A quicker DIY solution (less customizable)
– Advantages: reads matches from multiple log documents and saves & displays them in one text file
– http://searchdomino.techtarget.com/tip/Easily-find-a-string-in-a-Lotus-Domino-server-log
Log.nsfHow to search?
Option #3: Pro tip for data-range search (Kudos to Kim Greene)
– Select log documents and Actions \ Forward
• Aggregates EventList item contents in one document
• Use CTRL+F to search
Log.nsfHow to search?
Option #4 Great open source log parser via OpenNTF from JakobMajkilde
– Installed on server, access to Console Logs, Trace Logs, Log.nsf and more with a nice Xpages interface
– http://openntf.org/main.nsf/project.xsp?r=project/XPages%20Log%20File%20Reader
Log.nsfHow to search?
Log deletions managed by parameter defined in log= server notes.ini parameter
– Deletions made at once, all entries older than 7 days removed
– Add to that: Deletion Stub Purge interval
• Default 90 days could result in =< 120 days old deletion stubs
• Deletions happen every 1/3rd of the time defined here
Log.nsfLog file retention
Known problem in previous Domino versions: log.nsf continues to grow and ignores log= settings
– Workaround: use “Remove documents not modified in the last (days)
• Beware: this field also defines Deletion Stub Purging time
• Tip: Deleted documents are removed WITHOUT creation of deletion stubs
Log.nsfLog file retention
Recommendation
– Use the “Remove documents note modified in the last (days)” setting
– Use console logs to keep old log data
Log.nsfLog file retention
Log_AgentManager
– 0 for no logging, 1 for partial & successful agent execution events, 2 for successful events only
Log_Replication
– 1 for logging when DB replicates, 2 for summary info about each DB, 3 for detailed info about each replicated note
• Great for debugging!
RTR_Logging
– 1 (Default), 4 for Logging replications (attempted & performed)
• Cluster replication will be logged
Log.nsfA few notes.ini parameters in the back pocket
Mail_Log_To_MiscEvents
– 0/1 whether to log mail events into Misc. event documents
• Tip: MailLogToEventsOnly=1 to NOT show router messages in the Domino Console
SMTPClientDebug=1
– Use temporarily to debug outbound mail problems
• Tip: don’t get confused by client, this one’s for your server!
HTTPLogUnauthorized=1
– Logs 401 HTTP errors in server console AND Misc. log events
• HTTP Users attempting to access resources that aren’t available and failed user authentication requests
• This is GREAT: brings a bit of web server logs into your log.nsf
Log.nsfA few notes.ini parameters in the back pocket
Log_Console=2
– Logs all console commands even if prefixed with ! (By default !sh task won’t be logged!)
LOG_DisableTXNLogging=1
– 0/1 Take your Log.nsf out of DBs to be transaction logged
• Also clubusy.nsf and mail.box!
No_Force_Activity_Logging
– 0/1 Controls whether the statlog task automatically enables activity recording for all DBs (Default = 0)
• Tip even if disabled activity is being recorded in the Log.nsf Usage views
Log.nsfA few notes.ini parameters in the back pocket
USAGE ACTIVITY
What are our most used databases?
How do we know what a given user has been up to?
How do we get user activity for multiple databases?
User Activity can be a *VERY* rich source of data
– Enable in the database properties window
• Tip: No_Force_Activity_Logging=0 (Default value) allows on ALL dbs
• Tip: Set to confidential to prevent users with lower than Designer access from accessing it
Database User ActivityRecorded User Activity
Did you upgrade your database On Disk Structures?
– I’m not proud of it
– Tip: use Compact –REPLICA for upgrading system database ODS’s without downtime!
Database User ActivityBefore we go any further
Prior to ODS 48
– Entry size allocated 44 bytes, Object size 61600 bytes (max 1400 entries)
ODS 48+ (undocumented improvement)
– Entry size allocated 92 bytes, Object size 128800 bytes (max 1400 entries)
Curious: only two additional counters added, yet ODS 48+ entry is over 2x the size of those prior to ODS48: Why?
Database User ActivityUser Activity – What is really captured?
Date &
TimeReads Adds Updates Deletes User
Date &
TimeReads Writes User
Does Notes only capture as much as it displays?
– Nope
Database User ActivityUser Activity – What is really captured?
Reads Adds Updates Deletes
Data
Non
Data
Σ
Data
Non
Data
Σ
Data
Non
Data
Σ
Data
Non
Data
Σ
Differentiating between Data (as in: Document class notes) and Non-Data (as in: everything else such as Designs, ACLs, etc…) can help us look for more specific scenarios:
– Databases where no DOCUMENT class notes were read for some time
– Databases where no DOCUMENT class notes were added or updated since awhile
– Users who’ve deleted Designs or ACLs
– And more.
Database User ActivityUser Activity – Data & Non-Data
Makes no difference between users and servers
– Workaround: use a tool to parse the data and input exceptions
Only maintains 1400 entries
– Wait, this doesn’t need to be a problem!
Still answers questions like “What databases aren’t used on a regular basis?”
– Databases that contain 1400 entries are properly used apps
• @Date(FirstEntry) - @Date(LastEntry) tells us how frequently used
– Databases with lower than 1400 entries indicate they aren’t used that much
Database User ActivityUser Activity - Downside
How do we get the user activity for multiple databases
– Third party tools like http://www.agecom.com.au/useractivity
– …or Ytria’s databaseEZ
– …or we can implement it ourselves using the NotesUserActivity class:
• Uses the W32_NSFDbGetUserActivity API call (restrictive: no Data vs. Non-Data)
http://www.bananahome.com/ldd/sandbox.nsf/ByDate/c12a2fd2142758b68525688d00708397?OpenDocument
Database User ActivityUser Activity – How to get it?
Set via Log_Replication=value
– 0 Do not log replication events
– 1 Log that DB is replicating (Default value)
– 2 Log summary info about each DB
– 3 Log info about each replicated document (both design and doc class)
– 4 Log info about each replicated field
Note: only impacts logging of replication events performed by the current server!
Replication Logs Replication verbosity logs
Keeps track of what the current database replicated with
– Tip1: Local <> Server replication leaves NO TRACE in server replica’s replication history!
– Tip2: When there’s nothing to replicate, no replication history entry created
Replication Logs Replication History
Feature aiming to optimize performance introduced in Domino 7
– Result: Replication History Entries indicating replication between A & C which never happened
Replication Logs Replication Triangulation
A
B
C
A – B
A – B
B – C
B – C
A – C
Idea: to prevent complete recalculation when replicating with an unknown server
– Results show 400% CPU increase in some cases (1700 servers)
– Disable (server side):
• NSF_REPLHIST_NO_TRI=1
• REPL_NO_WS_TRI_HIST=1
• REPL_NO_REMOTE_TRI_HIST=1
– Gotcha: need to clear replication history entries for all impacted DBs!
– Disable (client side):
• NSF_REPLHIST_NO_TRI=1 [No triangulated entries read]
• REPL_NO_WS_TRI_HIST=1 [No triangulated entries written]
Replication Logs Replication Triangulation
Purge Interval Replication Control
– New(ish) awesome option for preventing old documents from coming back (Introduced in Domino 8.5.3)
• Set via Replication Options > Space Savers
– DEBUG_REPL_PIRC=1 gives you date and summary info about documents not allowed to replicate
– DEBUG_REPL_PIRC=2 or greater provides more details about documents blocked by PIRC
Replication Logs PIRC logging
Please fill out your evaluation forms
– Your feedback is very important!
Feel free to get in touch!
– Contact - Ben Menesi
ThanksThank you for attending!
ca.linkedin.com/in/benedekmenesi
@BenMenesi
Engage Online
SocialBiz User Group socialbizug.org
– Join the epicenter of Notes and Collaboration user groups
Social Business Insights blog ibm.com/blogs/socialbusiness
– Read and engage with our bloggers
Follow us on Twitter
– @IBMConnect and @IBMSocialBiz
LinkedIn http://bit.ly/SBComm
– Participate in the IBM Social Business group on LinkedIn
Facebook https://www.facebook.com/IBMConnected
– Like IBM Social Business on Facebook
Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from
IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial
publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS"
WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION,
INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted
according to the terms and conditions of the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have
used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which
IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and
discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific
situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any
relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide
legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested
those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-
party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual
property right.
IBM, the IBM logo, ibm.com, BrassRing®, Connections™, Domino®, Global Business Services®, Global Technology Services®, SmartCloud®, Social Business®, Kenexa®, Notes®,
PartnerWorld®, Prove It!®, PureSystems®, Sametime®, Verse™, Watson™, WebSphere®, Worklight®, are trademarks of International Business Machines Corporation, registered in
many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright
and trademark information" at: www.ibm.com/legal/copytrade.shtml.
Notices and Disclaimers