Necessary Supplied Packages - Springer

Necessary Supplied Packages

In this section of the book, we will cover the supplied database packages that, in my opinion, everyone needs to be aware of. Each of these packages is documented in the Oracle document entitled Oracle8i Supplied PL/SQL Packages Reference. The supplied documentation typically shows the entry points (externalized procedures and functions) of the supplied package, and gives you an overview of each function/procedure's usage. In this section we will describe in more detail when you might (or might not) choose to use such a package. We will not go into depth on each and every procedure in every package. Rather, we will discuss the most commonly used package entry points, and show how they are used. For a comprehensive list of all procedures available in a given package, along with all possible parameters, I will refer you to the aforementioned document.

This appendix will serve as a 'jumping off' point for using these supplied packages. After you are done with it, you will have a good feel for the intended use of many of them. We do not cover every supplied package in this section. This does not imply that they are not useful, just that their use falls outside the scope of typical development. We will explore the packages that applications will employ in most cases.

The packages we will cover are:

❑ DBMS_ALERT and DBMS_PIPE – Inter-process communication facilities in the database. DBMS_ALERT can be used to signal all interested sessions that some event has taken place. DBMS_PIPE allows two sessions to 'talk' to each other, much like a TCP/IP socket.

❑ DBMS_APPLICATION_INFO – Allows an application to register useful information in the V$tables. Extremely useful to monitor what your stored procedure is doing, and register other information.

❑ DBMS_JAVA – A PL/SQL package useful for working with Java stored procedures.

Appendix A

1028

❑ DBMS_JOB – A database job scheduler. Used when you have that stored procedure you want to execute every night at 2am, or when you just want to run something in the background.

❑ DBMS_LOB – For working with Large OBjects (LOBs) in the database.

❑ DBMS_LOCK – To create your own user-defined locks, separate and distinct from Oracle's row row or table level locks.

❑ DBMS_LOGMNR – To review and analyze the contents of your online redo log files

❑ DBMS_OBFUSCATION_TOOLKIT – Provides data encryption in the database.

❑ DBMS_OUTPUT – Provides simple screen I/O capabilities for PL/SQL in SQL*PLUS and SVRMGRL.

❑ DBMS_PROFILER – A PL/SQL source code profiler built into the database.

❑ DBMS_UTILITY – A 'hodge-podge' collection of useful procedures.

❑ UTL_FILE – Provides text file I/O for PL/SQL. Allows PL/SQL to read and write text files on the server.

❑ UTL_HTTP – Provides access to the HTTP (Hyper Text Transfer Protocol) protocol from within PL/SQL. Allows PL/SQL to 'grab' web pages.

❑ UTL_RAW – Provides conversion between the RAW and VARCHAR2 types. Extremely useful when working with TCP/IP, BLOBs and BFILEs, and encryption.

❑ UTL_SMTP – Provides access to the SMTP (Simple Mail Transfer Protocol) from within PL/SQL. Specifically, it allows you to send an e-mail from PL/SQL.

❑ UTL_TCP –Provide TCP/IP socket abilities for PL/SQL. Allows PL/SQL to open a connection to any TCP/IP service.

Why Use the Supplied Packages? The reasoning behind using the supplied packages is simple, it is much easier and more maintainable to develop using supplied functionality then it is to build your own. If Oracle supplies a package for doing something (for example, data encryption) it would not be productive to write your own. Often, I find people implementing functionality that they did not know already existed in the database, purely out of ignorance. Knowing what tools you have available to you will make your life much easier.

About The Supplied Packages The supplied packages from Oracle all begin with either DBMS_ or UTL_. Historically, packages that were created by Server Technologies (the guys who write the database) begin with DBMS_. The UTL_packages were derived from other sources. The UTL_HTTP package, for performing HTTP calls from PL/SQL (to retrieve web pages and such), is an example of such an external package. The Application Server Division at Oracle developed this package in order to support the concept of ICX (Inter-Cartridge eXchange) with OAS (the Oracle Application Server), which has now been replaced with iAS,(the internet Application Server). This naming difference does not mean anything to us, the developers, really – it is just interesting to note.

Necessary Supplied Packages

1029

Most of these packages are stored in a compiled, wrapped format in the database. This wrapped format protects the code from snooping eyes. We can see the specification of the code but we cannot see the code itself. If you were to select the code of DBMS_OUTPUT PACKAGE BODY from the database itself, it might look like something like this:

tkyte@TKYTE816> select text 2 from all_source 3 where name = 'DBMS_OUTPUT' 4 and type = 'PACKAGE BODY' 5 and line < 10 6 order by line 7 /

TEXT------------------------------------------

package body dbms_output wrapped 0abcdabcdabcdabcdabcdabcdabcd

9 rows selected.

Not very useful. What is very useful however, is if we select out the specification of the PACKAGE:

tkyte@TKYTE816> select text 2 from all_source 3 where name = 'DBMS_OUTPUT' 4 and type = 'PACKAGE' 5 and line < 26 6 order by line 7 /

TEXT--------------------------------------------------------------------------

package dbms_output as

------------ -- OVERVIEW -- -- These procedures accumulate information in a buffer (via "put" and -- "put_line") so that it can be retrieved out later (via "get_line" or -- "get_lines"). If this package is disabled then all -- calls to this package are simply ignored. This way, these routines -- are only active when the client is one that is able to deal with the -- information. This is good for debugging, or SP's that want to want -- to display messages or reports to sql*dba or plus (like 'describing -- procedures', etc.). The default buffer size is 20000 bytes. The -- minimum is 2000 and the maximum is 1,000,000.

Appendix A

1030

----------- -- EXAMPLE -- -- A trigger might want to print out some debugging information. To do -- do this the trigger would do -- dbms_output.put_line('I got here:'||:new.col||' is the new value'); -- If the client had enabled the dbms_output package then this put_line -- would be buffered and the client could, after executing the statement -- (presumably some insert, delete or update that caused the trigger to -- fire) execute

25 rows selected.

Hidden in the database is an online source of documentation. Each of these packages has a specification that has a nice overview of what the package is, what each function or procedure does, and how to use it. This is obviously very handy when you don't have the documentation, but is also useful even when you do, since the specification sometimes contains data that the documentation doesn't mention, or has further examples that are useful.

We will now look at the various packages I find useful in day-to-day work with Oracle. These are the packages, which not only I use frequently, but find others using as well. Additionally, we'll introduce some new packages, or ways to do things to work around some of the limitations of these built-in packages – limits people frequently hit, in my experience.

DBMS_ALERT and DBMS_PIPE

1031


The two packages, DBMS_ALERT and DBMS_PIPE, are very powerful inter-process communication packages. Both allow for one session to talk to another session in the database. DBMS_ALERT is very much like a UNIX operating system 'signal', and DBMS_PIPE is very much like a UNIX 'named pipe'. Since a lot of confusion exists over which package to use and when, I've decided to handle them together.

The package DBMS_ALERT is designed to allow a session to signal the occurrence of some event in the database. Other sessions that are interested in this event would be notified of its occurrence. Alerts are designed to be transactional in nature, meaning that you might signal an alert in a trigger, or some stored procedure, but until your transaction actually commits, the alert will not be sent out to the waiting sessions. If you rollback, your alert is never sent. It is important to understand that the session wishing to be notified of the alert in the database must either occasionally 'poll' for the event (ask the database if it has been signaled), or block (wait) in the database, waiting for the event to occur.

The package DBMS_PIPE on the other hand, is a more generic inter-process communication package. It allows one or more sessions to 'read' on one end of a named pipe, and one or more sessions to 'write' messages onto this pipe. Only one of the 'read' sessions will ever get the message (and at least one session will), and it is not possible to direct a given message on a single named pipe to a specific session. It will be somewhat arbitrary as to which session will read a given message written to a pipe when there is more then one 'reader' available. Pipes are, by design, not transactional in nature – as soon as you send a message, the message will become available to other sessions. You do not need to commit, and committing or rolling back will not affect the outcome of sending the pipe.

Appendix A

1032

Why You Might Use Them The major difference between alerts and pipes is the transactional (or not) nature of the two. Alerts are useful when you desire to transmit a message to one or more sessions after it has been successfully committed to the database. Pipes are useful when you desire to transmit a message to a single session immediately. Examples of when you might use alerts are:

❑ You have a GUI chart to display stock data on a screen. When the stock information is modified in the database, the application should be notified so that it knows to update the screen

❑ You wish to put a notification dialogue up in an application when a new record is placed into a table so the end user can be notified of 'new work'

Examples of when you might choose to use a database pipe would be:

❑ You have a process running on some other machine in the network that can perform an operation for you. You would like to send a message to this process to ask it to do something for you. In this way, a database pipe is much like a TCP/IP socket.

❑ You would like to queue some data up in the SGA so that another process will ultimately come along, read out and process. In this fashion, you are using a database pipe like a non-persistent FIFO queue that can be read by many different sessions.

There are other examples of both, but these cover the main uses of alerts and pipes, and give a good characterization of when you might use one over the other. You use alerts when you want to notify a community of users of an event that has definitely taken place (after the commit). You use pipes when you want to immediately send a message to some other session out there (and typically wait for a reply).

Now that we understand the basic intention of alerts and pipes, we'll take a look at some of the implementation details of each.

Set Up DBMS_ALERT and DBMS_PIPE are both installed by default in the database. Unlike many of the supplied packages, EXECUTE on these packages is not granted to PUBLIC. In Oracle 8.0 and up, EXECUTE on these packages is granted to the EXECUTE_CATALOG_ROLE. In prior releases, these packages had no default grants whatsoever.

Since EXECUTE is granted to a role, and not to PUBLIC, you will find that you cannot create a stored procedure that is dependent on these packages, since roles are never enabled during the compilation of a procedure/package. You must have EXECUTE granted directly to your account.

DBMS_ALERTThe DBMS_ALERT package is very small, consisting of only seven entry points. I shall discuss the six of most interest here. The application that wishes to receive an alert will be primarily interested in:

❑ REGISTER – To register interest in a named alert. You may call REGISTER many times in a session with different names, in order to be notified when any one of a number of events occurs.


1033

❑ REMOVE – To remove your interest in an event, in order to prevent the server from attempting to notify of you an event.

❑ REMOVEALL – To remove your interest in all named alerts you registered for.

❑ WAITANY – To wait for any of the named alerts, in which you have registered your interest, to be fired. This routine will tell you the name of the event that was fired, and provide access to the brief message that might accompany it. You may either wait for a specific duration of time, or not wait at all (to allow for an occasional 'poll' from the application to see if any event has taken place, but not block waiting for an event to occur).

❑ WAITONE – To wait for a specific named alert to be fired. Like WAITANY, you may wait for a specified duration of time, or not wait at all.

And the application that wishes to signal, or fire an alert, is interested only in the routine:

❑ SIGNAL – To signal an alert upon the commit of the current transaction. A rollback will 'unsignal'.

So, DBMS_ALERT is very easy to use. A client application interested in being notified of an event might contain code such as:

tkyte@TKYTE816> begin 2 dbms_alert.register( 'MyAlert' ); 3 end; 4 /

PL/SQL procedure successfully completed.

tkyte@TKYTE816> set serveroutput on tkyte@TKYTE816> declare 2 l_status number; 3 l_msg varchar2(1800); 4 begin 5 dbms_alert.waitone( name => 'MyAlert', 6 message => l_msg, 7 status => l_status, 8 timeout => dbms_alert.maxwait ); 9 10 if ( l_status = 0 ) 11 then 12 dbms_output.put_line( 'Msg from event is ' || l_msg ); 13 end if; 14 end; 15 /

They simply register their interest in the named alert, MyAlert, and then call DBMS_ALERT.WAITONE to wait for this alert to be fired. Notice that since DBMS_ALERT.MAXWAIT is used, a constant from the DBMS_ALERT package, this code will just 'sit there'. It is blocked in the database waiting for this event to occur. The interested client application might use a much smaller timeout period specified in seconds (perhaps 0, meaning no waiting should occur) so it could poll for an event. For example, an Oracle Forms application might have a timer that goes off every minute and calls DBMS_ALERT.WAITONE to see if some event has occurred. If so, the screen will be updated. A Java thread might become active every so often to check for an event, and update some shared data structure, and so on.

Appendix A

1034

Now, in order to signal this alert, all we need to do is:

tkyte@TKYTE816> exec dbms_alert.signal( 'MyAlert', 'Hello World' );


tkyte@TKYTE816> commit;

Commit complete.

in another session. You should immediately see:

... 15 / Msg from event is Hello World


in the session that was blocked waiting for the alert, so this session will no longer be blocked. This simple example shows the most commonly used format of DBMS_ALERT. Some sessions wait on a named alert, and another session signals it. Until the signaling session commits, the alert does not go through. You'll see this yourself easily using two SQL*PLUS sessions.

It gets more interesting with alerts when we ask ourselves:

❑ What happens when many messages get 'signaled' at more or less the same time by different sessions?

❑ What happens if I call signal repeatedly – how many alerts will be generated in the end?

❑ What happens if more than one session signals an alert after I registered interest in it, but before I've called one of the wait routines? Same question, only what happens when more than one session signals an alert between my calls to wait?

The answers to these questions will point out some of the side effects of alerts; some of the things you need to be aware of when using them. I'll also suggest ways to avoid some of the issues these questions raise.

Concurrent Signals by More than One Session If we re-execute our small test from above, have the one session register its interest in MyAlert, wait on it, and then start up two additional sessions, we can easily see what happens when more than one session signals an alert simultaneously. In this test, both of the other two sessions will execute:

tkyte@TKYTE816> exec dbms_alert.signal( 'MyAlert', 'Hello World' );

and nothing else (no commit). What you will observe in this case is that the session, which issued the second signal, is blocked. This shows that if N sessions attempt to signal the same named event concurrently, N-1 of them will block on the DBMS_ALERT.SIGNAL call. Only one of the sessions will continue forward. Alerts are serial in nature, and care must be taken to avoid issues with this.


1035

The database is designed to provide highly concurrent access to data. DBMS_ALERT is one of those tools that can definitely limit scalability in this area. If you place an INSERT trigger on a table and this trigger places a DBMS_ALERT.SIGNAL call when fired then, if the table is subject to frequent INSERT statements, you will serialize all INSERTs on that particular table whenever someone is registered for that alert. For this reason, you may want to consider limiting the number of overall sessions that might signal an alert. For example, if you have a live data feed coming into your database so that there is only one session inserting data into this table, DBMS_ALERT would be appropriate. On the other hand, if this is an audit trail table that everyone must INSERT into frequently, DBMS_ALERT would not be an appropriate technology.

One method to avoid this serialization by many sessions could be to use DBMS_JOB (detailed in its own section in this appendix). You might write a procedure in which the only thing you do is signal the alert and commit:

tkyte@TKYTE816> create table alert_messages 2 ( job_id int primary key, 3 alert_name varchar2(30), 4 message varchar2(2000) 5 ) 6 /

Table created.

tkyte@TKYTE816> create or replace procedure background_alert( p_job in int ) 2 as 3 l_rec alert_messages%rowtype; 4 begin 5 select * into l_rec from alert_messages where job_id = p_job; 6 7 dbms_alert.signal( l_rec.alert_name, l_rec.message ); 8 delete from alert_messages where job_id = p_job; 9 commit; 10 end; 11 /

Procedure created.

Then, your database trigger would look like this:

tkyte@TKYTE816> create table t ( x int ); Table created.

tkyte@TKYTE816> create or replace trigger t_trigger 2 after insert or update of x on t for each row 3 declare 4 l_job number; 5 begin 6 dbms_job.submit( l_job, 'background_alert(JOB);' ); 7 insert into alert_messages 8 ( job_id, alert_name, message ) 9 values 10 ( l_job, 'MyAlert', 'X in T has value ' || :new.x ); 11 end; 12 /

Trigger created.

Appendix A

1036

to have the alert signaled by a background process after you commit. In this fashion:

❑ Alerts are still transactional

❑ They will not serialize your foreground processes (interactive applications)

The drawback is that jobs are not necessarily run right away; it might be a little while before the alert gets out. In many cases, I have found this to be acceptable (it is important to notify the waiting process that something has occurred, but a short lag time is generally OK). Advanced queues (AQ) also supply a highly scalable method of signaling events in the database. They are more complex to use than DBMS_ALERT, but offer more flexibility in this area.

Repeated Calls to Signal by a Session Now the question is, what if I signal the same named alert many times in my application, and then commit? How many alerts actually get signaled? Here, the answer is simple: one. DBMS_ALERT works very much like a UNIX signal would. The UNIX OS uses signals to notify processes of events that have occurred in the operating system. One such event for example is 'I/O is ready', meaning that one of the files (or sockets or such) you have open is ready for more I/O. You might use this signal when building a TCP/IP-based server for example. The OS will notify you when a socket you have opened,has data that is waiting to be read on it, rather than you going to each socket, and peeking into it to see if it has more data ready. If the OS determines five times that the socket has data to be read, and it did not get a chance to notify you yet, it will not tell you five times, it will only tell you once. You get the event, 'socket X is ready to be read'. You do not get all of the prior events about that socket. DBMS_ALERTworks in the same exact fashion.

Returning to our simple example from above, we would run the snippet of code that registers its interest in an event, and calls the WAITONE routine to wait for this event. In another session, we will execute:

tkyte@TKYTE816> begin 2 for i in 1 .. 10 loop 3 dbms_alert.signal( 'MyAlert', 'Message ' || i ); 4 end loop; 5 end; 6 /



Commit complete.

In the other window, we will see the feedback:

Msg from event is Message 10


Only the very last message we signaled will get out – the intervening messages will never be seen. You must be aware the DBMS_ALERT will, by design, drop messages by a session. It is not a method to deliver a sequence of messages, it is purely a signaling mechanism. It gives you the ability to tell a client application 'something has happened'. If you rely on each and every event you ever signal to be received by all sessions, you will be disappointed (and most likely have a bug in your code on your hands).


1037

Again, DBMS_JOB can be used to some extent to resolve this issue if it is paramount for each event to be signaled. However, at this point, an alternate technology comes to mind. Advanced queues, (a topic outside the scope of this book), can be used to satisfy that requirement in a much better fashion.

Many Calls to Signal by Many Sessions before a Wait Routine is Called

This is the last question; what happens if more than one session signals an alert after I registered interest in it, but before I've called one of the wait routines? Same question, only what happens when more than one session signals an alert between my calls to wait? The answer is the same as when a single session makes many calls to DBMS_ALERT.SIGNAL. Only the last event is remembered, and signaled out. You can see this by placing a PAUSE in the simple SQL*PLUS script we have been using so that it reads like this:

begin dbms_alert.register( 'MyAlert' ); end;/pause

Now, in some other sessions, call the DBMS_ALERT.SIGNAL with unique messages (so you can distinguish them) and commit each message. For example, modify our simple loop from above as follows:

tkyte@TKYTE816> begin 2 for i in 1 .. 10 loop 3 dbms_alert.signal( 'MyAlert', 'Message ' || i ); 4 commit; 5 end loop; 6 end; 7 / PL/SQL procedure successfully completed.

After you do that and return to this original session, simply hit the Enter key, and the block of code that calls WAITONE will execute. Since the alert we are waiting on has been signaled already, this block of code will return immediately, and will show us we received the last message that was signaled by this alert. All of the other intervening messages from the other sessions are lost, by design.

Summary The DBMS_ALERT package is suitable for those cases where you wish to notify a large audience of interested clients about events in the database. These named events should be signaled by as few sessions as possible, due to inherent serialization issues with DBMS_ALERT. Since messages will by design be 'lost', DBMS_ALERT is suitable as an eventnotification process. You can use it to notify an interested client that data in a table T has changed for example, but to try and use it to notify these clients of the changes of individual rows in T would not work (due to the fact that only the 'last' message is saved). DBMS_ALERT is a very simple package to use and requires little to no set up.

Appendix A

1038

DBMS_PIPEDBMS_PIPE is a package supplied to allow two sessions to communicate with each other. It is an inter-process communication device. One session can write a 'message' on a pipe, and another session can 'read' this message. In UNIX, the same concept exists in the form of a named pipe in the operating system. With named pipes, we can allow one process to write data to another process.

The DBMS_PIPE package, unlike DBMS_ALERT, is a 'real time' package. As soon as you call the SEND_MESSAGEfunction, the message is sent. It does not wait for a COMMIT; it is not transactional. This makes DBMS_PIPEsuitable for cases where DBMS_ALERT is not (and vice-versa). We can use DBMS_PIPE to allow two sessions to have a conversation (not something we can do with DBMS_ALERT). Session one could ask session two to perform some operation. Session two could do it, and return the results to session one. For example, assume session two is a C program that can read a thermometer attached to the serial port of the computer it is running on, and return the temperature to session one. Session one needs to record in a database table the current temperature. It can send a 'give me the temperature' message to session two, which would find out what this is, and write the answer back to session one. Session one and session two may or may not be on the same computer – all we know is that they are both connected to the database. I am using the database much like I might use a TCP/IP network to perform communication between two processes. In the case of DBMS_PIPE however, I do not need to know a hostname and a port number to connect to, like you would with TCP/IP – just the name of the database pipe upon which to write my request.

There are two types of pipes available in the database – public and private. A public pipe can either be created explicitly via a call to CREATE_PIPE, or you may just implicitly create one upon sending a message on it. The major difference between an explicit and implicit pipe is that the pipe created via the CREATE_PIPE call should be removed by your application when it is done using it, whereas the implicit pipe will age out of the SGA after it hasn't been accessed for a while. A public pipe is set up such that any session, which has access to the DBMS_PIPEpackage can read and write messages on the pipe. Therefore, public pipes are not suitable for sensitive or even just 'important' data. Since pipes are typically used to perform a conversation of sorts, and a public pipe allows anyone to read or write this conversation, a malicious user could either remove messages from your pipe, or add additional 'garbage' messages onto your pipe. Either action would have the effect of breaking the conversation or protocol between the sessions. For this reason, most applications will use a private pipe.

Private pipes may be read or written, only by sessions that operate under the effective user ID of the owner of the pipe and the special users SYS and INTERNAL. This means only definer rights (see the Chapter 23, Invoker and Definer Rights) stored procedures owned by the owner of the pipe or sessions logged in as the owner of the pipe, SYS, or INTERNAL can read or write on this pipe. This significantly enhances the reliability of pipes as no other session or piece of code can corrupt or intercept your protocol.

A pipe is an object that will live in the SGA of your Oracle instance. It is not a disk-based mechanism at all. Data in a pipe will not survive a shutdown and startup – any information in this pipe at shutdown will be flushed, and will not be in the pipe again upon startup.

The most common usage of pipes is to build your own customized services or servers. Prior to the introduction of external procedures in Oracle 8.0, this was the only way to implement a stored procedure in a language other than PL/SQL. You would create a 'pipe' server. In fact, ConText (the precursor to interMedia text) was implemented using database pipes in Oracle 7.3, onwards. Over time, some of its functionality was implemented via external procedures, but much of the indexing logic is still implemented via database pipes.


1039

Due to the fact that any number of sessions can attempt to read off of a pipe, and any number may attempt to write on a given pipe, we must implement some logic to ensure that we can deliver messages to the correct session. If we are going to create our own customized service (for example the thermometer demonstration from earlier) and add it to the database, we must make sure that the answer for session A's question gets to session A, and not session B. In order to satisfy that very typical requirement, we generally write our requests in one message onto a pipe with a well-known name, and include in this message, a unique name of a pipe we expect to read our response on. We can show this in the following figure:

Oracle8i

2

TemperatureServer

75

A

B

temperature pipe

Application A

Application B

4

What’s thetemperature?Answer on pipe “B”

What’s thetemperature?Answer on pipe “A” 1

3

❑ Step 1 – Session A will write it's request, 'What is the temperature? Answer on pipe A ' onto the well-known pipe named 'temperature pipe'. At the same time, other sessions may be doing the same thing. Each message will be queued into the pipe in a 'first in, first out' fashion.

❑ Step 2 – The temperature server will read a single message out of the pipe, and query whatever service it is providing access to.

❑ Step 3 – The temperature server will use the unique pipe name, which the session requesting the information wrote onto the pipe, to write a response (pipe A in this example). We use an implicit queue for this response (so the response pipe disappears right after we are done). If we planned on making many such calls, we would want to use an explicitly created pipe to keep it in the SGA during our session (but we would have to remember to clean it out upon logout!).

❑ Step 4 – Session A reads the response back from the pipe it told the temperature server to write the answer on.

The same sequence of events would take place for Session B. The temperature server would read its request, query the temperature, look at the message to find the name of the pipe to answer on, and write the response back.

One of the interesting aspects of database pipes is that many sessions can read from the pipe. Any given message placed onto the pipe will be read by exactly one session, but many sessions can be reading at the same time. This allows us to 'scale' up the above picture. In the above, it is obvious we could have many sessions requesting data from the 'temperature server', and it would serially process them one after the other. There is nothing stopping us from starting more than one temperature server as thus:

Appendix A

1040

Oracle8i

A

B

temperature pipeTemperature

Server

TemperatureServer

75

We can now service two concurrent requests. If we started five of them, we could do five at a time. This is similar to connection pooling, or how the multi-threaded server in Oracle itself works. We have a pool of processes ready to do work, and the maximum amount of concurrent work we can do at any point in time is dictated by the number of processes we start up. This aspect of database pipes allows us to scale up this particular implementation easily.

Pipe Servers versus External Routines Oracle8 release 8.0 introduced the ability to implement a stored procedure in C directly, and Oracle8i gave us the ability to implement the stored procedure in Java. Given this, is the need for DBMS_PIPE and 'pipe servers' gone? The short answer is no, absolutely not.

When we covered external routines, we described their architecture. C-based external routines for example, execute in an address space separate from the PL/SQL stored procedure. There exists a one-to-one mapping between the number of sessions concurrently using an external procedure, and the number of separate address spaces created. That is, if 50 sessions concurrently call the external routine, there will be 50 EXTPROC processes or threads at least. C-based external routines are architected in a manner similar to the dedicated server mode of Oracle. Just as Oracle will create a dedicated server for each concurrent session, it will also create an EXTPROCinstance for every concurrent external routine call. Java external routines are executed in much the same fashion – one-to-one. For every session using a Java external routine, there will be a separate JVM instance running in the server with its own state and resources.

A pipe server on the other hand works like the MTS architecture does in Oracle. You create a pool of shared resources (start up N pipe servers), and they will service the requests. If more requests come in concurrently than can be handled, the requests will be queued. This is very analogous to the MTS mode of Oracle, whereby requests will be queued in the SGA, and dequeued by a shared server as soon as they completed processing the prior request they were working on. The temperature example we walked through earlier is a good example of this. The first diagram depicts a single pipe server running; one temperature at a time will be retrieved and returned to a client. The second diagram depicts two pipe servers running to service all of the requests. Never more than two concurrent requests will be processed. The thermometer will never have more than two clients hitting it.

The reason this is important is that it gives us a great capability to limit concurrent accesses to this shared resource. If we used external routines, and 50 sessions simultaneously requested the temperature, they may very well 'crash' the thermometer if it was not designed to scale up to so many requests. Replace the thermometer with many other shared resources, and you may find the same problem arises. It can handle a couple of concurrent requests, but if you tried to hit it with many simultaneous requests, either it would fail, or performance would suffer to the point of making it non-functional.


1041

Another reason why a pipe server might make sense is in accessing some shared resource that takes a long time to 'connect to'. For example, I worked on a project a couple of years ago at a large university. They needed access to some mainframe transactions (then needed to call up to a mainframe to get some student information). The initial connection to the mainframe might take 30 to 60 seconds to complete but after that, it was very fast (as long as we didn't overload the mainframe with tons of concurrent requests). Using a pipe server, we were able to initiate the connection to the server once, when the pipe server started up. This single pipe server would run for days using that initial connection. Using an external routine we would have to initiate the connect once per database session. An implementation that used external routines would quite simply not work in this environment due to the high startup costs associated with the mainframe connection. The pipe server not only gave them the ability to limit the number of concurrent mainframe requests, but it also provided the ability to do the expensive mainframe connection once, and then reuse this connection many hundreds of thousands of times.

If you are familiar with the reasoning behind using connection pooling software in a 3-tier environment, you are already familiar with why you would want to use pipes in certain circumstances. They provide the ability to reuse the outcome of a long running operation (the connection to the database in the case of the connection pooling software) over an over, and they give you the ability to limit the amount of resources you consume concurrently (the size of your connection pool).

One last difference between a pipe server and external routines is where the pipe server can run. Suppose in the temperature server example, the database server was executing on Windows. The temperature probe is located on a UNIX machine. The only object libraries available to access it are on UNIX. Since a pipe server is just a client of the database like any other client, we can code it, compile it, and run it on UNIX. The pipe server need not be on the same machine, or even platform, as the database itself. An external routine on the other hand, must execute on the same machine with the database server itself – they cannot execute on remote machines. Therefore, a pipe server can be used in circumstances where an external routine cannot.

Online Example On the Apress web site (http://www.apress.com), you'll find an example of a small pipe server. It answers the frequently asked question, 'How can I run a host command from PL/SQL?' With the addition of Java to the database and C external procedures, we could easily implement a host command function with either technology. However, what if I do not have access to a C compiler, or I don't have the Java component of the database available – what then? The example shows how we could very simply setup a small 'pipe server' that can do host commands using nothing more than SQL*PLUS and the csh scripting language. It is fairly simple, consisting of only a few lines of csh, and even fewer of PL/SQL. It shows much of the power of database pipes though, and should give you some ideas for other interesting implementations.

Summary Database pipes are a powerful feature of the Oracle database that allow for any two sessions to have a 'conversation' with each other. Modeled after pipes in UNIX, they allow you to develop your own protocol for sending and receiving messages. The small example available on the Apress web site demonstrates how easy it can be to create a 'pipe server', an external process that receives requests from database sessions and does something 'special' on their behalf. Database pipes are not transactional, differentiating them from database alerts, but it is this non-transactional feature that makes them so useful in many cases. Amongst many others, I have used database pipes to add features to the database such as:

❑ Sending e-mail.

❑ Printing files.

❑ Integrating non-Oracle, non-SQL data sources.

❑ Implementing the equivalent of DBMS_LOB.LOADFROMFILE for LONGs and LONG RAWs.

Appendix A

1042

DBMS_APPLICATION_INFO

This is one of the most under-utilized features in the set of supplied packages, yet I cannot think of a single application that would not benefit from its use. Have you ever asked yourself:

❑ I wonder what that session is doing, what form is it running, what code module is executing?

❑ I wonder how far along that stored procedure is?

❑ I wonder how far along that batch job is?

❑ I wonder what bind variable values were being used on that query?

DBMS_APPLICATION_INFO is the package that can be used to answer all of these questions, and more. It allows us to set up to three columns in our row of the V$SESSION table – the CLIENT_INFO, ACTION, and MODULEcolumns. It provides functions not only to set these values, but also to return them. Further, there is a parameter to the built-in USERENV or SYS_CONTEXT function that will allow us to access the CLIENT_INFO column easily in any query. I can SELECT USERENV('CLIENT_INFO') FROM DUAL for example, or use WHERESOME_COLUMN = SYS_CONTEXT( 'USERENV','CLIENT_INFO') in my queries. The values we set in the V$tables are immediately visible. We do not need to commit them to 'see' them, making them very useful for communicating with the 'outside'. Lastly, it allows us to set values in the dynamic performance view V$SESSION_LONGOPS (LONG OPerationS) as well – useful for recording the progress of long running jobs.

Many Oracle tools, such as SQL*PLUS, already make use of this facility. For example, I have a script, SHOWSQL.SQL, which I use to see what SQL people are currently running in the database (this is available on the Apress web site at http://www.apress.com). Part of this script dumps out the V$SESSION table for all entries where CLIENT_INFO, MODULE, or ACTION is NOT NULL. Whenever I run it, I see, for example:


1043

USERNAME MODULE ACTION CLIENT_INFO-------------------- --------------- --------------- ---------------------- OPS$TKYTE(107,19225) 01@ showsql.sql OPS$TKYTE(22,50901) SQL*Plus

The first line shows my current session running the script SHOWSQL.SQL with a level of 01. This means that this script has not called another script yet. If I were to create a script TEST.SQL with just @SHOWSQL in it, then SQL*PLUS would set 02 in front of SHOWSQL to show that it is nested. The second line shows another SQL*PLUS session. It is not running any scripts right now (it may have been executing a command entered on the command line directly). If you add the appropriate calls to DBMS_APPLICATION_INFO to your application, you can do the same thing, enhancing the abilities of you and your DBA to monitor your application.

The calls to set these values in the V$SESSION table are simply:

❑ SET_MODULE – This API call allows you to set both the MODULE, and ACTION columns in V$SESSION. The name of the module is limited to 48 bytes and the value of the action is limited to 32 bytes. The name of the module would typically be your application name. The initial action might be something like STARTUP or INITIALIZING to indicate the program is just starting.

❑ SET_ACTION – This API calls allows you to set the ACTION column in V$SESSION. ACTIONshould be a descriptive term to let you know where in your program you are. You might set action to be the name of the currently active form in a forms application for example, or the name of a subroutine in a Pro*C or PL/SQL routine.

❑ SET_CLIENT_INFO – This API call allows you to store up to 64 bytes of any sort of application specification information you might wish to keep. A common use of this is to parameterize views (see below) and queries.

There are corresponding API calls to read this information back out as well. In addition to setting values in the V$SESSION table, this package allows you to set information in the V$SESSION_LONGOPS dynamic performance view. This view allows you to store more than one row of information in various columns. We will take an in depth look at this functionality in a moment.

Using the Client Info The SET_CLIENT_INFO call gives us the ability to not only set a value in a column of the V$SESSION table, but also gives us access to that variable via the built-in function userenv (Oracle 7.3 and up) or sys_context(preferred function in Oracle 8i and up). For example, with this we can create a parameterized view, a view whose results depend on the value in the CLIENT_INFO field. The following example demonstrates this concept:

scott@TKYTE816> exec dbms_application_info.set_client_info('KING');


scott@TKYTE816> select userenv('CLIENT_INFO') from dual;

USERENV('CLIENT_INFO')------------------------------------KING

Appendix A

1044

scott@TKYTE816> select sys_context('userenv','client_info')from dual;

SYS_CONTEXT('USERENV','CLIENT_INFO') ------------------------------------KING

scott@TKYTE816> create or replace view 2 emp_view 3 as 4 select ename, empno 5 from emp 6 where ename = sys_context( 'userenv', 'client_info');

View created.

scott@TKYTE816> select * from emp_view;

ENAME EMPNO ---------- ---------- KING 7839

scott@TKYTE816> exec dbms_application_info.set_client_info('BLAKE');


scott@TKYTE816> select * from emp_view;

ENAME EMPNO ---------- ---------- BLAKE 7698

As you can see, we can set this value and we can also easily use it in queries where we could use a constant. This allows us to create complex views with predicates that get their values at run-time. One of the issues with views can be in the area of predicate merging. If the optimizer were able to 'merge' the predicate into the view definition, it would run really fast. If not, it runs really slow. This feature, using the client info, allows us to 'merge' the predicate ahead of time when the optimizer cannot. The application developer must set the value and just SELECT * from the view. Then, the 'right' data will come out.

Another place where I make use of this functionality is to store the bind variables I am using in my query (and other pieces of information), so I can see what my procedures are doing very quickly. For example, if you have a long running process you might instrument it like this:

tkyte@TKYTE816> declare 2 l_owner varchar2(30) default 'SYS'; 3 l_cnt number default 0; 4 begin 5 dbms_application_info.set_client_info( 'owner='||l_owner ); 6 7 for x in ( select * from all_objects where owner = l_owner ) 8 loop 9 l_cnt := l_cnt+1; 10 dbms_application_info.set_action( 'processing row ' || l_cnt ); 11 end loop; 12 end; 13 /


1045

Now, using that SHOWSQL.SQL script once again, I can see:

tkyte@TKYTE816> @showsql

USERNAME SID SERIAL# PROCESS STATUS ------------------------------ ---------- ---------- --------- ---------- TKYTE 8 206 780:716 ACTIVE TKYTE 11 635 1004:1144 ACTIVE --------------------TKYTE(11,635) ospid = 1004:1144 program = SQLPLUS.EXE Saturday 15:59 Saturday 16:15 SELECT * FROM ALL_OBJECTS WHERE OWNER = :b1

USERNAME MODULE ACTION CLIENT_INFO--------------- --------------- --------------- --------------------------- TKYTE(8,206) 01@ showsql.sql TKYTE(11,635) SQL*Plus processing row owner=SYS 5393

Session (11,635) is running the query SELECT * FROM ALL_OBJECTS WHERE OWNER = :B1. The report also shows me that owner=SYS in this case, and at the point in time we were looking at it, it had already processed 5,393 rows. In the next section, we'll see how using SESSION LONGOPS can take this a step further, if you know how many operations or steps your procedure will be performing.

Using V$SESSION_LONGOPS Many operations in the database may take a considerable amount of time. Parallel execution, Recovery Manager, large sorts, loads, and so on fall into this category. These long running operations take advantage of their ability to set values in the dynamic performance view, V$SESSION_LONGOPS to let us know how far along in their work they are, and so can your applications. This view displays the status of various database operations that run for longer than six seconds. That is, functions the database performs that the Oracle developers felt would normally take longer than six seconds have been instrumented to populate the V$SESSION_LONGOPS view. This does not mean anything that takes longer than six seconds will automatically appear in this view. These operations currently include many backup and recovery functions, statistics gathering, and query execution. More operations are added for every Oracle release.

Changes made to this view are immediately visible to other sessions, without the need to commit your transaction. For any process that updates this view, you will be able to monitor their progress from another session by querying the V$SESSION_LONGOPS view. You too have the ability to populate rows in this view, typically one row, but you may use others if you like.

The API to set the values in this view is defined as:

PROCEDURE SET_SESSION_LONGOPS Argument Name Type In/Out Default? ------------------------------ ----------------------- ------ -------- RINDEX BINARY_INTEGER IN/OUT SLNO BINARY_INTEGER IN/OUT OP_NAME VARCHAR2 IN DEFAULT TARGET BINARY_INTEGER IN DEFAULT CONTEXT BINARY_INTEGER IN DEFAULT SOFAR NUMBER IN DEFAULT TOTALWORK NUMBER IN DEFAULT TARGET_DESC VARCHAR2 IN DEFAULT UNITS VARCHAR2 IN DEFAULT

Appendix A

1046

with the following meanings:

❑ RINDEX – Tells the server which row to modify in the V$SESSION_LONGOPS view. If you set this value to DBMS_APPLICATION_INFO.SET_SESSION_LONGOPS_NOHINT, a new row will be allocated in this view for you, and the index of this row will be returned in RINDEX.Subsequent calls to SET_SESSION_LONGOPS with the same value for RINDEX will update this already existing row.

❑ SLNO – An internal value. You should initially pass a Null number in, and ignore its value otherwise. You should pass the same value in with each call.

❑ OP_NAME – The name of the long running process. It is limited to 64 bytes in size, and should be set to some string that will be easily identified and provides some meaning to you.

❑ TARGET – Typically used to hold the object ID that is the target of the long running operation (for example, the object ID of the table being loaded). You may supply any number you wish here, or leave it Null.

❑ CONTEXT – A user-defined number. This number would have meaning only to you. It is simply any number you wish to store.

❑ SOFAR – This is defined as any number you want to store, but if you make this number be some percentage or indicator of the amount of work done, the database will attempt to estimate your time to completion for you. For example, if you have 25 things to do, and they all take more or less the same amount of time, you could set SOFAR to the number of things done so far, and then set the next parameter TOTALWORK. The server will figure out how long it took you to get to where you are, and estimate how long it will take you to complete.

❑ TOTALWORK – This is defined as any number you want to store, but the same caveat for SOFARapplies here. If SOFAR is a percentage of TOTALWORK, representing your progress, the server will compute the time remaining to complete your task.

❑ TARGET_DESC – This is used to describe the contents of the TARGET input from above. If the TARGET actually contained an object ID, this might contain the object name for that object ID.

❑ UNITS – A descriptive term that categorizes what SOFAR and TOTALWORK are measured in. Units might be 'files', 'iterations', or 'calls' for example.

These are the values you can set. When you look at the V$SESSION_LONGOPS view, you'll see it has many more columns than these however:

[email protected]> desc v$session_longops Name Null? Type -------------------------------- -------- ---------------------- SID NUMBER SERIAL# NUMBER OPNAME VARCHAR2(64) ** TARGET VARCHAR2(64) ** TARGET_DESC VARCHAR2(32) ** SOFAR NUMBER ** TOTALWORK NUMBER ** UNITS VARCHAR2(32) ** START_TIME DATE LAST_UPDATE_TIME DATE TIME_REMAINING NUMBER


1047

ELAPSED_SECONDS NUMBER CONTEXT NUMBER ** MESSAGE VARCHAR2(512) USERNAME VARCHAR2(30) SQL_ADDRESS RAW(4) SQL_HASH_VALUE NUMBER QCSID NUMBER

The columns marked with ** are the ones you have control over, and can set.

The meanings are as follows:

❑ The SID and SERIAL# columns are used to join back to V$SESSION, to pick up the session information.

❑ The START_TIME column marks the time this record was created (typically your first call to DBMS_APPLICATION_INFO.SET_SESSION_LONGOPS).

❑ The LAST_UPDATE_TIME column represents the time of your last call to SET_SESSION_LONGOPS.

❑ The TIME_REMAINING is an estimate in seconds of the time to completion. It is equal to ROUND(ELAPSED_SECONDS*((TOTALWORK/SOFAR)-1)).

❑ The ELAPSED_SECONDS column is the time in seconds since the start of the long running operation, and the last update time.

❑ The MESSAGE column is a derived column. It concatenates together pieces of the OPNAME,TARGET_DESC, TARGET, SOFAR, TOTALWORK, and UNITS column to make a readable description of the work in process.

❑ The USERNAME is the name of the user this process is executing under.

❑ The SQL_ADDRESS and SQL_HASH_VALUE may be used to look into V$SQLAREA to see what SQL statement this process was last executing.

❑ The QCSID is used with parallel query. It would be the session of the parallel coordinator.

So, what can you expect from this particular view? A small example will clearly show what it can provide for you. In one session, if you run a block of code such as:

tkyte@TKYTE816> declare 2 l_nohint number default dbms_application_info.set_session_longops_nohint; 3 l_rindex number default l_nohint; 4 l_slno number; 5 begin 6 for i in 1 .. 25 7 loop 8 dbms_lock.sleep(2); 9 dbms_application_info.set_session_longops 10 ( rindex => l_rindex, 11 slno => l_slno, 12 op_name => 'my long running operation',

Appendix A

1048

13 target => 1234, 14 target_desc => '1234 is my target', 15 context => 0, 16 sofar => i, 17 totalwork => 25, 18 units => 'loops' 19 ); 20 end loop; 21 end; 22 /

This is a long running operation that will take 50 seconds to complete (the DBMS_LOCK.SLEEP just sleeps for two seconds). In another session, we can monitor this session via the query below (see the Chapter 23 on Invoker and Definer Rights for the definition of the PRINT_TABLE utility used in this code):

tkyte@TKYTE816> begin 2 print_table( 'select b.* 3 from v$session a, v$session_longops b 4 where a.sid = b.sid 5 and a.serial# = b.serial#' ); 6 end; 7 / SID : 11 SERIAL# : 635 OPNAME : my long running operation TARGET : 1234 TARGET_DESC : 1234 is my target SOFAR : 2 TOTALWORK : 25 UNITS : loops START_TIME : 28-apr-2001 16:02:46 LAST_UPDATE_TIME : 28-apr-2001 16:02:46 TIME_REMAINING : 0 ELAPSED_SECONDS : 0 CONTEXT : 0 MESSAGE : my long running operation: 1234 is my target 1234: 2 out of 25 loops done USERNAME : TKYTE SQL_ADDRESS : 036C3758 SQL_HASH_VALUE : 1723303299 QCSID : 0 -----------------


[email protected]> / SID : 11 SERIAL# : 635 OPNAME : my long running operation TARGET : 1234 TARGET_DESC : 1234 is my target SOFAR : 6 TOTALWORK : 25 UNITS : loops START_TIME : 28-apr-2001 16:02:46 LAST_UPDATE_TIME : 28-apr-2001 16:02:55 TIME_REMAINING : 29 ELAPSED_SECONDS : 9


1049

CONTEXT : 0 MESSAGE : my long running operation: 1234 is my target 1234: 6 out of 25 loops done USERNAME : TKYTE SQL_ADDRESS : 036C3758 SQL_HASH_VALUE : 1723303299 QCSID : 0 -----------------


[email protected]> / SID : 11 SERIAL# : 635 OPNAME : my long running operation TARGET : 1234 TARGET_DESC : 1234 is my target SOFAR : 10 TOTALWORK : 25 UNITS : loops START_TIME : 28-apr-2001 16:02:46 LAST_UPDATE_TIME : 28-apr-2001 16:03:04 TIME_REMAINING : 27 ELAPSED_SECONDS : 18 CONTEXT : 0 MESSAGE : my long running operation: 1234 is my target 1234: 10 out of 25 loops done USERNAME : TKYTE SQL_ADDRESS : 036C3758 SQL_HASH_VALUE : 1723303299 QCSID : 0 -----------------


The first question you might ask is, 'why did I join V$SESSION_LONGOPS to V$SESSION if I did not actually select any information from V$SESSION?' This is because the view V$SESSION_LONGOPS will contain values from rows of current, as well as legacy sessions. This view is not 'emptied out' when you log out. The data you left there remains until some other session comes along, and reuses your slot. Therefore, to see long operations information for current sessions only, you want to join or use a sub-query to get current sessions only.

As you can see from the rather simple example, this information could be quite invaluable to you and your DBA, as far as monitoring long running stored procedures, batch jobs, reports, and so on, goes. A little bit of instrumentation can save a lot of guesswork in production. Rather than trying to 'guess' where a job might be and how long it might take to complete, you can get an accurate view of where it is, and an educated guess as to the length of time it will take to complete.

Summary Here, we have looked at the DBMS_APPLICATION_INFO package, an often overlooked and under-utilized package. Every application can, and should, make use of this particular package, just to register itself in the database so the DBA, or anyone monitoring the system, can tell what applications are using it. For any process that takes more than a few seconds, the use of V$SESSION_LONGOPS is critical. To show that a process is not 'hanging' but is moving along at a steady pace, this feature is the only way to go. Oracle Enterprise Manager (OEM), and many third party tools, are aware of these views and will automatically integrate your information into their display.

Appendix A

1050

DBMS_JAVA

The DBMS_JAVA package is somewhat of an enigma. It is a PL/SQL package but it is not documented in the Supplied PL/SQL Packages Reference guide. It is designed to support Java in the database, so you might expect to find it in the Supplied Java Packages Reference guide (but you won't). It is actually documented in the Oracle8i Java Developer's Guide. We've used it many times in this book already without really going through it, so here we will cover the procedures I use within this package, how to use them, and what they do.

The DBMS_JAVA package has almost 60 procedures and functions, only a very small handful of which are useful to us as developers. The bulk of this package is in support of debuggers (not for us to debug with, but for others to write debuggers for us), various internal convenience routines, and the export/import utilities. We will skip these functions and procedures altogether.

LONGNAME and SHORTNAME These are utility routines to convert between a 'short' 30-character identifier (all Oracle identifiers are 30 characters or less), and the 'long' Java name. If you look in the data dictionary, you will typically find a 'hashed' name for the Java classes that are loaded into the database. This is because they come with really long names, which the server cannot deal with. These two routines allow you to see what the 'real' name is, given a short name (OBJECT_NAMEcolumn in USER_OBJECTS), and what the short name would be given a long name. Here is an example of the usage of each when logged in as the user SYS (who happens to own lots of Java code, if you have Java installed in the database):

DBMS_JAVA

1051

sys@TKYTE816> column long_nm format a30 word_wrapped sys@TKYTE816> column short_nm format a30

sys@TKYTE816> select dbms_java.longname(object_name) long_nm, 2 dbms_java.shortname(dbms_java.longname(object_name)) short_nm 3 from user_objects where object_type = 'JAVA CLASS' 4 and rownum < 11 5 /

LONG_NM SHORT_NM------------------------------ ------------------------------ com/visigenic/vbroker/ir/Const /1001a851_ConstantDefImpl antDefImpl

oracle/sqlj/runtime/OraCustomD /10076b23_OraCustomDatumClosur atumClosure

com/visigenic/vbroker/intercep /10322588_HandlerRegistryHelpe tor/HandlerRegistryHelper...

10 rows selected.

As you can see, using LONGNAME on the OBJECT NAME turns it into the original class name for the Java class. If we take this long name and pass it through SHORTNAME, we get back the hashed-shortened name Oracle uses internally.

Setting Compiler Options You may specify most compiler options for the Java compiler in the database, in one of two places; the command line when using loadjava, or in the JAVA$OPTIONS database table. A setting on the command line will always override the JAVA$OPTIONS table. This only applies if you use the Oracle Java compiler in the database, of course. If you use a standalone Java compiler outside of the database (JDeveloper perhaps), you will set compiler options in that environment.

There are three compiler options we may set, and they all relate to the SQLJ compiler (a pre-compiler for Java, converts embedded SQL statements into JDBC calls) built-in to the database. They are:

Option Meaning Values

ONLINE Whether type checking is done at compile-time (online), or run-time.

True/False

DEBUG Whether the Java code is compiled with debugging enabled. Equivalent to javac -g in a command line environment.

True/False

ENCODING Identifies the source file encoding for the compiler. Latin1 is the default

The values in bold are the default settings.

Appendix A

1052

We'll demonstrate the use of DBMS_JAVA to set compiler options using the online SQLJ pre-compiler option. Normally, this option defaults to True, and will cause the SQLJ pre-compiler to attempt to perform semantic checking on our SQLJ code. What this means is that the SQLJ pre-compiler would normally verify each and every referenced database object exists, that the host variable bind types match, and so on. If you would like this checking to be performed at run-time (perhaps the tables your SQLJ code will access are not yet created, but you would like to install your code cleanly), we can use the DBMS_JAVA.SET_COMPILER_OPTIONS routine to disable this type checking.

As an example, we'll use this snippet of code. It attempts to INSERT into a table that does not exist in the database:

tkyte@TKYTE816> create or replace and compile 2 java source named "bad_code" 3 as 4 import java.sql.SQLException; 5 6 public class bad_code extends Object 7 { 8 public static void wont_work() throws SQLException 9 { 10 #sql { 11 insert into non_existent_table values ( 1 ) 12 }; 13 } 14 } 15 /

Java created.

tkyte@TKYTE816> show errors java source "bad_code" Errors for JAVA SOURCE bad_code:

LINE/COL ERROR -------- ----------------------------------------------------------------- 0/0 bad_code:7: Warning: Database issued an error: PLS-00201: identifier 'NON_EXISTENT_TABLE' must be declared

0/0 insert into non_existent_table values ( 1 ) 0/0 ^^^^^^^^^^^^^^^^^^ 0/0 ; 0/0 #sql { 0/0 ^ 0/0 Info: 1 warnings

Now, we'll set the compiler option ONLINE to FALSE. In order to do this, we have to disconnect and connect again. There is an issue whereby the Java run-time will look for the existence of the JAVA$OPTIONS table once it starts up. If this table does not exist, it never attempts to read it again in that session. The DBMS_JAVA.SET_COMPILER_OPTION routine will create this table for us, but only if it is invoked prior to the Java run-time being started. So, we need a 'clean' session for this to work.

In the following example, we establish a new session, and then see that the JAVA$OPTIONS table does not exist. We'll set the compiler option, and see that the table has been created for us. Lastly, we'll create the same Java routine as above, and see that it compiles without warnings this time, due to the compiler option setting:

DBMS_JAVA

1053

tkyte@TKYTE816> disconnect Disconnected from Oracle8i Enterprise Edition Release 8.1.6.0.0 - Production With the Partitioning option JServer Release 8.1.6.0.0 – Production

tkyte@TKYTE816> connect tkyte/tkyte Connected.tkyte@TKYTE816> column value format a10 tkyte@TKYTE816> column what format a10

tkyte@TKYTE816> select * from java$options; select * from java$options * ERROR at line 1: ORA-00942: table or view does not exist

tkyte@TKYTE816> begin 2 dbms_java.set_compiler_option 3 ( what => 'bad_code', 4 optionName => 'online', 5 value => 'false' ); 6 end; 7 /


tkyte@TKYTE816> select * from java$options;

WHAT OPT VALUE ---------- -------------------- ---------- bad_code online false

tkyte@TKYTE816> create or replace and compile 2 java source named "bad_code" 3 as 4 import java.sql.SQLException; 5 6 public class bad_code extends Object 7 { 8 public static void wont_work() throws SQLException 9 { 10 #sql { 11 insert into non_existent_table values ( 1 ) 12 }; 13 } 14 } 15 /

Java created.

tkyte@TKYTE816> show errors java source "bad_code" No errors.

Appendix A

1054

The SET_COMPILER_OPTION takes three inputs in this case:

❑ WHAT – A pattern to be matched against. Normally, Java programs would use packages and hence, the above name would be a.b.c.bad_code, not just bad_code. If you want to set an option for a package a.b.c, you may. Then, anything that matched a.b.c would use this option, unless there was a more specific pattern, which matches this package. Given a WHAT of a.b.c, and a.b.c.bad_code, then a.b.c.bad_code would be used, since it matches more of the name.

❑ OPTIONNAME – One of the three values ONLINE, DEBUG, or ENCODING.

❑ VALUE – The value for that option.

There are two routines related to SET_COMPILER_OPTION. They are:

❑ GET_COMPILER_OPTION – This returns the value of a given compiler option, even if the value is defaulted.

❑ RESET_COMPILER_OPTION – This removes any row from the JAVA$OPTIONS table that matches the WHAT pattern, and the OPTIONNAME.

Here are examples of both in action. We'll begin by using GET_COMPILER_OPTION to see the value of the online option:

tkyte@TKYTE816> set serveroutput on tkyte@TKYTE816> begin 2 dbms_output.put_line 3 ( dbms_java.get_compiler_option( what => 'bad_code', 4 optionName => 'online' ) ); 5 end; 6 / false


and now we'll reset it using RESET_COMPILER_OPTION:

tkyte@TKYTE816> begin 2 dbms_java.reset_compiler_option( what => 'bad_code', 3 optionName => 'online' ); 4 end; 5 /


Now we'll see that GET_COMPILER_OPTION will always return us a value for the compiler option, even though the JAVA$OPTIONS table is now empty (the RESET deleted the row):

tkyte@TKYTE816> begin 2 dbms_output.put_line 3 ( dbms_java.get_compiler_option( what => 'bad_code', 4 optionName => 'online' ) ); 5 end; 6 /

DBMS_JAVA

1055

true


tkyte@TKYTE816> select * from java$options;

no rows selected

SET_OUTPUTThis procedure is a lot like the SQL*PLUS command SET SERVEROUTPUT ON. Just as you need to use it to enable DBMS_OUTPUT, we need to use DBMS_JAVA.SET_OUTPUT to enable the results of System.out.println and System.err.print calls to come to the screen in SQL*PLUS. If you fail to call:

SQL> set serveroutput on size 1000000 SQL> exec dbms_java.set_output( 1000000 )

before running a Java stored procedure in SQL*PLUS, you must be aware that any of its System.out.println messages will be written to a trace file in the directory specified by the USER_DUMP_DEST init.ora parameter on the server. This procedure is truly useful when debugging Java stored procedures, as you can put calls to System.out.println in the code, much as you would put DBMS_OUTPUT.PUT_LINE calls in your PL/SQL. Later, you can disable this in your Java code by redirecting System.out to the 'bit bucket'.

So, if you ever wondered where your System.out calls where going in a Java stored procedure, now you know. They were going to a trace file. Now you can cause that output to come to your screen in SQL*PLUS.

loadjava and dropjava These functions provide PL/SQL APIs to perform the job of the command line utilities loadjava and dropjava. As you might expect with these internal routines, you do not need to specify a -uusername/password, or specify the type of JDBC driver to use – you are already connected! These routines will load the Java objects into the currently logged in schema. The supplied routines are:

PROCEDURE loadjava(options varchar2) PROCEDURE loadjava(options varchar2, resolver varchar2) PROCEDURE dropjava(options varchar2)

We could use this to load the activation8i.zip file, which we also use in the UTL_SMTP section, and more information on JavaMail API can be found at http://java.sun.com/products/javamail/index.html. For example:

sys@TKYTE816> exec dbms_java.loadjava( '-r -v -f -noverify -synonym -g p ublic c:\temp\activation8i.zip' ) initialization complete loading : com/sun/activation/registries/LineTokenizer creating : com/sun/activation/registries/LineTokenizer

Appendix A

1056

loading : com/sun/activation/registries/MailcapEntry creating : com/sun/activation/registries/MailcapEntry loading : com/sun/activation/registries/MailcapFile creating : com/sun/activation/registries/MailcapFile loading : com/sun/activation/registries/MailcapParseException creating : com/sun/activation/registries/MailcapParseException ...

Permission Procedures These are strange ones indeed. Do a DESCRIBE on DBMS_JAVA in the database, and tell me if you see GRANT_PERMISSION in that package. You won't, although you know it must exist since you've seen me use it quite a few times. It does exist, as do a couple of other permission-related functions. We'll describe the GRANT_PERMISSION/REVOKE_PERMISSION here, and its usage. For complete details on using the permissions routines, and all of the options, refer to the Oracle Java Developers Guide. Chapter 5 in this manual, Security for Oracle 8i Java Applications, covers these functions.

In Oracle 8.1.5, the granularity of privileges in Java was very coarse. You either had JAVAUSERPRIV or JAVASYSPRIV, pretty much. This would be like having just RESOURCE and DBA roles in the database – in both cases these roles may offer too much functionality to the end users. With Oracle 8.1.6, the Java in the database supports the Java 2 security classes. Now we have very granular privileges we can grant and revoke, just like the database has for its privilege set. For a general discussion and overview of these permission classes, I'll refer you to this web page http://java.sun.com/j2se/1.3/docs/api/java/security/Permission.html.

So, the two main APIs we'll use here are GRANT_PERMISSION and REVOKE_PERMISSION. The question is, how do I find out what permissions I need? The easiest way is to install the Java, run it, and see what it tells you it needs. For example, I will refer you to the UTL_SMTP section. In there, I create the stored procedure SEND to send mail. I also show you the two grants we need to perform with GRANT_PERMISSION in order to get that to work. The way in which I discover exactly what those grants was to run SEND and see how it fails. For example:

tkyte@TKYTE816> set serveroutput on size 1000000 tkyte@TKYTE816> exec dbms_java.set_output( 1000000 )


tkyte@TKYTE816> declare 2 ret_code number; 3 begin 4 ret_code := send( 5 p_from => '[email protected]', 6 p_to => '[email protected]', 7 p_cc => NULL, 8 p_bcc => NULL, 9 p_subject => 'Use the attached Zip file', 10 p_body => 'to send email with attachments....', 11 p_smtp_host => 'aria.us.oracle.com', 12 p_attachment_data => null, 13 p_attachment_type => null, 14 p_attachment_file_name => null ); 15 if ret_code = 1 then

DBMS_JAVA

1057

16 dbms_output.put_line ('Successful sent message...'); 17 else 18 dbms_output.put_line ('Failed to send message...'); 19 end if; 20 end; 21 / java.security.AccessControlException: the Permission (java.util.Property Permission * read,write) has not been granted by dbms_java.grant_permission to SchemaProtectionDomain(TKYTE|PolicyTableProxy(TKYTE))

Now, that is about as clear as you can get. It is telling me that TKYTE needs the permission type java.util.PropertyPermission with * and read and write. This is how I knew I needed to execute:

sys@TKYTE816> begin 2 dbms_java.grant_permission( 3 grantee => 'TKYTE', 4 permission_type => 'java.util.PropertyPermission', 5 permission_name => '*', 6 permission_action => 'read,write' 7 );

After I did this, I discovered the error:

java.security.AccessControlException: the Permission (java.net.SocketPer mission aria.us.oracle.com resolve) has not been granted by dbms_java.grant_permission to SchemaProtectionDomain(TKYTE|PolicyTableProxy(TKYTE))

and after granting that, it told me I needed CONNECT in addition to RESOLVE. This is how I knew to add:

8 dbms_java.grant_permission( 9 grantee => 'TKYTE', 10 permission_type => 'java.net.SocketPermission', 11 permission_name => '*', 12 permission_action => 'connect,resolve' 13 ); 14 end; 15 /

to the privileges that this schema had. Note that I used * in the permission_name so I could actually resolve and connect to any host, not just my SMTP server.

Now, the opposite of GRANT_PERMISSION is REVOKE_PERMISSION. It operates exactly as you might think. If you pass it the same exact parameters you pass to GRANT_PERMISSION, it will revoke that privilege from the schema.

Appendix A

1058

Summary In this section, we covered using the DBMS_JAVA package to perform various operations for us. We started out by looking at how Oracle, which has a 30-character name limit, handles the very long names used in Java. It hashes a unique, 30-character name for each of the long Java names. The DBMS_JAVA package gives us a function to convert either a short name back into its corresponding long name, or to convert a long name into its short name representation.

Next we investigated using DBMS_JAVA to set, retrieve, and reset various Java compiler options. We saw how this feature uses the JAVA$OPTIONS table to permanently store default compiler options for us, and how we can use it to reset these values back to their defaults. Then we looked briefly at the SET_OUTPUT routine. This redirects the output generated by System.out.println Java calls to a SQL*PLUS or SVRMGRL session, much in the same way SET SERVEROUTPUT ON does for the PL/SQL routine DBMS_OUTPUT. We also saw how the DBMS_JAVA package provides an alternative method of loading Java source code, class files and jars into the database, via a stored procedure call in Oracle8i release 2 (version 8.1.6) and up. Lastly, we looked at the permission procedures provided by this package in Oracle8i release 2 and up. This interface allows us to grant very granular privileges to our Java routines, allowing us to strictly control what they can, and cannot do.

All in all, if you are using Java inside the Oracle database, you will find these routines invaluable in your day-to-day programming.

DBMS_JOB

1059

DBMS_JOB

The DBMS_JOB package allows you to schedule one-off or recurring jobs in your database. A job is a stored procedure, anonymous PL/SQL block, or external procedure written in C or Java. These jobs are run in the background by the server processes themselves. They can be run on a recurring basis (every night at 2am), or one time (run this job right after I commit, and then remove it from the job queue). If you are familiar with the cron or at utilities in UNIX or Windows, you already have a good understanding of the DBMS_JOB package. They are run in the same environment (user, characterset, and so on) they were submitted in (minus roles). Jobs are run in an environment much as a definer rights stored procedure is – without any roles being enabled. We can see this by the following example:

The routines used in this example are explained in detail further down in this section.

tkyte@TKYTE816> create table t ( msg varchar2(20), cnt int );

Table created.

tkyte@TKYTE816> insert into t select 'from SQL*PLUS', count(*) from session_roles;

1 row created.

tkyte@TKYTE816> variable n number tkyte@TKYTE816> exec dbms_job.submit(:n,'insert into t select ''from job'', count(*) from session_roles;');


tkyte@TKYTE816> print n

Appendix A

1060

N ---------- 81

tkyte@TKYTE816> exec dbms_job.run(:n);


tkyte@TKYTE816> select * from t;

MSG CNT -------------------- ---------- from SQL*PLUS 10 from job 0

As you can see, in SQL*PLUS we had 10 roles active, in the job environment, we had none. Typically, since most people submit a stored procedure call as the job, this will not affect anything, since the stored procedure runs without roles in the first place. The only time you might notice this is if you try to schedule a stored procedure to which you have access via a role. This will not work – there are no roles enabled in jobs, ever.

Many times, people ask what the best method is for hiding a username/password associated with a batch job (for example, to analyze tables periodically) that is scheduled via cron, or some utility on Windows NT/2000. They are worried about the password being stored in file (as they should be), or being visible in the ps output on UNIX, and so on. My answer to this is to not use the OS to schedule operations against the database at all, but rather, write a stored procedure that performs your operation, and schedule it using DBMS_JOB. In this fashion, there is no stored username and password, and the job will only actually execute if the database is available. If the database is not available, the job will not run of course, as the database is responsible for running the job.

Another frequent question is, 'How can I speed this up?' You are faced with some long operation, and the end user does not want to wait. Sometimes the answer is that it cannot be sped up. For example, I've been sending e-mails from the database for many years. I've used different mechanisms over time; database pipes, UTL_HTTP, external procedures, and Java. They all worked at about the same speed, but they were slow. It takes a while for SMTP to finish its stuff sometimes. It definitely took too long in my application, where anything longer than quarter of a second is too long. The SMTP send might take 2 to 3 seconds at times. We cannot make it go faster, but we can give it the perception of being faster. Instead of sending the e-mail when the user hit the submit button on the application, we would submit a JOBthat would send the e-mail as soon as we committed. This had two nice side effects. The first was that the operation appeared to be much faster, the second was that it made e-mail 'transactional'. One of the properties of DBMS_JOB is that the job will be visible in the queue, only after you commit. If you roll back, the job is dequeued, and will never be executed. By using DBMS_JOB, not only did we make the application appear faster, but we made it more robust as well. No longer did we send e-mail alerts out from a trigger on an update of a row that got rolled back. Both the row was updated and we sent the e-mail, or the row was not updated and we did not send the e-mail.

So, DBMS_JOB has many uses. It can make 'non transactional' things transactional (like sending an e-mail, or creating a table upon an insert into another table). It can appear to speed things up, especially when you do not need any output from the really slow operation. It can schedule and automate many of the tasks you normally write scripts for outside of the database. It is one of those truly useful packages.

DBMS_JOB

1061

In order for DBMS_JOB to function correctly, we need to do a little set up in the database. There are two init.ora parameters that must be set:

❑ job_queue_interval – Specifies the frequency in seconds by which the job queues will be inspected for jobs that are ready to run. If you schedule a job to run every 30 seconds, but set job_queue_interval to 60 (the default), your job will never run every 30 seconds – it'll run every 60 seconds at best.

❑ job_queue_processes – Specifies the number of background processes available to run jobs. This is an integer number between 0 (the default) and 36. This value may be changed without restarting the database via the ALTER SYSTEM SET JOB_QUEUE_PROCESSES=<nn>command. If this value is left at 0, jobs in the job queue will never run automatically. These job queue processes are visible in the UNIX environment, where they will have the name ora_snpN_$ORACLE_SID where the N will be a number (0, 1, 2, ..., job_queue_processes-1). On Windows, the job queues execute as threads and will not be externally visible.

Many systems run with a value of 60 for job_queue_interval (in other words, check the queues every minute), and 1 for the job_queue_processes (run at most one job at a time). If you use jobs heavily, or make use of features that use the job queues as well (replication, and materialized views are two features that make use of the job queues), you might consider adding an additional job_queue_processes.

Once the job queues are set up to run automatically, we are ready to start using them. The main routine you will use with DBMS_JOB is the SUBMIT routine. Its interface is as follows:

PROCEDURE SUBMIT Argument Name Type In/Out Default? ------------------------------ ----------------------- ------ -------- JOB BINARY_INTEGER OUT WHAT VARCHAR2 IN NEXT_DATE DATE IN DEFAULT INTERVAL VARCHAR2 IN DEFAULT NO_PARSE BOOLEAN IN DEFAULT INSTANCE BINARY_INTEGER IN DEFAULT FORCE BOOLEAN IN DEFAULT

where the arguments to the SUBMIT routine have the following meanings:

❑ JOB – A job identifier. It is system-assigned (it is an OUT only parameter). You can use this to query the USER_JOBS or DBA_JOBS views by job ID to see information about that job. Additionally, some routines such as RUN and REMOVE take the job ID as their only input, to uniquely identify the job to run or be removed.

❑ WHAT – The SQL text of what will be run. It must be a valid PL/SQL statement or block of code. For example, to run a stored procedure P, you might pass the string P; (with the semi-colon) to this routine. Whatever you submit in the WHAT parameter, will be wrapped in the following PL/SQL block:

DECLARE job BINARY_INTEGER := :job; next_date DATE := :mydate; broken BOOLEAN := FALSE;BEGIN

Appendix A

1062

WHAT :mydate := next_date; IF broken THEN :b := 1; ELSE :b := 0; END IF; END;

This is why you need to add the ; to any statement. In order to just replace the WHAT with your code, it will need a semi-colon.

❑ NEXT_DATE – The next (or since we are just submitting, the first) time to run the job. The default is SYSDATE – run as soon as possible (after committing).

❑ INTERVAL – A string containing a date function that calculates the next time to run the job. You can consider this function to be 'selected from dual'. If you pass in the string sysdate+1,the database will in effect execute SELECT sysdate+1 INTO :NEXT_DATE FROM DUAL. See below for some caveats on setting the interval of a job to prevent 'sliding'.

❑ NO_PARSE – Determines whether the WHAT parameter is parsed upon submission. By parsing the string, you can be reasonably sure the string is in fact, executable. In general, NO_PARSEshould always be left with its default of False. When set to True, the WHAT parameter is accepted 'as is' with no validity checking.

❑ INSTANCE – Only meaningful in Parallel Server mode, a mode Oracle can run in, on a loosely coupled cluster of machines. This would specify the instance upon which this job should be executed. By default, this will have a value of ANY_INSTANCE.

❑ FORCE – Again, this is only meaningful in Parallel Server mode. If set to True (the default), you may submit the job with any instance number, even if that instance is not available at the time of job submission. If set to False, submit will fail the request if the associated instance is not available.

There are other entry points into the DBMS_JOB package as well. SUBMIT is the one you will use to schedule a job, and the others allow you to manipulate the scheduled jobs, and perform operations such as RUN it, REMOVE it, and CHANGE it. Below is a listing of the commonly used ones, what they expect as input, and what they do:

Entry Point Inputs Description

REMOVE job number Removes a job from the job queue. You should note that if the job is running, this cannot stop it. It will be removed from the queue so it will not execute again, but it will not stop an executing job. In order to stop a running job, you may use the ALTER SYSTEM command to kill the session.

CHANGE job number

WHAT, NEXT_DATE,INTERVAL,INSTANCE, FORCE

This acts like an UPDATE statement would on the JOBS view. It allows you to change any of the settings of the job.

DBMS_JOB

1063

Entry Point Inputs Description

BROKEN job number

BROKEN (Boolean)

NEXT_DATE

Allows you to 'break' or 'unbreak' a job. A broken job will not execute. A job that has failed 16 times in a row will automatically be set to broken, and Oracle will stop trying to execute it.

RUN job number Runs a job right now in the foreground (in your session). Useful for trying to debug why a job is failing.

Now that we have a working knowledge of how DBMS_JOB works, and what functions are available to us, we'll look at how to run a job once, how to set up a recurring job correctly, and how to monitor our jobs and find out what errors they have encountered.

Running a Job Once Many of the jobs I run are 'one-off' jobs. I use DBMS_JOB much as one would use the & in UNIX, or the start command in Windows, to run a process in the background. The example I gave above with regards to sending e-mail is a good example. I use DBMS_JOB to make the sending of e-mail not only transactional, but also appear to be fast. Here is one implementation of this to demonstrate how to run a job once. We'll start with a small stored procedure to send e-mail using the supplied UTL_SMTP package:

tkyte@TKYTE816> create or replace 2 PROCEDURE send_mail (p_sender IN VARCHAR2, 3 p_recipient IN VARCHAR2, 4 p_message IN VARCHAR2) 5 as 6 -- Note that you have to use a host 7 -- that supports SMTP and that you have access to. 8 -- You do not have access to this host and must change it 9 l_mailhost VARCHAR2(255) := 'aria.us.oracle.com'; 10 l_mail_conn utl_smtp.connection; 11 BEGIN 12 l_mail_conn := utl_smtp.open_connection(l_mailhost, 25); 13 utl_smtp.helo(l_mail_conn, l_mailhost); 14 utl_smtp.mail(l_mail_conn, p_sender); 15 utl_smtp.rcpt(l_mail_conn, p_recipient); 16 utl_smtp.open_data(l_mail_conn ); 17 utl_smtp.write_data(l_mail_conn, p_message); 18 utl_smtp.close_data(l_mail_conn ); 19 utl_smtp.quit(l_mail_conn); 20 end; 21 /

Procedure created.

Appendix A

1064

Now, to time how long this takes, I'll run it twice:

tkyte@TKYTE816> set serveroutput on tkyte@TKYTE816> declare 2 l_start number := dbms_utility.get_time; 3 begin 4 send_mail( '[email protected]', 5 '[email protected]', 'hey there' ); 6 dbms_output.put_line 7 ( round( (dbms_utility.get_time-l_start)/100, 2 ) || 8 ' seconds' ); 9 end; 10 / .81 seconds


tkyte@TKYTE816> / .79 seconds


It looks like it will consistently take the order of 8 tenths of a second to send a mail during the best of times. As far as I am concerned, that is far too long. We can do much better – well, we can 'apparently'do much better. We'll use jobs to give this the appearance of being much faster, and gain the benefit of a 'transactional' e-mail as well.

We will start by creating a table to store the e-mail, and a procedure we could run against it to send e-mail. This procedure will ultimately become our background job. A question is, why am I using a table to store the emails? Why not just pass parameters to the job? The reason is bind variables, and the shared pool. Since all jobs are created using the WHAT parameter, and the database will simply 'execute' this string at run-time, we want to make sure that the WHAT parameter we submit is something that will be in the shared pool. We could easily just submit a job such as:

dbms_job.submit( x, 'send_mail(''[email protected]'', ''[email protected]'', ''hello'' );' );

but that would have the effect of flooding our shared pool with hundreds or thousands of unique statements, killing our performance. Since we plan on sending lots of e-mails (anything more than one is lots, and would mandate the use of bind variables), we need to be able to submit something like:

dbms_job.submit( x, 'background_send_mail( constant );' );

Well, as it turns out, there is an easy way to do this. We simply need to create a table that contains a field for each parameter we really wanted to send to the routine (sender, recipient, and message in this case), plus an ID primary key field. For example:

tkyte@TKYTE816> create table send_mail_data( id number primary key, 2 sender varchar2(255), 3 recipient varchar2(255), 4 message varchar2(4000), 5 senton date default NULL );

Table created.

DBMS_JOB

1065

Here I added an ID column as a primary key, and in this case, a sent on senton column. We'll use this table not only as a place to queue up outgoing e-mails, but also to keep a persistent log of e-mails sent, and when they were sent (very handy, trust me, for when people say 'but I didn't get the notification'). Now all we need to do is figure out a way to generate a key for this table, and get it to our background process using a constant string. Fortunately DBMS_JOB already does that for us. When we schedule a job, it automatically creates a job ID for it, and returns this to us. Since the block of code it wraps around our WHAT parameter includes this job ID, we can simply pass it to ourselves! This means that our FAST_SEND_MAIL routine will look like this:

tkyte@TKYTE816> create or replace 2 PROCEDURE fast_send_mail (p_sender IN VARCHAR2, 3 p_recipient IN VARCHAR2, 4 p_message IN VARCHAR2) 5 as 6 l_job number; 7 begin 8 dbms_job.submit( l_job, 'background_send_mail( JOB );' ); 9 insert into send_mail_data 10 ( id, sender, recipient, message ) 11 values 12 ( l_job, p_sender, p_recipient, p_message ); 13 end; 14 /

Procedure created.

This routine will submit a job, BACKGROUND_SEND_MAIL, and pass it the JOB parameter. If you refer to the WHAT parameter description above, you'll see the block of code includes three local variables we have access to – we are simply passing ourselves one of them. The very next thing we do in this procedure is to insert the e-mail into our QUEUE table, for delivery later. So, DBMS_JOB creates the primary key, and then we insert the primary key with the associated data into this table. That's all we need to do. Now we need to create the BACKGROUND_SEND_MAIL routine and it is simply:

tkyte@TKYTE816> create or replace 2 procedure background_send_mail( p_job in number ) 3 as 4 l_rec send_mail_data%rowtype; 5 begin 6 select * into l_rec 7 from send_mail_data 8 where id = p_job; 9 10 send_mail( l_rec.sender, l_rec.recipient, l_rec.message ); 11 update send_mail_data set senton = sysdate where id = p_job; 12 end; 13 /

Procedure created.

It reads out the data we saved, calls the slow SEND_MAIL routine, and then updates the record, to record the fact that we actually sent the mail. Now, we can run FAST_SEND_MAIL, and see how fast it really is:

Appendix A

1066

tkyte@TKYTE816> declare 2 l_start number := dbms_utility.get_time; 3 begin 4 fast_send_mail( '[email protected]', 5 '[email protected]', 'hey there' ); 6 dbms_output.put_line 7 ( round( (dbms_utility.get_time-l_start)/100, 2 ) || 8 ' seconds' ); 9 end; 10 / .03 seconds


tkyte@TKYTE816> / .02 seconds


As far as our end users are concerned, this FAST_SEND_MAIL is 26 to 40 times faster than the original send mail. It is not really faster, but it just appears to be that much faster (and that is what really counts). The actual sending of the mail will happen in the background after they commit. This is an important note here. If you run this example, make sure you COMMIT when using the DBMS_JOB example, else the e-mail will never get sent. The job will not be visible to the job queue processes until you do (your session can see the job in the USER_JOBS view, but the job queue processes won't see it until you commit). Don't take this as a limitation, it is actually a feature – we've just made e-mail transactional. If you ROLLBACK, so does your send mail. When you COMMIT, it'll be delivered.

Ongoing Jobs The other main use of DBMS_JOB is to schedule recurring jobs in the database. As mentioned previously, many people try to use OS utilities such as cron or at to run jobs in the database, but then encounter the issue of, 'How do I protect the password?' and such. My answer to this is always to use the job queues. In addition to removing the need to store credentials anywhere, this ensures the jobs are only run if in fact, the database is up and running. It will also retry the job time and time again in the event of a failure. For example, if the first time we attempted to run the job, the database link it uses was unavailable, it will put the job back onto the queue, and will retry it later. The database will do this 16 times, waiting a little longer each time, before ultimately marking the job 'broken'. See the next section, Monitoring Jobs And Finding The Errors, for more details on that. These are things cron and at won't do for you. Also, since the jobs are in the database, we can just run queries to find their status – when they last ran, if they ran, and so on. Everything is integrated.

Other Oracle features such as replication and materialized views implicitly use the job queues themselves as part of their day-to-day functioning. The way a snapshot pulls its changes, or a materialized view refreshes, is by the job queues running the stored procedures that perform these operations.

Let's say you wanted to schedule an analysis of all of the tables in a certain schema to take place every night at 3am. The stored procedure for doing such a thing could be:

DBMS_JOB

1067

scott@TKYTE816> create or replace procedure analyze_my_tables 2 as 3 begin 4 for x in ( select table_name from user_tables ) 5 loop 6 execute immediate 7 'analyze table ' || x.table_name || ' compute statistics'; 8 end loop; 9 end; 10 /

Procedure created.

Now, in order to schedule this to run tonight at 3am (tomorrow morning really), and every day thereafter at 3am, we will use the following:

scott@TKYTE816> declare 2 l_job number; 3 begin 4 dbms_job.submit( job => l_job, 5 what => 'analyze_my_tables;', 6 next_date => trunc(sysdate)+1+3/24, 7 interval => 'trunc(sysdate)+1+3/24' ); 8 end; 9 /


scott@TKYTE816> select job, to_char(sysdate,'dd-mon'), 2 to_char(next_date,'dd-mon-yyyy hh24:mi:ss'), 3 interval, what 4 from user_jobs 5 /

JOB TO_CHA TO_CHAR(NEXT_DATE,'D INTERVAL WHAT ---- ------ -------------------- --------------------- ------------------ 33 09-jan 10-jan-2001 03:00:00 trunc(sysdate)+1+3/24 analyze_my_tables;

So, the next date for this job to run will be 3am on the 10th of January. We used a 'real' date for that, not a string as we did for interval. We used a date function so that no matter when it is executed, no matter what time during the day, it will always return 3am tomorrow morning. This is an important fact. We use the same exact function for the INTERVAL parameter as a string. We are using a function that always returns 3am tomorrow, regardless of when it is executed. The reason this is important is to prevent jobs from sliding. It might seem that since the first time the job is run, it'll be run at 3am, we could use an interval simply of sysdate+1. If we ran this at 3am on Tuesday, it should give us 3am on Wednesday. It would – if the jobs were guaranteed to run precisely on time, but they are not. Jobs are processed in the queue sequentially based on their time to be run. If I have one job queue process, and two jobs to be run at 3am, obviously one of them will not run at 3am exactly. It will have to wait for the first to finish to completion, and then it will be executed. Even if I have no overlapping jobs, the job queues are inspected at discrete points in time, say every 60 seconds. I might pick up the job to be run at 3am at 3:00:45am. If it used a simple sysdate+1 function, it might compute its next time to run as '3:00:46am' tomorrow. Tomorrow at 3:00:45am, this job would not be ready to run yet, and would be

Appendix A

1068

picked up on the next inspection of the queue at 3:01:45am. This job would slowly slip over time. Even more dramatic, let's say the tables were being operated on at 3am one morning, so the analysis failed. The stored procedure would fail, and the job queues would retry the job later. Now the job will 'slip' by many minutes for the next day since it runs at a time much later than 3am. For this reason, to prevent the job from slipping, you must use a function that returns a fixed point in time if you want the job to always be scheduled at a particular point in time. If it is important that this job runs at 3am, you must use a function that always returns 3am, and is not dependent on the time of day it is executed.

Many of these 'non-sliding' functions are typically very easy to write – others not so. For example, I was once requested to implement a job that would collect STATSPACK statistics Monday through Friday at 7am, and 3pm only. Well, the INTERVAL for this was certainly non-intuitive, but let's have a look at the pseudo-code:

if it is before 15:00then return TODAY at 15:00 (eg: if we are running at 7am, we want to run at 3pm today)else return today + either 3 (if it is Friday) or 1 (otherwise) at 7amend if

So, what we needed to do then, was turn this logic into a nice DECODE statement – or if that is too complex, I could have used a PL/SQL function to perform the complex logic. I used the interval:

decode(sign(15-to_char(sysdate,'hh24')), 1, trunc(sysdate)+15/24, trunc( sysdate + decode(to_char(sysdate,'d'), 6, 3, 1))+7/24)

The decode starts with the SIGN(15-TO_CHAR(SYSDATE,'HH24')). SIGN is a function that returns -1, 0, or 1 if the resulting number is negative, zero, or positive respectively. If this number was positive, it would imply that it was before 3pm in the afternoon (the hour was less than 15), and hence the next time we should run would be TRUNC(SYSDATE)+15/24 (15 hours after midnight today). On the other hand, if the sign came back 0 or -1, then we would use the TRUNC(SYSDATE + DECODE( TO_CHAR(SYSDATE,'D'), 6,3, 1))+7/24. This would use the DECODE to look at the current day of the week to see if we should add three days (on Friday to get to Monday), or one day (every other day of the week). We would add that many days to SYSDATE, truncate this date back to midnight, and add 7 hours to it.

There are times when a 'sliding' date is OK, and even desired. For example, if you would like a job to collect some statistics from the V$ tables every 30 minutes while the database is up and running, it would be totally appropriate to use an interval of SYSDATE+1/24/2 which adds a half hour to a date.

Custom Scheduling There are times, such as the above, where the NEXT_DATE is hard to compute in a simple SQL statement, or where the next time the job runs is dependent on some complex procedural set of rules. In this case, we can have the job itself set the next date to run.

DBMS_JOB

1069

If you recall from above, the PL/SQL block that runs a job is:

DECLARE job BINARY_INTEGER := :job; next_date DATE := :mydate; broken BOOLEAN := FALSE;BEGIN WHAT :mydate := next_date; IF broken THEN :b := 1; ELSE :b := 0; END IF;END;

We have already seen how we can make use of the fact that JOB is available there in the Running a Job Once section. We can use it as a primary key into a parameter table to make maximum use of shared SQL. Well, we can also make use of the NEXT_DATE variable as well. As you can see in the above block of code, Oracle uses the bind variable :mydate as an input into the routine, to set the NEXT_DATEvariable, but it also retrieves this value after what (your procedure) executes. If your procedure happens to modify this value, the value of NEXT_DATE, Oracle will use this as the next date to run the job. As an example, we'll set up a small procedure P that will write some informative message to a table T, and set it's NEXT_DATE:

tkyte@TKYTE816> create table t ( msg varchar2(80) ); Table created.

tkyte@TKYTE816> create or replace 2 procedure p( p_job in number, p_next_date in OUT date ) 3 as 4 l_next_date date default p_next_date; 5 begin 6 p_next_date := trunc(sysdate)+1+3/24; 7 8 insert into t values 9 ( 'Next date was "' || 10 to_char(l_next_date,'dd-mon-yyyy hh24:mi:ss') || 11 '" Next date IS ' || 12 to_char(p_next_date,'dd-mon-yyyy hh24:mi:ss') ); 13 end; 14 / Procedure created.

Now we will schedule this job using the method from the section, Running a Job Once. That is, without an INTERVAL:

tkyte@TKYTE816> variable n number

tkyte@TKYTE816> exec dbms_job.submit( :n, 'p(JOB,NEXT_DATE);' ); PL/SQL procedure successfully completed.

tkyte@TKYTE816> select what, interval, 2 to_char(last_date,'dd-mon-yyyy hh24:mi:ss') last_date, 3 to_char(next_date,'dd-mon-yyyy hh24:mi:ss') next_date 4 from user_jobs 5 where job = :n 6 /

WHAT INTERVAL LAST_DATE NEXT_DATE------------------------- -------- -------------------- -------------------- p(JOB,NEXT_DATE); null 28-apr-2001 18:23:01

Appendix A

1070

In this case, we send the JOB and the NEXT_DATE as parameters to our procedure. These will be supplied by the job queue at run-time. As you can see, this job has not yet run (LAST_DATE is Null), the INTERVAL is set to null so that the NEXT_DATE will be computed as SELECT NULL FROM DUAL.Normally, this means the job would run once, and be removed from the job queue. However, when this job runs, we'll discover:

tkyte@TKYTE816> exec dbms_job.run( :n );


tkyte@TKYTE816> select * from t;

MSG---------------------------------------------------------------------Next date was "" Next date IS 29-apr-2001 03:00:00

tkyte@TKYTE816> select what, interval, 2 to_char(last_date,'dd-mon-yyyy hh24:mi:ss') last_date, 3 to_char(next_date,'dd-mon-yyyy hh24:mi:ss') next_date 4 from user_jobs 5 where job = :n 6 /

WHAT INTERVAL LAST_DATE NEXT_DATE------------------------- -------- -------------------- -------------------- p(JOB,NEXT_DATE); null 28-apr-2001 18:23:01 29-apr-2001 03:00:00

that the NEXT_DATE is filled in. It is the NEXT_DATE computed in the procedure itself, and the job is still in the queue. As long as this job continues to fill in the NEXT_DATE field, it will remain in the job queue. If it ever exits successfully without setting NEXT_DATE, it will be removed from the queue.

This is very useful for those jobs with hard to compute NEXT_DATE values, or NEXT_DATE values that depend on data found in other database tables.

Monitoring the Jobs and Finding the Errors There are three main views used to monitor jobs in the database. They are simply:

❑ USER_JOBS – A list of all jobs submitted by the currently logged in user. There is also a public synonym, ALL_JOBS that references this view. ALL_JOBS is the same as USER_JOBS.

❑ DBA_JOBS – A comprehensive list of all jobs scheduled in the database.

❑ DBA_JOBS_RUNNING – A list of currently executing jobs.

Everyone has access to USER_JOBS as normal, and the DBA_* views are limited to people with the DBA privilege, or those who have been granted SELECT on these particular views directly. These views will give you information such as:

❑ LAST_DATE/LAST_SEC – Tells you when the job last ran. LAST_DATE is an Oracle date/time. LAST_SEC is a character string that has only the time component (hour:minute:second) formatted into it.

DBMS_JOB

1071

❑ THIS_DATE/THIS_SEC – If the job is currently running, this will be filled in with the time it started execution. Like LAST_DATE/LAST_SEC, THIS_DATE is a date/time, and THIS_SEC is a character string with only the time component.

❑ NEXT_DATE/NEXT_SEC – The time the job is schedule to be executed NEXT.

❑ TOTAL_TIME – The total time in seconds, which the job has spent executing. Includes times from other runs – this is a cumulative count.

❑ BROKEN – A Yes/No flag that shows if a job that is 'broken'. Broken jobs are not run by the job queue processes. A job will 'break' itself after 16 failures. You may use the DBMS_JOB.BROKEN API call to 'break' a job (temporarily prevent it from executing).

❑ INTERVAL – The date function to be evaluated at the beginning of the job's next execution, to determine when to run the job next.

❑ FAILURES – The number of times in a row the job has failed. A successful execution of the job will reset this count be to 0.

❑ WHAT – The body of the job, in other words, what to do.

❑ NLS_ENV – The NLS (National Language Support) environment that the job will be executed in. Includes things such as the language, the date format, the number format, and so on. The entire NLS environment is inherited from the environment that submits the job. If you change this environment and submit a job, the job will run with this modified environment.

❑ INSTANCE – Only valid in Parallel Server mode. This is the ID of the instance the job can execute on, or in DBA_JOBS_RUNNING, the instance it is running on.

Suppose you look into these views and see some jobs with a positive value in the FAILURES column – where would you go to see the error message for that job? It is not stored in the database, rather it can be found in the alert log for the database. For example, let's say you create a procedure such as:

tkyte@TKYTE816> create or replace procedure run_by_jobs 2 as 3 l_cnt number; 4 begin 5 select user_id into l_cnt from all_users; 6 -- other code here 7 end; 8 /

Procedure created.

tkyte@TKYTE816> variable n number tkyte@TKYTE816> exec dbms_job.submit( :n, 'run_by_jobs;' );



Commit complete.

tkyte@TKYTE816> exec dbms_lock.sleep(60);


Appendix A

1072

tkyte@TKYTE816> select job, what, failures 2 from user_jobs 3 where job = :n;

JOB WHAT FAILURES ---------- ------------------------------ ---------- 35 run_by_jobs; 1

If you have more than one user in your database (as all databases do) this procedure will most definitely fail. The SELECT ... INTO will always return too many rows; we have a programming error. Since this happens in the background however, it is hard for us to see what exactly might be wrong. Fortunately the error is recorded in the alert log for the database. If we were to edit that file and go to the bottom, we would find:

Tue Jan 09 13:07:51 2001 Errors in file C:\oracle\admin\tkyte816\bdump\tkyte816SNP0.TRC: ...ORA-12012: error on auto execute of job 35 ORA-01422: exact fetch returns more than requested number of rows ORA-06512: at "SCOTT.RUN_BY_JOBS", line 5 ORA-06512: at line 1

It tells us that job 35 (our job) failed to execute. More importantly, it tells us exactly why it failed; the same error stack you would get if you ran this in SQL*PLUS. This information is crucial to diagnosing why a job is failing. With this information we can fix it and get it to run correctly.

This is pretty much all there is to monitoring jobs. You need to either keep an eye on your alert.log(something your DBA should already be doing), or monitor the DBA_JOBS table from time to time to ensure things are running smoothly.

Summary DBMS_JOB is an excellent facility inside the database for running procedures in the background. It has uses in the automated scheduling of routine tasks such as analyzing your tables, performing some archival operation, cleaning up scratch tables – whatever. It has application functionality in the area of making long running operations 'apparently fast' (and apparently fast is all that matters to the end user really). It removes the need to code OS-dependent scripts to perform database operations on a recurring basis. Even better, it removes the need to hard code usernames and passwords in a script to log into the database. The job always runs as the person who submitted it – no credentials are required. Lastly, unlike an OS scheduling facility, these database jobs run only when the database is actually available. If the system is down when a job is scheduled to run, it will not run (obviously, if the database isn't up, the job queues are not up). All in all, DBMS_JOB is a robust facility for which I've found many uses.

DBMS_LOB

1073

DBMS_LOB

DBMS_LOB is a package supplied to manipulate Large OBjects (LOBs) in the database. LOBs are new data types available with Oracle 8, and upwards. LOBs support the storage, and retrieval of up to 4 GB of arbitrary data in a single database column. They replace the, now deprecated, data types LONG and LONG RAW. LONG types in Oracle had many shorting comings, such as:

❑ You could only have one per table

❑ You could not manipulate them in a stored procedure once they grew beyond 32 KB

❑ You could not piece-wise modify them readily

❑ Many database operations, such as INSERT INTO T SELECT LONG_COL FROM T2, were not supported

❑ You could not reference them in a WHERE clause

❑ You could not replicate them

❑ And so on...

The LOB data type overcomes all of these limitations.

Rather than go over each and every function/procedure of the DBMS_LOB package (there are some 25 of them), I am going to answer the most common questions that come up regarding using the DBMS_LOB package and LOBs. Much of it is either self-explanatory, or is well covered in the standard Oracle documentation. For LOBsthere are two main documents you are concerned with:

Appendix A

1074

❑ Oracle8i Supplied PL/SQL Packages Reference – An overview of the DBMS_LOB package, and every procedure within, along with a definition of all of the inputs and outputs. Handy to have for reference purposes. You should give this a quick read through to get an understanding of the functions you can perform on LOBs.

❑ Oracle8i Application Developer's Guide – Large Objects (LOBs) – An entire document dedicated to explaining how to program using LOBs in various languages and environments. A must read for the developer who will be using LOBs.

Additionally, many of the nuances of working with LOBs is language-specific. How you do something in Java, will be different in C, will be different in PL/SQL, and so on. To this end, Oracle Corporation has actually developed an Application Developer's Guide by language, for languages such as PL/SQL, OCI, Pro*C, COBOL, VB, and Java detailing how LOBs interact with each language. There is also a comprehensive Application Developer's Guide on LOBs, as mentioned above, that is useful, regardless of the language used. I would urge anyone who is considering using LOBs in their applications to read this document, as well as the language-specific guide for their language of choice. These documents answer most of the questions you will ask.

What I will cover here are the answers to the frequently asked questions about LOBs, from, 'How can I show them on the web?', to, 'How can I convert between BLOBs and CLOBs?' – things that aren't covered so well in the standard documentation. LOBs are extremely easy to use once you familiarize yourself with the DBMS_LOBpackage (see the Oracle 8i Supplied PL/SQL Packages Reference for an overview of this package) and if you haven't done so already, you should do so now before reading this section as it assumes you are ready to go and do things with LOBs.

How do I Load LOBs? There are quite a few methods available for loading LOBs. In Chapter 9 on Data Loading for example, I demonstrate how the SQLLDR tool may be used to load LOBs into the database. Additionally, the Application Developer's Guide for each language provided by Oracle demonstrate how to create and retrieve a LOB using a specific host language (it's a little different in each). In my opinion however, if I had a directory full of files to load, the use of a BFILE, a DIRECTORY object, and the LOADFROMFILE routine would by far be the way to go.

In Chapter 9 on Data Loading, we covered the topic of using DBMS_LOB.LOADFROMFILE in depth. I will refer you to that section for all of the details. Also, the section on Conversions here, contains a full example of loading a CLOB using LOADFROMFILE.

substrThis is just a quick note on the substr function provided by the DBMS_LOB package. Every other substrfunction I have ever seen (including the one provided with SQL and PL/SQL) has the following arguments in the following order:

substr( the-string, from-character, for-number-of-characters );

So, the substr('hello', 3, 2) would be ll – the third and fourth characters (from character 3, for 2 characters). DBMS_LOB.SUBSTR however, defines them as:

dbms_lob.substr( the-lob, for-number-of-characters, from-character )

DBMS_LOB

1075

So that same substr with DBMS_LOB would return ell. A very small simple test confirms this behavior:

tkyte@TKYTE816> create table t ( str varchar2(10), lob clob );

Table created.

tkyte@TKYTE816> insert into t values ( 'hello', 'hello' );

1 row created.

tkyte@TKYTE816> select substr( str, 3, 2 ), 2 dbms_lob.substr( lob, 3, 2) lob 3 from t 4 /

SU LOB -- -------------------- ll ell

I am constantly doing it backwards myself. It is just one of those things we have to remember to watch out for!

SELECT FOR UPDATE and Java In order to modify a database-based LOB (not a temporary LOB), the row that contains the LOB in the database must be locked by our session. This is a common point of confusion to Java/JDBC programmers. Consider the small Java program below. It simply:

❑ Inserts a record (hence you would assume its locked)

❑ Reads out the LOB locator just created

❑ Attempts to use this LOB locator with DBMS_LOB.WRITEAPPEND

As it turns out – this Java program will always encounter the error:

java Test java.sql.SQLException: ORA-22920: row containing the LOB value is not locked ORA-06512: at "SYS.DBMS_LOB", line 715 ORA-06512: at line 1

Apparently, the LOB we inserted is not locked by our session any more. This is an unfortunate side effect of the default 'transactional' mode of JDBC – by default it does not support transactions! After every statement, it commits work immediately. In the following application, unless you add conn.setAutoCommit (false);immediately after the getConnection – it will fail. That one line of code should (in my opinion) be the first line of code after every connect in a JDBC program!

import java.sql.*; import java.io.*; import oracle.jdbc.driver.*; import oracle.sql.*;

// You need a table: // create table demo ( id int primary key, theBlob blob ); // in order for this application to execute.

Appendix A

1076

class Test {

public static void main (String args []) throws SQLException , FileNotFoundException, IOException { DriverManager.registerDriver (new oracle.jdbc.driver.OracleDriver());

Connection conn = DriverManager.getConnection ("jdbc:oracle:thin:@aria:1521:ora8i", "scott", "tiger");

// If this program is to work, uncomment this next line! // conn.setAutoCommit(false);

Statement stmt = conn.createStatement();

// Insert an empty BLOB into the table // create it new for the very first time. stmt.execute ( "insert into demo (id,theBlob) " + "values (1,empty_blob())" );

// Now, we will read it back out so we can // load it. ResultSet rset = stmt.executeQuery ("SELECT theBlob " + "FROM demo "+ "where id = 1 ");

if(rset.next()) { // Get the BLOB to load into. BLOB l_mapBLOB = ((OracleResultSet)rset).getBLOB(1);

// Here is the data we will load into it. File binaryFile = new File("/tmp/binary.dat"); FileInputStream instream = new FileInputStream(binaryFile);

// We will load about 32 KB at a time. That's // the most dbms_lob can handle (PL/SQL limit). int chunk = 32000; byte[] l_buffer = new byte[chunk];

int l_nread = 0;

// We'll use the easy writeappend routine to add // our chunk of file to the end of the BLOB. OracleCallableStatement cstmt = (OracleCallableStatement)conn.prepareCall ( "begin dbms_lob.writeappend( :1, :2, :3 ); end;" );

// Read and write, read and write, until done. cstmt.registerOutParameter( 1, OracleTypes.BLOB ); while ((l_nread= instream.read(l_buffer)) != -1) { cstmt.setBLOB( 1, l_mapBLOB ); cstmt.setInt( 2, l_nread ); cstmt.setBytes( 3, l_buffer );

DBMS_LOB

1077

cstmt.executeUpdate();

l_mapBLOB = cstmt.getBLOB(1); } // Close up the input file and callable statement. instream.close(); cstmt.close(); } // Close out the statements. rset.close(); stmt.close(); conn.close (); }

}

This is a general shortcoming of JDBC, and it affects LOB operations in particular. I cannot tell you how many people are surprised to find that an API would presume to commit for them – something that must be done by the application itself. Only an ex-ODBC programmer might be expecting that! The same thing will happen in ODBC in its default mode of auto commit as well.

Conversions Frequently, people have their data in a BLOB, and need it for some reason to appear as a CLOB. Typically, someone has loaded a mixture of text and binary data into a BLOB column, and this person would like to parse the text. Parsing the BLOB is difficult since the database will constantly try to convert the raw BLOB data into hexadecimal, which is not the desired effect. In other cases, people have data in a LONG or LONG RAW that they would like to process as if it were a CLOB or BLOB, given the APIs for these types are so superior to anything available for LONGs and LONG RAWs.

Fortunately, these conversions are easy to solve. We can convert:

❑ BLOB data into VARCHAR2

❑ VARCHAR2 into RAW

❑ LONGs into CLOBs

❑ LONG RAWs into BLOBs

We'll deal first with the BLOB to VARCHAR2, and vice versa, conversion and then look at the LONG to CLOB, or LONG RAW to BLOB conversion.

From BLOB to VARCHAR2 and Back Again The UTL_RAW package has two very handy routines in it for us to use with BLOBs. We'll cover this package in more depth later on in thesection on UTL_RAW. These two routines are:

❑ CAST_TO_VARCHAR2 – Takes a RAW input and just changes the data type from RAW to VARCHAR2. No conversion of data actually happens, it is all really just a data type change.

❑ CAST_TO_RAW – Take a VARCHAR2 as input and makes it RAW. It doesn't change the data, just changes the data type again.

Appendix A

1078

So, if you know the BLOB you have is actually text information, and in the right characterset, and everything, these functions are truly useful. Let's say someone used the LOADFROMFILE routine we briefly looked at earlier to load a series of files into a BLOB column. We would like to have the ability to view them in SQL*PLUS (masking out any 'bad' characters that would cause SQL*PLUS to behave improperly). We can use UTL_RAW to do this for us. First, we will load up some files into a DEMO table:

scott@DEV816> create table demo 2 ( id int primary key, 3 theBlob blob 4 ) 5 /

Table created.

scott@DEV816> create or replace directory my_files as '/export/home/tkyte';

Directory created.

scott@DEV816> create sequence blob_seq;

Sequence created.

scott@DEV816> create or replace 2 procedure load_a_file( p_dir_name in varchar2, 3 p_file_name in varchar2 ) 4 as 5 l_blob blob; 6 l_bfile bfile; 7 begin 8 -- First we must create a LOB in the database. We 9 -- need an empty CLOB, BLOB, or a LOB created via the 10 -- CREATE TEMPORARY API call to load into. 11 12 insert into demo values ( blob_seq.nextval, empty_blob() ) 13 returning theBlob into l_Blob; 14 15 -- Next, we open the BFILE we will load 16 -- from. 17 18 l_bfile := bfilename( p_dir_name, p_file_name ); 19 dbms_lob.fileopen( l_bfile ); 20 21 22 -- Then, we call LOADFROMFILE, loading the CLOB we 23 -- just created with the entire contents of the BFILE 24 -- we just opened. 25 dbms_lob.loadfromfile( l_blob, l_bfile, 26 dbms_lob.getlength( l_bfile ) ); 27 28 -- Close out the BFILE we opened to avoid running 29 -- out of file handles eventually. 30 31 dbms_lob.fileclose( l_bfile ); 32 end; 33 /

DBMS_LOB

1079

Procedure created.

scott@DEV816> exec load_a_file( 'MY_FILES', 'clean.sql' );


scott@DEV816> exec load_a_file( 'MY_FILES', 'expdat.dmp' );


So, now I have two files loaded up. One is the script I am working on right here – clean.sql. The other is some expdat.dmp (export file) I have. Now I will write a routine that is callable from SQL to allow me to view any arbitrary 4000 byte slice of a BLOB in SQL*PLUS. We can only view 4,000 bytes, as this is a SQL limitation on the size of a VARCHAR2 data type. The CLEAN function below works much as SUBSTR would work on a regular string, but it takes a BLOB as input and optionally FROM_BYTE and FOR_BYTES arguments. These allow us to pick off an arbitrary substring of the BLOB to display. Note here how we use UTL_RAW.CAST_TO_VARCHAR2 to convert the RAW into a VARCHAR2. If we did not use this routine, the RAWbytes would be converted into hexadecimal before being placed into the VARCHAR2 field. By using this routine, we simply 'change the data type' from RAW to VARCHAR2, and no translation whatsoever takes place:

scott@DEV816> create or replace 2 function clean( p_raw in blob, 3 p_from_byte in number default 1, 4 p_for_bytes in number default 4000 ) 5 return varchar2 6 as 7 l_tmp varchar2(8192) default 8 utl_raw.cast_to_varchar2( 9 dbms_lob.substr(p_raw,p_for_bytes,p_from_byte) 10 ); 11 l_char char(1); 12 l_return varchar2(16384); 13 l_whitespace varchar2(25) default 14 chr(13) || chr(10) || chr(9); 15 l_ws_char varchar2(50) default 16 'rnt'; 17 18 begin 19 for i in 1 .. length(l_tmp) 20 loop 21 l_char := substr( l_tmp, i, 1 ); 22 23 -- If the character is 'printable' (ASCII non-control) 24 -- then just add it. If it happens to be a \, add another 25 -- \ to it, since we will replace newlines and tabs with 26 -- \n and \t and such, so need to be able to tell the 27 -- difference between a file with \n in it, and a newline. 28 29 if ( ascii(l_char) between 32 and 127 ) 30 then 31 l_return := l_return || l_char; 32 if ( l_char = '\' ) then 33 l_return := l_return || '\';

Appendix A

1080

34 end if; 35 36 -- If the character is a 'whitespace', replace it 37 -- with a special character like \r, \n, \t 38 39 elsif ( instr( l_whitespace, l_char ) > 0 ) 40 then 41 l_return := l_return || 42 '\' || 43 substr( l_ws_char, instr(l_whitespace,l_char), 1 ); 44 45 -- Else for all other non-printable characters 46 -- just put a '.'. 47 48 else 49 l_return := l_return || '.'; 50 end if; 51 end loop; 52 53 -- Now, just return the first 4000 bytes as 54 -- this is all that the SQL will let us see. We 55 -- might have more than 4000 characters since CHR(10) will 56 -- become \n (double the bytes) and so, this is necessary. 57 58 return substr(l_return,1,4000); 59 end; 60 /

Function created.

scott@DEV816> select id, 2 dbms_lob.getlength(theBlob) len, 3 clean(theBlob,30,40) piece, 4 dbms_lob.substr(theBlob,40,30) raw_data 5 from demo;

ID LEN PIECE RAW_DATA---------- ----- -------------------- ------------------------------ 1 3498 \ndrop sequence 0A64726F702073657175656E636520 blob_seq;\n\ncreate 626C6F625F7365713B0A0A63726561 table d 7465207461626C652064

2 2048 TE\nRTABLES\n1024\n0 54450A525441424C45530A31303234 \n28\n4000\n........ 0A300A32380A343030300A0001001F ...... 00010001000000000000

As you can see, we can view the textual component of the BLOB in SQL*PLUS as clear text now using CLEAN.If we just use DBMS_LOB.SUBSTR, which returns a RAW, we get a hexadecimal dump. Looking at the hexadecimal dump, we can see the first byte of the first BLOB is 0A, which is a CHR(10), which is a newline. We can see in our text dump of the BLOB, that our CLEAN function converted the 0A into \n (newline). This just confirms our routine is working as expected. Further, in the second BLOB, we can see many binary zeroes (hexadecimal 00) in the raw dump of the expdat.dmp data. We can see that we turned them into . in our CLEAN function, as many of these special characters, if dumped to the terminal directly, would display in a non-sensical fashion.

DBMS_LOB

1081

In addition to the CAST_TO_VARCHAR2 function, UTL_RAW contains the CAST_TO_RAW function. As demonstrated above, you may have plain ASCII text stored in a BLOB. If you want to be able to use STRINGsto update this data, you would have to know how to encode the string in hexadecimal. For example:

scott@DEV816> update demo 2 set theBlob = 'Hello World' 3 where id = 1 4 / set theBlob = 'Hello World' * ERROR at line 2: ORA-01465: invalid hex number

does not work. The implicit conversion from VARCHAR2 to RAW assumes the string Hello World is a string of hexadecimal characters. Oracle would take the first two bytes, convert them from hexadecimal to decimal, and assign this number as byte 1 of the RAW data, and so on. We could either take the time to figure out what the hexadecimal representation of Hello World was, or we could simply cast our VARCHAR2 into a RAW type – just change the data type and don't change the bytes contained therein. For example:

scott@DEV816> update demo 2 set theBlob = utl_raw.cast_to_raw('Hello World') 3 where id = 1 4 /

1 row updated.

scott@DEV816> commit;

Commit complete.

scott@DEV816> select id, 2 dbms_lob.getlength(theBlob) len, 3 clean(theBlob) piece, 4 dbms_lob.substr(theBlob,40,1) raw_data 5 from demo 6 where id =1;

ID LEN PIECE RAW_DATA---------- ----- -------------------- ------------------------------ 1 11 Hello World 48656C6C6F20576F726C64

Using UTL_RAW.CAST_TO_RAW('Hello World') is typically much easier than converting Hello Worldinto 48656C6C6F20576F726C64.

Converting From LONG/LONG RAW to a LOB Converting from a LONG or LONG RAW to a LOB is rather straightforward. The supplied SQL function TO_LOBdoes the job for us. TO_LOB is a rather restricted function however, in that:

❑ It can only be used in an INSERT or CREATE TABLE AS SELECT statement.

❑ It can only be used in SQL, not in PL/SQL.

Appendix A

1082

The ramification of the first restriction is that you cannot perform a statement such as:

alter table t add column clob_column; update t set clob_column = to_lob( long_column ); alter table t drop column long_column;

The above will fail with:

ORA-00932: inconsistent datatypes

during the UPDATE. In order to bulk convert existing tables with LONGs/LONG RAWs, you must create a new table. This is probably for the best in any case, since LONGs and LONG RAWs were stored 'inline', in other words, with the table data itself. If we simply converted them to LOBs and then removed the LONG column, we would leave the table in pretty bad shape. There would be lots of allocated, but not used, space in the table now. Rebuilding these objects is for the best.

The ramification of the second restriction is that you cannot use TO_LOB in a PL/SQL block. In order to use TO_LOB in PL/SQL we must use dynamic SQL. We'll demonstrate this in a moment.

We will take a look at two ways of using TO_LOB in the following examples. One is in the use of the TO_LOBfunction in a CREATE TABLE AS SELECT or INSERT INTO statement. The other is useful when the source data must remain in a LONG or LONG RAW column for the time being. For example, a legacy application needs it to be in a LONG. You would like other applications to be able to access it as a LOB, giving PL/SQL the opportunity to have full access to it via the piece-wise DBMS_LOB functions, such as READ and SUBSTR for example.

We'll start by synthesizing some LONG and LONG RAW data:

ops$tkyte@DEV816> create table long_table 2 ( id int primary key, 3 data long 4 ) 5 /

Table created.

ops$tkyte@DEV816> create table long_raw_table 2 ( id int primary key, 3 data long raw 4 ) 5 /

Table created.

ops$tkyte@DEV816> declare 2 l_tmp long := 'Hello World'; 3 l_raw long raw; 4 begin 5 while( length(l_tmp) < 32000 ) 6 loop 7 l_tmp := l_tmp || ' Hello World'; 8 end loop; 9 10 insert into long_table 11 ( id, data ) values

DBMS_LOB

1083

12 ( 1, l_tmp ); 13 14 l_raw := utl_raw.cast_to_raw( l_tmp ); 15 16 insert into long_raw_table 17 ( id, data ) values 18 ( 1, l_raw ); 19 20 dbms_output.put_line( 'created long with length = ' || 21 length(l_tmp) ); 22 end; 23 / created long with length = 32003


Performing a Mass One-Time Conversion Illustration So, we have two tables, each with one row and either a LONG or a LONG RAW column. We can do a conversion from LONG to CLOB as easily as a CREATE TABLE AS SELECT statement now:

ops$tkyte@DEV816> create table clob_table 2 as 3 select id, to_lob(data) data 4 from long_table;

Table created.

Additionally, we could have created the table at another point in time, and use the INSERT INTO variant to populate this table:

ops$tkyte@DEV816> insert into clob_table 2 select id, to_lob(data) 3 from long_table;

1 row created.

The following simply shows that the TO_LOB function does not operate in a PL/SQL block, and that this is to be expected:

ops$tkyte@DEV816> begin 2 insert into clob_table 3 select id, to_lob(data) 4 from long_table; 5 end; 6 / begin*ERROR at line 1: ORA-06550: line 3, column 16: PLS-00201: identifier 'TO_LOB' must be declared ORA-06550: line 2, column 5: PL/SQL: SQL Statement ignored

Appendix A

1084

This is easy to work around using dynamic SQL (you will just have to dynamically execute the INSERT, not statically as above). Now that we've seen how to convert a LONG or LONG RAW into a CLOB or BLOB, we'll consider performance of the conversion. Typically, tables with LONGs and LONG RAWs are huge. By definition they are big tables – we are using them to store very large objects. They are in many cases, many gigabytes in size. The question is, how can we perform a bulk conversion in a timely fashion? I suggest using the following features:

❑ Unrecoverable operations such as a direct path INSERT and NOLOGGING LOBs

❑ Parallel DML (parallel INSERTs specifically)

❑ Parallel query

Here is an example using these features. I have a rather large IMAGE table, which contains many hundreds of uploaded files (uploaded from the Web). The fields in this table are the NAME of the document, the MIME_TYPE(for example, application/MS-Word), the IMG_SIZE of the document in bytes, and finally the document itself in a LONG RAW. I would like to convert this table into an equivalent table where the document is stored in a BLOB column. I might start by creating the new table:

scott@DEV816> CREATE TABLE "SCOTT"."T" 2 ("NAME" VARCHAR2(255), 3 "MIME_TYPE" VARCHAR2(255), 4 "IMG_SIZE" NUMBER, 5 "IMAGE" BLOB) 6 PCTFREE 0 PCTUSED 40 7 INITRANS 1 8 MAXTRANS 255 9 NOLOGGING 10 TABLESPACE "USERS" 11 LOB ("IMAGE") STORE AS 12 (TABLESPACE "USERS" 13 DISABLE STORAGE IN ROW CHUNK 32768 14 PCTVERSION 10 15 NOCACHE 16 NOLOGGING 17 ) ;

Table created.

Notice the TABLE and the LOB are NOLOGGING – this is important. You can alter them instead of creating them this way. Now, to convert the data from the existing IMAGE table, I would execute:

scott@DEV816> ALTER SESSION ENABLE PARALLEL DML;

Session altered.

scott@DEV816> INSERT /*+ APPEND PARALLEL(t,5) */ INTO t 2 SELECT /*+ PARALLEL(long_raw,5) */ 3 name, mime_type, img_size, to_lob(image) 4 FROM long_raw;

DBMS_LOB

1085

This performs a direct path, parallel insert into non-logged BLOBs. As a matter of comparison, I ran the INSERTINTO with and without logging enabled, and this was the result (using a subset of rows to be converted):

scott@DEV816> create table t 2 as 3 select name, mime_type, img_size, to_lob(image) image 4 from image where 1=0; Table created.

scott@DEV816> set autotrace on

scott@DEV816> insert into t 2 select name, mime_type, img_size, to_lob(image) image 3 from image; 99 rows created.

Execution Plan ---------------------------------------------------------- 0 INSERT STATEMENT Optimizer=CHOOSE 1 0 TABLE ACCESS (FULL) OF 'IMAGE'

Statistics---------------------------------------------------------- 1242 recursive calls 36057 db block gets 12843 consistent gets 7870 physical reads 34393500 redo size 1006 bytes sent via SQL*Net to client 861 bytes received via SQL*Net from client 4 SQL*Net roundtrips to/from client 2 sorts (memory) 0 sorts (disk) 99 rows processed

Note how that generated 34 MB of redo (if you add up the bytes of the 99 images, then I have 32 MB of data). Now, using the CREATE for T I have above with the NOLOGGING clauses and just using a direct path insert, I find:

scott@DEV816> INSERT /*+ APPEND */ INTO t 2 SELECT name, mime_type, img_size, to_lob(image) 3 FROM image;

99 rows created.

Execution Plan ---------------------------------------------------------- 0 INSERT STATEMENT Optimizer=CHOOSE 1 0 TABLE ACCESS (FULL) OF 'IMAGE'

Statistics---------------------------------------------------------- 1242 recursive calls 36474 db block gets 13079 consistent gets 6487 physical reads

Appendix A

1086

1355104 redo size 1013 bytes sent via SQL*Net to client 871 bytes received via SQL*Net from client 4 SQL*Net roundtrips to/from client 2 sorts (memory) 0 sorts (disk) 99 rows processed

I generated about 1 MB of log. This conversion ran dramatically faster, and generated much less redo log. Of course, as is the case with all unrecoverable operations, you must ensure that a database backup takes place in the near future to ensure the recoverability of these new objects. Otherwise, you may find yourself reconverting the converted data in the event of a disk failure!

The above example is not actually executable by itself. I just happened to have an IMAGE table lying around, which had about 200 MB of data in it. This is used to demonstrate large, one-time conversions, and the differences that NOLOGGING clauses had on the size of the redo log generated.

Performing an 'on the fly' Conversion In many cases, you would like to be able to access (read) a LONG or LONG RAW from various environments, but find that you cannot. For example, when using PL/SQL, if the LONG RAW exceeds 32KB in size, you will find it to be quite impossible to access it. Other languages and interfaces have issues with LONGs and LONG RAWs as well. Well, using the TO_LOB function and a temporary table, we can easily convert a LONG or LONG RAW into a CLOB or BLOB on the fly. This is very handy for example when using OAS4.x or WebDB with its file upload functionality. These tools will upload documents over the Web into a database table, but unfortunately, the data type of the column they upload into is a LONG RAW. This makes accessing this column via PL/SQL virtually impossible. The functions below show how to provide access to this data via a BLOB, a snap.

We will start with a temporary table to hold the converted CLOB/BLOB, and a sequence to identify our row:

ops$tkyte@DEV816> create global temporary table lob_temp 2 ( id int primary key, 3 c_lob clob, 4 b_lob blob 5 ) 6 /

Table created.

ops$tkyte@DEV816> create sequence lob_temp_seq;

Sequence created.

Now we'll create functions TO_BLOB and TO_CLOB. These functions use the following logic to convert a LONGor LONG RAW on the fly:

❑ The end user of this function will select the row ID from the table with the LONG or LONG RAW,instead of selecting the LONG or LONG RAW column. They will pass to us the column name of the LONG column, the table name and the row ID, identifying the row they want.

❑ We will get a sequence number to identify the row we will create in the temporary table.

DBMS_LOB

1087

❑ Using dynamic SQL, we will TO_LOB their LONG or LONG RAW column. The use of dynamic SQL not only makes this routine generic (works for any LONG column in any table), but it also solves the issue that TO_LOB cannot be invoked in PLSQL directly.

❑ We read the BLOB or CLOB we just created, back out, and return it to the caller.

Here is the code for TO_BLOB and TO_CLOB:

ops$tkyte@DEV816> create or replace 2 function to_blob( p_cname in varchar2, 3 p_tname in varchar2, 4 p_rowid in rowid ) return blob 5 as 6 l_blob blob; 7 l_id int; 8 begin 9 select lob_temp_seq.nextval into l_id from dual; 10 11 execute immediate 12 'insert into lob_temp (id,b_lob) 13 select :id, to_lob( ' || p_cname || ' ) 14 from ' || p_tname || 15 ' where rowid = :rid ' 16 using IN l_id, IN p_rowid; 17 18 select b_lob into l_blob from lob_temp where id = l_id ; 19 20 return l_blob; 21 end; 22 /

Function created.

ops$tkyte@DEV816> create or replace 2 function to_clob( p_cname in varchar2, 3 p_tname in varchar2, 4 p_rowid in rowid ) return clob 5 as 6 l_clob clob; 7 l_id int; 8 begin 9 select lob_temp_seq.nextval into l_id from dual; 10 11 execute immediate 12 'insert into lob_temp (id,c_lob) 13 select :id, to_lob( ' || p_cname || ' ) 14 from ' || p_tname || 15 ' where rowid = :rid ' 16 using IN l_id, IN p_rowid; 17 18 select c_lob into l_clob from lob_temp where id = l_id ; 19 20 return l_clob; 21 end; 22 /

Function created.

Appendix A

1088

Now, to demonstrate their usage, we can use a simple PL/SQL block. We convert the LONG RAW into a BLOB,and show its length and a little of the data it holds:

ops$tkyte@DEV816> declare 2 l_blob blob; 3 l_rowid rowid; 4 begin 5 select rowid into l_rowid from long_raw_table; 6 l_blob := to_blob( 'data', 'long_raw_table', l_rowid ); 7 dbms_output.put_line( dbms_lob.getlength(l_blob) ); 8 dbms_output.put_line( 9 utl_raw.cast_to_varchar2( 10 dbms_lob.substr(l_blob,41,1) 11 ) 12 ); 13 end; 14 / 32003Hello World Hello World Hello World Hello


The code to test TO_CLOB is virtually the same, with the exception that we do not need to utilize the UTL_RAWfunctionality:

ops$tkyte@DEV816> declare 2 l_clob clob; 3 l_rowid rowid; 4 begin 5 select rowid into l_rowid from long_table; 6 l_clob := to_clob( 'data', 'long_table', l_rowid ); 7 dbms_output.put_line( dbms_lob.getlength(l_clob) ); 8 dbms_output.put_line( dbms_lob.substr(l_clob,41,1) ); 9 end; 10 / 32003Hello World Hello World Hello World Hello


How to Write a BLOB/CLOB to Disk This functionality is missing from the DBMS_LOB package. We have methods to load LOBs from files, but not create a file from a LOB. I mention it here, simply because we have a solution for it in this book. If you refer to Chapters 18 and 19 on C-Based External Procedures and Java Stored Procedures, I provide both the C and Java code for an external procedure that will write any BLOB, CLOB, or TEMPORARY LOB to a file on the server's file system. Both implementations perform the same function – just using different languages. Use whichever is appropriate with your server (for example, if you do not have the Java option, but you have Pro*C and a C compiler, then the C-based external procedure would be more appropriate for you).

DBMS_LOB

1089

Displaying a LOB on the Web Using PL/SQL This is a frequently asked question. This example assumes you have one of the following installed and running on your system:

❑ WebDB's lightweight listener.

❑ OAS 2.x, 3.x or 4.x with the PL/SQL cartridge.

❑ iAS with the mod_plsql module.

Without one of the above three, this example will not work. It relies on the PL/SQL Web Toolkit (commonly referred to as the HTP functions), and the PL/SQL cartridge or module.

Another assumption we must make is that the character set of the web server (the client of the database) is the same as the database itself. This is due to the fact that the PL/SQL cartridge or module uses VARCHAR2s as the data type to return pages from the database. If the client's character set (the web server is the client in this case) is different from the database's character set, then character set conversion will take place. This conversion will typically corrupt a BLOB. For example, say you are running the web server on Windows NT. The typical character set for a client on Windows NT is WE8ISO8859P1 – Western European 8bit. Now, say the database is running on Solaris. The default and typical character set on that platform is US7ASCII – a 7bit character set. If you attempt to return a BLOB through a VARCHAR2 interface given these two character sets, you'll find that the 'high bit' is stripped off of the data as it comes out of the database. The data will be changed. Only if both the client (the web server) and the database server have the same character set will the data be passed 'as is', unchanged.

So, given that you have the above two assumptions satisfied, we can now see how to use the PL/SQL web toolkit to display a BLOB on the Web. We'll continue using the example from above (conversions) with the DEMO table. We'll load one more file:

ops$tkyte@DEV816> exec load_a_file( 'MY_FILES', 'demo.gif' );


a GIF file. Now, we need a package that can retrieve this GIF, and display it on the Web. It might look like this:

ops$tkyte@DEV816> create or replace package image_get 2 as 3 -- You might have a procedure named 4 -- after each type of document you want 5 -- to get, for example: 6 -- procedure pdf 7 -- procedure doc 8 -- procedure txt 9 -- and so on. Some browsers (MS IE for example) 10 -- seem to prefer file extensions over 11 -- mime types when deciding how to handle 12 -- documents. 13 procedure gif( p_id in demo.id%type ); 14 end; 15 /

Package created.

Appendix A

1090

ops$tkyte@DEV816> create or replace package body image_get 2 as 3 4 procedure gif( p_id in demo.id%type ) 5 is 6 l_lob blob; 7 l_amt number default 32000; 8 l_off number default 1; 9 l_raw raw(32000); 10 begin 11 12 -- Get the LOB locator for 13 -- our document. 14 select theBlob into l_lob 15 from demo 16 where id = p_id; 17 18 -- Print out the mime header for this 19 -- type of document. 20 owa_util.mime_header( 'image/gif' ); 21 22 begin 23 loop 24 dbms_lob.read( l_lob, l_amt, l_off, l_raw ); 25 26 -- It is vital to use htp.PRN to avoid 27 -- spurious line feeds getting added to your 28 -- document. 29 htp.prn( utl_raw.cast_to_varchar2( l_raw ) ); 30 l_off := l_off+l_amt; 31 l_amt := 32000; 32 end loop; 33 exception 34 when no_data_found then 35 NULL; 36 end; 37 end; 38 39 end; 40 /

Package body created.

So, now if I had a DAD (Database Access Descriptor; part of the normal setup for the PL/SQL cartridge and module) set up called mydad I can use the URL:

http://myhost:myport/pls/mydata/image_get.gif?p_id=3

DBMS_LOB

1091

to retrieve my image. Here we are passing P_ID=3 argument into image_get.gif, asking it to find the LOBlocator we stored in the row with id=3. We could embed this image in a page using the IMG tag as such:

<html><head><title>This is my page</title></head> <body>Here is my GIF file <img src=http://myhost:myport/pls/mydata/image_get.gif?p_id=3> </body></html>

Summary LOBs provide much more functionality than the now deprecated LONG data type. This section answered some of the questions I receive frequently regarding LOB manipulations. We discussed how to load LOBs into the database. We saw how to convert from a BLOB to a CLOB, and back again. We investigated how you might efficiently convert all of your existing legacy LONG and LONG RAW data into CLOB and BLOB data using unrecoverable and parallel operations. Lastly, we discussed how you might use the PL/SQL Web Toolkit to retrieve the contents of a CLOB or BLOB, and display this on a web page.

Appendix A

1092

DBMS_LOCK

The DBMS_LOCK package exposes, to the programmer, the locking mechanism used by Oracle itself. It allows them to create their own named locks. These locks can be monitored in the same way as any other Oracle lock. They will show up in the dynamic performance view V$LOCK with a type of UL (userlock). Also any standard tool such as Oracle Enterprise Manager, and the UTLOCKT.SQL script (found in [ORACLE_HOME]/rdbms/admin) will display them as well. In addition to exposing the locking mechanism for a programmer to use, DBMS_LOCK has one other utility function, a SLEEP function, which allows a PL/SQL program to pause for a given number of seconds.

The DBMS_LOCK package has many uses, for example:

❑ You have a routine that uses UTL_FILE to write audit messages to an operating system file. Only one process at a time should write to this file. On some operating systems, such as Solaris, many can write simultaneously (the OS does not prevent it). This results in inter-leaved audit messages that are hard or impossible to read. DBMS_LOCK can be used to serialize access to this file.

❑ To prevent mutually exclusive operations from occurring concurrently. For example, assume you have a data purge routine that can run only when other sessions that need the data are not running. These other sessions cannot begin while a purge is happening – they must wait. The purge session would attempt to get a named lock in X (exclusive) mode. The other sessions would attempt to get this same named lock in S (shared) mode. The X lock request will block while any S locks are present, and the S lock request will block while the X lock is held. You will have made it so the purge session will wait until there are no 'normal' sessions, and if the purge session is executing, all other sessions will be blocked until it is finished.

These are two common uses of this package. They work well as long as all sessions co-operate in the use of locks (there is nothing stopping a session from using UTL_FILE to open and write to that audit file without getting the appropriate lock). As an example, we will implement a solution to a mutual

DBMS_LOCK

1093

exclusion problem that many applications could benefit from. This problem arises from two sessions attempting to INSERT into the same table, and that table has a primary key or unique constraint on it. If both sessions attempt to use the same value for the constrained columns, the second (and third, and so on) sessions will block indefinitely, waiting for the first session to commit or rollback. If the first session commits, these blocked sessions will get an error. Only if the first session rolls back will one of the subsequent sessions have their INSERT succeed. The gist of this is that people will wait for a while to find out they cannot do what they wanted to.

This issue is avoidable when using UPDATE, because we can lock the row we want to update in a non-blocking fashion, prior to updating it. That is, instead of just executing:

update emp set ename = 'King' where empno = 1234;

you can code:

select ename from emp where empno = 1234 FOR UPDATE NOWAIT;update emp set ename = 'King' where empno = 1234;

The use of the FOR UPDATE NOWAIT on the SELECT will have the effect of locking the row for your session (making it so the UPDATE will not block), or returning an ORA-54 'Resource Busy' error. If we do not get an error from the SELECT, the row is locked for us.

When it comes to INSERTs however, we have no such method. There is no existing row to SELECT and lock, and hence, no way to prevent others from inserting a row with the same value, thus blocking our session and causing us to wait indefinitely. Here is where DBMS_LOCK comes into play. To demonstrate this, we will create a table with a primary key and a trigger that will prevent two (or more) sessions from inserting the same values simultaneously. We will place a trigger on this table as well. This trigger will use DBMS_UTILITY.GET_HASH_VALUE (see the DBMS_UTILITY section later in this appendix for more information) to hash the primary key into some number between 0 and 1,073,741,823 (the range of lock ID numbers permitted for our use by Oracle). In this example, I've chosen a hash table of size 1,024, meaning we will hash our primary keys into one of 1,024 different lock IDs. Then, we will use DBMS_LOCK.REQUEST to allocate an exclusive lock based on that ID. Only one session at a time will be able to do this, so if someone else tries to insert a record into our table with the same primary key, their lock request will fail (and the error RESOURCE BUSY will be raised to them):

tkyte@TKYTE816> create table demo ( x int primary key );

Table created.

tkyte@TKYTE816> create or replace trigger demo_bifer 2 before insert on demo 3 for each row 4 declare 5 l_lock_id number; 6 resource_busy exception; 7 pragma exception_init( resource_busy, -54 ); 8 begin 9 l_lock_id := 10 dbms_utility.get_hash_value( to_char( :new.x ), 0, 1024 ); 11 12 if ( dbms_lock.request

Appendix A

1094

13 ( id => l_lock_id, 14 lockmode => dbms_lock.x_mode, 15 timeout => 0, 16 release_on_commit => TRUE ) = 1 ) 17 then 18 raise resource_busy; 19 end if; 20 end; 21 /

Trigger created.

If, in two separate sessions you execute:

tkyte@TKYTE816> insert into demo values ( 1 );

1 row created.

it will succeed in the first one, but immediately issue:

tkyte@TKYTE816> insert into demo values ( 1 ); insert into demo values ( 1 ) * ERROR at line 1: ORA-00054: resource busy and acquire with NOWAIT specified ORA-06512: at "TKYTE.DEMO_BIFER", line 15 ORA-04088: error during execution of trigger 'TKYTE.DEMO_BIFER'

in the second session (unless the first session commits, and then a UNIQUE CONSTRAINT violation will be the error message).

The concept here is to take the primary key of the table in the trigger, and put it in a character string. We can then use DBMS_UTILITY.GET_HASH_VALUE to come up with a 'mostly unique' hash value for the string. As long as we use a hash table smaller than 1,073,741,823, we can 'lock' that value exclusively using DBMS_LOCK. We could use the DBMS_LOCK routine ALLOCATE_UNIQUE as well, but it comes with some amount of overhead. ALLOCATE_UNIQUE creates a unique lock identifier in the range of 1,073,741,824 to 1,999,999,999. It does this using another database table, and a recursive (autonomous) transaction. The hashing approach uses less resource, and avoids this recursive SQL call.

After hashing, we take this value, and use DBMS_LOCK to request that lock ID to be exclusively locked with a timeout of zero (it returns immediately if someone else has locked that value). If we timeout, we raise ORA-54 RESOURCE BUSY. Else, we do nothing – it is OK to INSERT, we won't block.

Of course, if the primary key of your table is an INTEGER, and you don't expect the key to go over 1 billion, you can skip the hash, and just use the number as the lock ID as well.

You'll need to play with the size of the hash table (1,024 in my example) to avoid artificial RESOURCEBUSY messages, due to different strings hashing to the same number. The size of the hash table will be application (data) specific and will be influenced by the number of concurrent insertions as well. Also, the owner of the trigger will need EXECUTE on DBMS_LOCK granted directly to them (not via a role). Lastly, you might find you run out of ENQUEUE_RESOURCES if you insert lots of rows this way without committing. If you do, you need to modify the init.ora parameter ENQUEUE_RESOURCES to be high

DBMS_LOCK

1095

enough (you'll get an error message about ENQUEUE_RESOURCES if you hit this). You might instead add a flag to the trigger to allow people to turn the check on and off. If I was going to insert hundreds/thousands of records, I might not want this check enabled for example.

We can 'see' our locks in the V$LOCK table, as well as the number of primary keys hashed to (the lock) it. For example, using our DEMO table from above with the trigger in place:

tkyte@TKYTE816> insert into demo values ( 1 );

1 row created.

tkyte@TKYTE816> select sid, type, id1 2 from v$lock 3 where sid = ( select sid from v$mystat where rownum = 1 ) 4 /

SID TY ID1 ---------- -- ---------- 8 TX 589913 8 TM 30536 8 UL 827

tkyte@TKYTE816> begin 2 dbms_output.put_line 3 ( dbms_utility.get_hash_value( to_char(1), 0, 1024 ) ); 4 end; 5 / 827


Notice the UL lock, our user lock, with an ID1 of 827. It just so happens that 827 is the hash value of TO_CHAR(1), our primary key.

To complete this example, we need to discuss what would happen if your application permits an UPDATE to the primary key. Ideally, you would not UPDATE a primary key, but some applications do. We would have to consider what would happen if one session updates the primary key:

tkyte@TKYTE816> update demo set x = 2 where x = 1;

1 row updated.

and another session attempts to INSERT a row with that newly updated primary key value:

tkyte@TKYTE816> INSERT INTO DEMO VALUES (2);

This second session will block once again. The issue here is that every process that can modify the primary key is not yet participating in our modified locking scheme. In order to solve this issue, the case whereby you UPDATE the primary key, we need to modify the times our trigger will fire to be:

before insert OR UPDATE OF X on demo

If the trigger we coded fires before any INSERT, or the UPDATE of the column X, our expected behavior will be observed (and the UPDATE will become non-blocking as well).

Appendix A

1096

Summary DBMS_LOCK exposes the internal Oracle locking mechanism for our applications to exploit. As demonstrated above, we can use this functionality to implement our own custom locking that goes above, and beyond the supplied functionality. We reviewed potential uses for this facility such as a serialization device for accessing a shared resource (an OS file for example), or as a method to coordinate various conflicting processes. We took an in-depth look at using DBMS_LOCK as a tool to prevent blocking INSERTs. This example demonstrated how to use DBMS_LOCK, and how to see your locks in the V$LOCK table itself. Lastly, we closed with the importance of ensuring all sessions coordinate their activities with regards to your custom locking, by discussing how an UPDATE of a primary key could subvert our non-blocking insert logic.

DBMS_LOGMNR

1097

DBMS_LOGMNR

The LogMiner packages, DBMS_LOGMNR and DBMS_LOGMNR_D, allow for analysis of Oracle's redo log files. You would make use of this feature for some of the following reasons:

❑ You want to find out when a table was 'accidentally' dropped, and by whom.

❑ You want to perform some auditing on a given table, or set of tables, to see who has been modifying what pieces of it. You can do this auditing 'after the fact'. Normally, you might use the AUDIT command, but this must be enabled ahead of time, and it only tells you someone modified the table – not what they modified. LogMiner is good for post-facto 'who did that' discovery, and to see what data changed exactly.

❑ You would like to 'undo' a given transaction. In order to undo it, we'll need to see what it did, and get the PL/SQL for undoing it.

❑ You would like to get some empirical counts of rows modified in an average transaction.

❑ You would like to perform a historical analysis of how the database has been used over time.

❑ You would like to find out why your database is suddenly generating 10 MB of log every minute. It never used to do this, and now it is all of a sudden. Are there any obvious culprits to be found in a quick review of the logs?

❑ You would like to see what is really happening 'under the cover'. The contents of the redo logs show you what actually happened when you did that INSERT on a table with a trigger that does an UPDATE of another table. All of the effects of your transaction are recorded in the log. LogMiner is an excellent exploration tool.

LogMiner provides you the tools to do all this, and more. What I will provide here is a quick overview of how to use LogMiner, and then explain some of the caveats of its use that are not spelled out in the Supplied PL/SQL Packages Reference guide shipped with Oracle. As with all of the other packages, it is recommended that you read the section in the Supplied PL/SQL Packages Reference on DBMS_LOGMNR and DBMS_LOGMNR_D to get an overview of the functions and procedures they contain, and how they are used. Below in the Options and Usagesection, we will give an overview of these procedures, and their inputs as well.

Appendix A

1098

LogMiner works best on archived redo log files, although it can be used with online redo log files that are not active. Attempting to use an active online redo log file could lead to an error message or just confusion on your part, as the redo log file will contain a mixture of old and new transaction data. An interesting thing to note about LogMiner is that you do not need to analyze a log file in the database that originally created it. It doesn't even have to be the same exact database version (you can analyze version 8.0 archive files in an 8.1 database). You can move an archived redo log file to another system, and analyze it there instead. This can be quite convenient for auditing and looking at historical usage patterns, without impacting the existing system. In order to do this however, you must use a database that is on the same hardware platform (byte-ordering, word sizes, and so on will be affected by this). Also, you will want to make sure that the database block sizes are the same (or that the database doing the analysis has a block size at least as big as the database originating the redo log), and have the same character set.

Using LogMiner is a two-step process. Step one involves creating a data dictionary for LogMiner to operate with. This is what allows a redo log file from one database to be analyzed on another – LogMiner does not use the existing data dictionary, it uses the data dictionary that was exported to an external file by the DBMS_LOGMNR_D package. LogMiner can be used without this data dictionary, but you will find the resulting output virtually unreadable. We'll take a look at what this would look like later.

Step two involves importing the redo log files and starting LogMiner. Once LogMiner is started, you can review the contents of the redo log files using SQL. There are four V$ views associated with LogMiner. The main view is V$LOGMNR_CONTENTS. This is the view you will use to review the contents of the redo log files you have loaded. We will take a look at this view in more detail in the example and at the end of this section we have a table that defines each column. The other three views are:

❑ V$LOGMNR_DICTIONARY – This view contains information about the dictionary file that has been loaded. This is the dictionary you created in step one. In order to make sense of the contents of a redo log file, we need to have a dictionary file that tells us what object name goes with what object ID, what the columns and data types of each table are, and so on. This view contains at most, one row for the currently loaded dictionary only.

❑ V$LOGMNR_LOGS – This view contains information about the redo log files you have requested LogMiner to load into the system. The contents of these redo log files will be found in V$LOGMNR_CONTENTS. This view tells you about the redo log file itself. Attributes such as the name of the redo log file, the database name of the database it came from, the SCNs (system change numbers) contained in it and so on, are found here. This view will have an entry per log file you are analyzing.

❑ V$LOGMNR_PARAMETERS – This view shows the parameters that were passed to LogMiner during it's start up. This view will have one entry after you call the start up routine for log miner.

An important point to note here is that because LogMiner's memory allocation comes from the PGA, LogMiner cannot be used in an MTS environment. This is because with MTS, you will be assigned to a different shared server (process or thread) each time you make a request into the database. The data you loaded into Process One (Shared Server One) is simply not available to Process Two (Shared Server Two). You must be using a dedicated server configuration for LogMiner to function. Also, the output is only visible in a single session, and only for the life of that session. If further analysis is needed, you must either reload the information, or make it permanent, perhaps using a CREATE TABLE AS SELECT. If you are analyzing a large amount of data, making the data permanent via a CREATE TABLE AS SELECT or INSERT INTO makes even more sense. You would then be able to index this information whereas with the V$LOGMNR_CONTENTS table, you will always be performing a full scan of a V$ table, since it has no indexes. This full scanning of a V$ table can be quite resource-intensive.

DBMS_LOGMNR

1099

Overview What we'll do now is present an overview of how to use the LogMiner facility. After that, we'll look at all of the inputs to the two LogMiner supplied packages, and what they mean. Then, we will investigate using LogMiner to find out when some operation took place in the database. After that, we'll take a quick look at how LogMiner affects your session's memory usage, and how it caches the redo log files internally. Lastly, we'll look at some of the limitations of LogMiner that are not mentioned in the documentation.

Step 1: Creating the Data Dictionary In order for LogMiner to map internal object IDs and columns to their appropriate tables, it needs a data dictionary. It will not use the data dictionary already present in the database. Rather, it relies on an external file to provide the data dictionary. LogMiner works this way in order to allow redo log files from other databases to be analyzed in different one. Additionally, the data dictionary that is current today in your database may not support all of the objects that were in the database when the redo log file was generated, hence the need to be able to import a data dictionary.

To see the purpose of this data dictionary file, we'll look at some output from LogMiner without having a data dictionary loaded. We'll do this by loading an archived redo log file and starting LogMiner. Then a quick query in V$LOGMNR_CONTENTS to see what is there:

tkyte@TKYTE816> begin 2 sys.dbms_logmnr.add_logfile 3 ( 'C:\oracle\oradata\tkyte816\archive\TKYTE816T001S01263.ARC', 4 sys.dbms_logmnr.NEW ); 5 end; 6 /


tkyte@TKYTE816> begin 2 sys.dbms_logmnr.start_logmnr; 3 end; 4 /


tkyte@TKYTE816> column sql_redo format a30 tkyte@TKYTE816> column sql_undo format a30 tkyte@TKYTE816> select scn, sql_redo, sql_undo from v$logmnr_contents 2 /

SCN SQL_REDO SQL_UNDO ---------- ------------------------------ ------------------------------ 6.4430E+126.4430E+12 set transaction read write; 6.4430E+12 update UNKNOWN.Objn:30551 set update UNKNOWN.Objn:30551 set Col[2] = HEXTORAW('787878') wh Col[2] = HEXTORAW('534d495448' ere ROWID = 'AAAHdXAAGAAAAJKAA ) where ROWID = 'AAAHdXAAGAAAA A'; JKAAA';

6.4430E+126.4430E+12 commit;

Appendix A

1100

tkyte@TKYTE816> select utl_raw.cast_to_varchar2(hextoraw('787878')) from dual;

UTL_RAW.CAST_TO_VARCHAR2(HEXTORAW('787878')) ------------------------------------------------------------------xxx

tkyte@TKYTE816> select utl_raw.cast_to_varchar2(hextoraw('534d495448')) from dual;

UTL_RAW.CAST_TO_VARCHAR2(HEXTORAW('534D495448')) ---------------------------------------------------------------------SMITH

This output is fairly unreadable. We know that object number 30551 was updated and column 2 was modified. Further, we can turn the HEXTORAW('787878') into a character string. We could go to the data dictionary and find out that object 30551 is:

tkyte@TKYTE816> select object_name 2 from all_objects 3 where data_object_id = 30551;

OBJECT_NAME------------------------------EMP

but only if we are in the same database in which the redo log file was originally generated, and only if that object still exists. Further we could DESCRIBE EMP and discover that column 2 is ENAME. Therefore, the SQL_REDOcolumn from LogMiner is really UPDATE EMP SET ENAME = 'XXX' WHERE ROWID = .... Fortunately, we do not need to go through this laborious conversion each and every time we use log miner. We'll find that by building and then loading a dictionary, we'll get much better results. The following example shows what output we could expect if we build a dictionary file for LogMiner to work with, and then load it.

We start by creating the dictionary file. Creating this data dictionary file is rather straightforward. The prerequisites for doing this are:

❑ UTL_FILE has been configured in your init.ora file so that there is at least one directory that can be written to. See the section on UTL_FILE for information on setting this up. DBMS_LOGMNR_D, the package that builds the data dictionary file, relies on UTL_FILE to perform I/O.

❑ The schema that will execute the DBMS_LOGMNR_D package has been granted EXECUTE ONSYS.DBMS_LOGMNR_D, or has a role that is able to execute this package. By default, the EXECUTE_CATALOG_ROLE has the privilege to run this package.

Once you have UTL_FILE set up and EXECUTE ON DBMS_LOGMNR_D, creating the data dictionary file is trivial. There is only one call inside of DBMS_LOGMNR_D, and this is called BUILD. You would simply execute something along the lines of:

tkyte@TKYTE816> set serveroutput on

tkyte@TKYTE816> begin 2 sys.dbms_logmnr_d.build( 'miner_dictionary.dat', 3 'c:\temp' );

DBMS_LOGMNR

1101

4 end; 5 / LogMnr Dictionary Procedure started LogMnr Dictionary File Opened TABLE: OBJ$ recorded in LogMnr Dictionary File TABLE: TAB$ recorded in LogMnr Dictionary File TABLE: COL$ recorded in LogMnr Dictionary File TABLE: SEG$ recorded in LogMnr Dictionary File TABLE: UNDO$ recorded in LogMnr Dictionary File TABLE: UGROUP$ recorded in LogMnr Dictionary File TABLE: TS$ recorded in LogMnr Dictionary File TABLE: CLU$ recorded in LogMnr Dictionary File TABLE: IND$ recorded in LogMnr Dictionary File TABLE: ICOL$ recorded in LogMnr Dictionary File TABLE: LOB$ recorded in LogMnr Dictionary File TABLE: USER$ recorded in LogMnr Dictionary File TABLE: FILE$ recorded in LogMnr Dictionary File TABLE: PARTOBJ$ recorded in LogMnr Dictionary File TABLE: PARTCOL$ recorded in LogMnr Dictionary File TABLE: TABPART$ recorded in LogMnr Dictionary File TABLE: INDPART$ recorded in LogMnr Dictionary File TABLE: SUBPARTCOL$ recorded in LogMnr Dictionary File TABLE: TABSUBPART$ recorded in LogMnr Dictionary File TABLE: INDSUBPART$ recorded in LogMnr Dictionary File TABLE: TABCOMPART$ recorded in LogMnr Dictionary File TABLE: INDCOMPART$ recorded in LogMnr Dictionary File Procedure executed successfully - LogMnr Dictionary Created


It is recommended that you issue a SET SERVEROUTPUT ON prior to executing DBMS_LOGMNR_D, as this will allow informational messages from DBMS_LOGMNR_D to be printed. This can be extremely useful when trying to diagnose an error from DBMS_LOGMNR_D. What the above command did was to create a file C:\TEMP\MINER_DICTIONARY.DAT. This is a plain text, ASCII file that you may edit to see what is in there. This file consists of a lot of SQL-like statements that are parsed and executed by the LogMiner start routine. Now that we have a dictionary file on hand, we are ready to see what the contents of V$LOGMNR_CONTENTSmight look like now:

tkyte@TKYTE816> begin 2 sys.dbms_logmnr.add_logfile 3 ( 'C:\oracle\oradata\tkyte816\archive\TKYTE816T001S01263.ARC', 4 sys.dbms_logmnr.NEW ); 5 end; 6 /


tkyte@TKYTE816> begin 2 sys.dbms_logmnr.start_logmnr 3 ( dictFileName => 'c:\temp\miner_dictionary.dat' ); 4 end; 5 /


Appendix A

1102

tkyte@TKYTE816> column sql_redo format a30 tkyte@TKYTE816> column sql_undo format a30 tkyte@TKYTE816> select scn, sql_redo, sql_undo from v$logmnr_contents 2 /

SCN SQL_REDO SQL_UNDO ---------- ------------------------------ ------------------------------ 6.4430E+126.4430E+12 set transaction read write; 6.4430E+12 update TKYTE.EMP set ENAME = ' update TKYTE.EMP set ENAME = ' xxx' where ROWID = 'AAAHdXAAGA SMITH' where ROWID = 'AAAHdXAA AAAJKAAA'; GAAAAJKAAA';

6.4430E+126.4430E+12 commit;

Now that's more like it – we can actually read the SQL that LogMiner generates for us, which would 'replay' (or undo) the transaction we are looking at. Now we are ready to go into Step 2 – Using LogMiner.

Step 2: Using Log Miner Here, we will take the dictionary file we just generated, and use it to review the contents of some archived redo log files. Before we load a redo log file, we will generate one with some known transactions in it. For the first time around, this will make it easier to see what we have. We'll be able to correlate what we find in the V$LOGMNR_CONTENTS view with what we just did. For this to work, it is important to have a 'test' database, one where you can be ensured you are the only one logged in. This allows us to artificially constrain just exactly what gets put into the redo log. Also, it would be necessary to have the ALTER SYSTEM privilege in this database so we can force a log file archive. Lastly, this is easiest to do if the database is in archive log mode with automatic archiving. In this fashion, finding the redo log file is trivial (it will be the one just archived – we'll see below how to find this). If you are using a NOARCHIVELOGMODE database, you will need to find the active log, and determine which log file was active just prior to it. So, to generate our sample transaction we could:

tkyte@TKYTE816> alter system archive log current;

System altered.

tkyte@TKYTE816> update emp set ename = lower(ename);

14 rows updated.

tkyte@TKYTE816> update dept set dname = lower(dname);

4 rows updated.


Commit complete.


System altered.

DBMS_LOGMNR

1103

tkyte@TKYTE816> column name format a80 tkyte@TKYTE816> select name 2 from v$archived_log 3 where completion_time = ( select max(completion_time) 4 from v$archived_log ) 5 /

NAME--------------------------------------------------------------C:\ORACLE\ORADATA\TKYTE816\ARCHIVE\TKYTE816T001S01267.ARC

Now, given that we were the only user logged in doing work, the archive redo log we just generated will have our two updates in it and nothing else. This last query against V$ARCHIVED_LOG shows us the name of the archive redo log file we actually want to analyze. We can load this into LogMiner and get started by using the following SQL. It will add the last archive redo log file to the LogMiner list, and then start LogMiner:

tkyte@TKYTE816> declare 2 l_name v$archived_log.name%type; 3 begin 4 5 select name into l_name 6 from v$archived_log 7 where completion_time = ( select max(completion_time) 8 from v$archived_log ); 9 10 sys.dbms_logmnr.add_logfile( l_name, sys.dbms_logmnr.NEW ); 11 end; 12 /




The first call, to DBMS_LOGMNR.ADD_LOGFILE, loaded an archive redo log file into LogMiner. I passed in the name of the archived redo log file, as well as the option of DBMS_LOGMNR.NEW. Since this is the first log file I am adding in, I used DBMS_LOGMNR.NEW. The other options are ADDFILE to add another log file to an existing list of files and REMOVEFILE to remove a file from consideration. After we load the log files we are interested in, we can call DBMS_LOGMNR.START_LOGMNR and tell it the name of the dictionary file we created. We used a minimal call to START_LOGMNR here, passing just the name of the dictionary file. We will look at some of the other options to START_LOGMNR in the next section Options and Usage.

Now that we've loaded a log file and started LogMiner, we are ready to take our first look at the contents of V$LOGMNR_CONTENTS. V$LOGMNR_CONTENTS has lots of information in it and for now, we'll look at a very small slice of the data available. Specifically, we'll investigate the SCN, SQL_REDO, and SQL_UNDO columns. In case you are not familiar with it, the SCN is a simple timing mechanism that Oracle uses to guarantee ordering of transactions, and to enable recovery from failure. They are also used to guarantee read-consistency, and checkpointing in the database. Think of the SCN as a ticker – every time someone commits, the SCN is incremented by one. Here is a sample query from our example above where we lowercased the names in EMPand DEPT tables:

Appendix A

1104

tkyte@TKYTE816> column sql_redo format a20 word_wrappedtkyte@TKYTE816> column sql_undo format a20 word_wrapped

tkyte@TKYTE816> select scn, sql_redo, sql_undo from v$logmnr_contents 2 /

SCN SQL_REDO SQL_UNDO---------- -------------------- -------------------- 6.4430E+12 set transaction read write;

6.4430E+12 update TKYTE.EMP set update TKYTE.EMP set ENAME = 'smith' ENAME = 'SMITH' where ROWID = where ROWID = 'AAAHdYAAGAAAAJKAAA' 'AAAHdYAAGAAAAJKAAA' ; ;

6.4430E+126.4430E+12 update TKYTE.EMP set update TKYTE.EMP set ENAME = 'allen' ENAME = 'ALLEN' where ROWID = where ROWID = 'AAAHdYAAGAAAAJKAAB' 'AAAHdYAAGAAAAJKAAB' ; ;

...(many similar rows snipped out)...

6.4430E+12 update TKYTE.DEPT update TKYTE.DEPT set DNAME = 'sales' set DNAME = 'SALES' where ROWID = where ROWID = 'AAAHdZAAGAAAAKKAAC' 'AAAHdZAAGAAAAKKAAC' ; ;

6.4430E+12 update TKYTE.DEPT update TKYTE.DEPT set DNAME = set DNAME = 'operations' where 'OPERATIONS' where ROWID = ROWID = 'AAAHdZAAGAAAAKKAAD' 'AAAHdZAAGAAAAKKAAD' ; ;

6.4430E+12 commit;

22 rows selected.

As you can see, our two SQL statements generated many more than two SQL statements from the redo log. The redo log contains the bits and bytes that we changed – not SQL. Therefore our multi-row statement UPDATEEMP SET ENAME = LOWER(ENAME) is presented by LogMiner as a series of single row updates. LogMiner currently cannot be used to retrieve the actual SQL performed at run-time. It can only reproduce equivalent SQL, SQL that does the same thing but in many individual statements.

Now, we'll go one step further with this example. The V$LOGMNR_CONTENTS view has 'placeholder' columns. These placeholder columns are useful for finding particular updates for up to five columns in your table. The placeholder columns can tell us the name of the changed column, and show us the 'before' value of the column, and the 'after' column value. Since these columns are broken out from the SQL, it would be very easy to find the transaction such that the ENAME column was updated (the name placeholder column would have ENAME) from KING (the before image placeholder column would have KING in it) to king. We'll do another quick UPDATEexample, and set up the necessary column mapping file to demonstrate this. The column mapping file (colmap for short) is used to tell LogMiner which columns are of interest to you by table. We can map up to five columns per table to be mapped, into these placeholder columns. The format of a colmap file is simply:

DBMS_LOGMNR

1105

colmap = TKYTE DEPT (1, DEPTNO, 2, DNAME, 3, LOC); colmap = TKYTE EMP (1, EMPNO, 2, ENAME, 3, JOB, 4, MGR, 5, HIREDATE);

This will map the DEPT DEPTNO column to the first placeholder column when we are looking at a row for the DEPT table. It will map the EMP EMPNO column to this placeholder column when we are looking at an EMP row.

The column mapping file in general has lines that consist of the following (items in bold are constants, <sp> represent a single, mandatory space)

colmap<sp>=<sp>OWNER<sp>TABLE_NAME<sp>(1,<sp>CNAME[,<sp>2,<sp>CNAME]...);

The case of everything is important – the OWNER must be uppercase, the table name must be the 'correct' case (uppercase is usually the correct case unless you've used quoted identifiers to create objects). The spaces are mandatory as well. In order to make using the column mapping file a little easier, I use a script such as:

set linesize 500 set trimspool on set feedback off set heading off set embedded on spool logmnr.opt select 'colmap = ' || user || ' ' || table_name || ' (' || max( decode( column_id, 1, column_id , null ) ) || max( decode( column_id, 1, ', '||column_name, null ) ) || max( decode( column_id, 2, ', '||column_id , null ) ) || max( decode( column_id, 2, ', '||column_name, null ) ) || max( decode( column_id, 3, ', '||column_id , null ) ) || max( decode( column_id, 3, ', '||column_name, null ) ) || max( decode( column_id, 4, ', '||column_id , null ) ) || max( decode( column_id, 4, ', '||column_name, null ) ) || max( decode( column_id, 5, ', '||column_id , null ) ) || max( decode( column_id, 5, ', '||column_name, null ) ) || ');' colmap from user_tab_columnsgroup by user, table_name /spool off

in SQL*PLUS to generate the logmnr.opt file for me. For example, if I execute this script in a schema that contains only the EMP and DEPT tables from the SCOTT/TIGER account, I will see:

tkyte@TKYTE816> @colmap colmap = TKYTE DEPT (1, DEPTNO, 2, DNAME, 3, LOC); colmap = TKYTE EMP (1, EMPNO, 2, ENAME, 3, JOB, 4, MGR, 5, HIREDATE);

It always picks the first five columns of the table. If you desire a different set of five columns, just edit the resulting logmnr.opt file this creates, and change the column names. For example, the EMP table has three more columns that are not shown in the above colmap – SAL, COMM, and DEPTNO. If you wanted to see the SAL column instead of the JOB column, the colmap would simply be:

tkyte@TKYTE816> @colmap colmap = TKYTE DEPT (1, DEPTNO, 2, DNAME, 3, LOC); colmap = TKYTE EMP (1, EMPNO, 2, ENAME, 3, SAL, 4, MGR, 5, HIREDATE);

Appendix A

1106

Important considerations for the colmap file, beyond its needs to have the correct case and whitespace, are

❑ The file must be named logmnr.opt. No other name may be used.

❑ This file must be in the same directory as your dictionary file.

❑ You must be using a dictionary file in order to use a colmap file.

So, we will now modify all of the columns in the DEPT table. I am using four different UPDATEs, each against a different row and set of columns. This is so we'll see better the effect of the placeholder columns:


tkyte@TKYTE816> update dept set deptno = 11 2 where deptno = 40 3 /

tkyte@TKYTE816> update dept set dname = initcap(dname) 2 where deptno = 10 3 /

tkyte@TKYTE816> update dept set loc = initcap(loc) 2 where deptno = 20 3 /

tkyte@TKYTE816> update dept set dname = initcap(dname), 2 loc = initcap(loc) 3 where deptno = 30 4 /



We can review the column-by-column changes now by loading the newly generated archived redo log file and starting Log Miner with the option USE_COLMAP. Note that I did generate the logmnr.opt file using the script above and I placed that file in the same directory with my dictionary file:

tkyte@TKYTE816> declare 2 l_name v$archived_log.name%type; 3 begin 4 5 select name into l_name 6 from v$archived_log 7 where completion_time = ( select max(completion_time) 8 from v$archived_log ); 9 10 sys.dbms_logmnr.add_logfile( l_name, sys.dbms_logmnr.NEW ); 11 end; 12 /


DBMS_LOGMNR

1107

tkyte@TKYTE816> begin 2 sys.dbms_logmnr.start_logmnr 3 ( dictFileName => 'c:\temp\miner_dictionary.dat', 4 options => sys.dbms_logmnr.USE_COLMAP ); 5 end; 6 /


tkyte@TKYTE816> select scn, ph1_name, ph1_undo, ph1_redo, 2 ph2_name, ph2_undo, ph2_redo, 3 ph3_name, ph3_undo, ph3_redo 4 from v$logmnr_contents 5 where seg_name = 'DEPT' 6 /

SCN PH1_NA PH1 PH1 PH2_N PH2_UNDO PH2_REDO PH3 PH3_UNDO PH3_REDO ---------- ------ --- --- ----- ---------- ---------- --- -------- -------- 6.4430E+12 DEPTNO 40 11 6.4430E+12 DNAME accounting Accounting 6.4430E+12 LOC DALLAS Dallas 6.4430E+12 DNAME sales Sales LOC CHICAGO Chicago

So, this output clearly shows us (from line 1 for example) that DEPTNO had a before image value of 40 (PH1)and became 11. This makes sense, since we did SET DEPTNO = 11 WHERE DEPTNO = 40. Notice that the remaining columns in that first row of output are Null. This is because Oracle logs changed bytes only; there is no before/after image of the DNAME and LOC columns for that row. The second row shows the update of the DNAME column from accounting to Accounting, and no changes for DEPTNO or LOC, as these columns were not affected. The last row shows that when we modified two columns with our UPDATE statement, they both show up in the placeholder columns.

As you can see, using the placeholder columns can be very convenient if you are trying to locate a specific transaction in a large set of redo. If you know that the transaction updated the X table, and changed the Ycolumn from a to b, finding this transaction will be a breeze.

Options and Usage These are the two packages that implement the LogMiner functionality – DBMS_LOGMNR and DBMS_LOGMNR_D. The DBMS_LOGMNR_D (the _D stands for 'dictionary') package has exactly one procedure in it, which is BUILD. This is used the build the data dictionary used by the DBMS_LOGMNR package when loading a redo log file. It will map object IDs to table names, determine data types, map column positions to column name, and so on. Using the DBMS_LOGMNR_D.BUILD routine is very straightforward. It takes two parameters:

❑ DICTIONARY_FILENAME – The name of the dictionary file to be created. In our examples, we have been using the file name miner_dictionary.dat.

❑ DICTIONARY_LOCATION – The path to where this file will be created. This routine uses UTL_FILE to create the file, so this path must be a valid path as set in the utl_file_dirinit.ora parameter See the UTL_FILE section later in this appendix for more details on configuring UTL_FILE.

Appendix A

1108

That's it for BUILD. Neither parameter is optional, so both must be supplied. If you receive an error from this routine similar to the following:

tkyte@TKYTE816> exec sys.dbms_logmnr_d.build( 'x.dat', 'c:\not_valid\' ); BEGIN sys.dbms_logmnr_d.build( 'x.dat', 'c:\not_valid\' ); END;

*ERROR at line 1: ORA-01309: specified dictionary file cannot be opened ORA-06510: PL/SQL: unhandled user-defined exception ORA-06512: at "SYS.DBMS_LOGMNR_D", line 793 ORA-06512: at line 1

it will mean that the directory you are attempting to use is not set in the utl_file_dir init.ora parameter.

The DBMS_LOGMNR package itself has only three routines:

❑ ADD_LOGFILE – To register the set of log files to be analyzed.

❑ START_LOGMNR – To populate the V$LOGMNR_CONTENTS view

❑ END_LOGMNR – To release all resources allocated by the LogMiner processing. This should be called before exiting your session to release resources cleanly, or when you are done using LogMiner.

The ADD_LOGFILE routine, as we have seen above, is called before actually starting LogMiner. It simply builds a list of log files that START_LOGMNR will process, and populate into the V$LOGMNR_CONTENTS view for us. The inputs to ADD_LOGFILE are:

❑ LOGFILENAME – The fully qualified filename of the archived redo log file you want to analyze.

❑ OPTIONS – Specifies how to add (or remove) this file. We use the DBMS_LOGMNR constants:

DBMS_LOGMNR.NEW – Start a new list. If a list already exists, this will empty the list

DBMS_LOGMNR.ADD – Add to an already started list or empty list

DBMS_LOGMNR.REMOVEFILE – Remove a file from the list

If we wanted to analyze the last two archive redo log files, we would call ADD_LOGFILE twice. For example:

tkyte@TKYTE816> declare 2 l_cnt number default 0; 3 begin 4 for x in (select name 5 from v$archived_log 6 order by completion_time desc ) 7 loop 8 l_cnt := l_cnt+1; 9 exit when ( l_cnt > 2 ); 10 11 sys.dbms_logmnr.add_logfile( x.name ); 12 end loop; 13

DBMS_LOGMNR

1109

14 sys.dbms_logmnr.start_logmnr 15 ( dictFileName => 'c:\temp\miner_dictionary.dat', 16 options => sys.dbms_logmnr.USE_COLMAP ); 17 end; 18 /


Within that same session, after we've started LogMiner, we may call ADD_LOGFILE to add more log files, remove uninteresting ones, or if you use DBMS_LOGMNR.NEW, reset the list of log files to be just that one new file. Calling DBMS_LOGMNR.START_LOGMNR after making changes to the list will effectively flush the V$LOGMNR_CONTENTS view, and repopulate it with the contents of the log files that are on the list now.

Moving onto DBMS_LOGMNR.START_LOGMNR, we see that we have many inputs. In the above examples, we have only been using two out of the six available to us. We've used the dictionary file name and the options (to specify we wanted to use a colmap file). The available inputs in full are:

❑ STARTSCN and ENDSCN – If you know the system change number range you are interested in exactly, this will limit the rows in V$LOGMNR_CONTENTS to be between these values. This is useful after you've loaded a full log file, and determined the upper and lower SCNs you are interested in. You can restart LogMiner with this range to cut down on the size of V$LOGMNR_CONTENTS. These values default to 0, implying they are not used by default.

❑ STARTTIME and ENDTIME – Instead of using the SCN, you can supply a date/time range. Only log entries that fall between the start and end time will be visible in the V$LOGMNR_CONTENTS view. These values are ignored if STARTSCN and ENDSCN are used, and default to 01-Jan-1988 and 01-Jan-2988.

❑ DICTFILENAME – The fully qualified path to the dictionary file created by DBMS_LOGMNR_D.BUILD.

❑ OPTIONS – Currently there is only one option to DBMS_LOGMNR.START_LOGMNR and this is DBMS_LOGMNR.USE_COLMAP. This directs LogMiner to look for a logmnr.opt file in the same directory in which it finds the DICTFILENAME. It is important to note that the name of the colmap file must be logmnr.opt, and it must reside in the same directory as the dictionary file.

The last procedure in DBMS_LOGMNR is simply the DBMS_LOGMNR.END_LOGMNR routine. This terminates the LogMiner session, and empties out the V$LOGMNR_CONTENTS view. After you call DBMS_LOGMNR.END_LOGMNR, any attempt to query the view will result in:

tkyte@TKYTE816> exec dbms_logmnr.end_logmnr;


tkyte@TKYTE816> select count(*) from v$logmnr_contents; select count(*) from v$logmnr_contents * ERROR at line 1: ORA-01306: dbms_logmnr.start_logmnr() must be invoked before selecting from v$logmnr_contents

Appendix A

1110

Using Log Miner to Find Out When... This is one of the more common uses of LogMiner I've seen. Someone 'accidentally' dropped a table. You would like to get it back, or just find out who did it. Or maybe someone updated an important table and you don't know whom, but no one will own up to it. In any case, something happened, you do not have auditing enabled, but you have been running in archive log mode, and have the backups. You would like to restore your backups and do a point-in-time recovery to the point in time immediately prior to the DROP TABLE. In this fashion, you can restore and recover that table, stop recovery (hence not dropping the table again), and then export this table from the restored database and import it into the correct database. This allows you to restore the table with all of the changes intact.

In order to do this, we'll need to know either the exact time or the SCN of the DROP TABLE. Since clocks are out of sync, and people are probably panicked, they may give bad information in this case. What we can do is load up the archived log files from around the time of the DROP TABLE, and find the exact SCN of the point we want to recover.

We'll do another quick example here to isolate the statements you might see in LogMiner when a table is dropped. I'm using locally managed tablespaces, so if you are using dictionary-managed tablespaces, you may see more SQL than I do below. This extra SQL you see with a dictionary-managed tablespace will be the SQL executed to return the extents back to the system, and free up the space allocated to the table. So, here we go to drop the table:


System altered.

tkyte@TKYTE816> drop table dept;

Table dropped.


System altered.

Now, we want to locate the SQL_REDO that represents the DROP TABLE. If you recall, the actual SQL executed at run-time is not reported by LogMiner. Rather, the equivalent SQL is reported. We will not see a DROPTABLE statement – we will see data dictionary modifications. The one we are looking for will be a DELETEagainst SYS.OBJ$, which is the base table for holding all objects. Part of dropping a table involves deleting a row from SYS.OBJ$. Fortunately, when LogMiner builds the SQL_REDO to process a DELETE, it includes the column values in the SQL along with the row ID. We can use this fact to search for the DELETE of DEPT from OBJ$. Here is how:

tkyte@TKYTE816> declare 2 l_name v$archived_log.name%type; 3 begin 4 select name into l_name 5 from v$archived_log 6 where completion_time = ( select max(completion_time) 7 from v$archived_log ); 8 9 sys.dbms_logmnr.add_logfile( l_name, sys.dbms_logmnr.NEW ); 10 end; 11 /

DBMS_LOGMNR

1111


tkyte@TKYTE816> begin 2 sys.dbms_logmnr.start_logmnr 3 ( dictFileName => 'c:\temp\miner_dictionary.dat', 4 options => sys.dbms_logmnr.USE_COLMAP ); 5 end; 6 /


tkyte@TKYTE816> select scn, sql_redo 2 from v$logmnr_contents 3 where sql_redo like 'delete from SYS.OBJ$ %''DEPT''%' 4 /

SCN SQL_REDO ----------------- ---------------------------------------- 6442991097246 delete from SYS.OBJ$ where OBJ# = 30553 and DATAOBJ# = 30553 and OWNER# = 337 an d NAME = 'DEPT' and NAMESPACE = 1 and SU BNAME IS NULL and TYPE# = 2 and CTIME = TO_DATE('29-APR-2001 12:32:11', 'DD-MON- YYYY HH24:MI:SS') and MTIME = TO_DATE('2 9-APR-2001 12:32:11', 'DD-MON-YYYY HH24: MI:SS') and STIME = TO_DATE('29-APR-2001 12:32:11', 'DD-MON-YYYY HH24:MI:SS') an d STATUS = 1 and REMOTEOWNER IS NULL and LINKNAME IS NULL and FLAGS = 0 and OID$ IS NULL and SPARE1 = 6 and ROWID = 'AAA AASAABAAAFz3AAZ';

That is all there is to it. Now that we have found that the SCN was 6442991097246, we can do a point in time recovery elsewhere to recover this table, and restore it to our system. We can recover it to the very point in time immediately prior to it being dropped.

PGA Usage LogMiner uses PGA memory in order to perform its task. We have mentioned previously, that this implies you cannot use DBMS_LOGMNR with MTS. What we haven't looked at is how much PGA memory LogMiner might actually use.

The log files on my system are 100 MB each. I loaded up two of them for analysis, measuring the before and after PGA memory use:

tkyte@TKYTE816> select a.name, b.value 2 from v$statname a, v$mystat b 3 where a.statistic# = b.statistic# 4 and lower(a.name) like '%pga%' 5 /

Appendix A

1112

NAME VALUE ------------------------------ ---------- session pga memory 454768 session pga memory max 454768

tkyte@TKYTE816> declare 2 l_name varchar2(255) default 3 'C:\oracle\ORADATA\tkyte816\archive\TKYTE816T001S012'; 4 begin 5 for i in 49 .. 50 6 loop 7 sys.dbms_logmnr.add_logfile( l_name || i || '.ARC' ); 8 end loop; 9 10 sys.dbms_logmnr.start_logmnr 11 ( dictFileName => 'c:\temp\miner_dictionary.dat', 12 options => sys.dbms_logmnr.USE_COLMAP ); 13 end; 14 /




So, 200 MB of archive redo log is currently taking about 11.5 MB of PGA memory. This means that either the archived redo was mostly 'fluff', or that Oracle doesn't actually cache the entire redo log file in RAM. The answer is that Oracle doesn't actually cache the entire redo log file in RAM. Rather, it reads it from disk as needed. Only some information in cached in RAM.

If we actually query the V$LOGMNR_CONTENTS view right now and measure the amount of PGA memory in use after this operation, we'll see that the memory requirements for this will go up as we access it:

tkyte@TKYTE816> create table tmp_logmnr_contents unrecoverable 2 as 3 select * from v$logmnr_contents 4 / Table created.


DBMS_LOGMNR

1113


So, as you can see, our session now needs almost 20 MB of PGA memory.

Log Miner Limits LogMiner has some serious limits that you need to be aware of. These limits are in regards to using Oracle object types and chained rows.

Oracle Object Types Object types are somewhat supported by LogMiner. LogMiner is not able to rebuild the SQL you would typically use to access object types, and it is not able to support all object types. A quick example demonstrates best some of the limitations in this area. We'll start with a small schema with some common object types such as VARRAYs and nested tables in it:

tkyte@TKYTE816> create or replace type myScalarType 2 as object 3 ( x int, y date, z varchar2(25) ); 4 /

Type created.

tkyte@TKYTE816> create or replace type myArrayType 2 as varray(25) of myScalarType 3 /

Type created.

tkyte@TKYTE816> create or replace type myTableType 2 as table of myScalarType 3 /

Type created.

tkyte@TKYTE816> drop table t;

Table dropped.

tkyte@TKYTE816> create table t ( a int, b myArrayType, c myTableType ) 2 nested table c store as c_tbl 3 /

Table created.

tkyte@TKYTE816> begin 2 sys.dbms_logmnr_d.build( 'miner_dictionary.dat', 3 'c:\temp' ); 4 end; 5 /

Appendix A

1114


tkyte@TKYTE816> alter system switch logfile;

System altered.

tkyte@TKYTE816> insert into t values ( 1, 2 myArrayType( myScalarType( 2, sysdate, 'hello' ) ), 3 myTableType( myScalarType( 3, sysdate+1, 'GoodBye' ) ) 4 );

1 row created.

tkyte@TKYTE816> alter system switch logfile;

System altered.

So, in the above example, we created some object types, added a table that utilizes these types, re-exported our data dictionary, and then finally did some isolated DML on this object. Now we are ready to see what LogMiner is able to tell us about these operations:

tkyte@TKYTE816> begin 2 sys.dbms_logmnr.add_logfile( 'C:\oracle\rdbms\ARC00028.001', 3 dbms_logmnr.NEW ); 4 end; 5 /




tkyte@TKYTE816> select scn, sql_redo, sql_undo 2 from v$logmnr_contents 3 /

SCN SQL_REDO SQL_UNDO---------- -------------------- -------------------- 824288 824288 824288 824288 set transaction read write;

824288 insert into delete from TKYTE.C_TBL(NESTED_T TKYTE.C_TBL where ABLE_ID,X,Y,Z) NESTED_TABLE_ID = values HEXTORAW('252cb5fad8 (HEXTORAW('252cb5fad 784e2ca93eb432c2d35b

DBMS_LOGMNR

1115

8784e2ca93eb432c2d35 7c') and X = 3 and Y b7c'),3,TO_DATE('23- = JAN-2001 16:21:44', TO_DATE('23-JAN-2001 'DD-MON-YYYY 16:21:44', HH24:MI:SS'),'GoodBy 'DD-MON-YYYY e'); HH24:MI:SS') and Z = 'GoodBye' and ROWID = 'AAAFaqAADAAAAGzAAA' ;

824288 824288 824288 824288 insert into delete from TKYTE.T TKYTE.T(A,B,SYS_NC00 where A = 1 and B = 00300004$) values Unsupported Type and (1,Unsupported SYS_NC0000300004$ = Type,HEXTORAW('252cb HEXTORAW('252cb5fad8 5fad8784e2ca93eb432c 784e2ca93eb432c2d35b 2d35b7c')); 7c') and ROWID = 'AAAFapAADAAAARjAAA' ;

824288

10 rows selected.

As you can see, our original single INSERT:

tkyte@TKYTE816> insert into t values ( 1, 2 myArrayType( myScalarType( 2, sysdate, 'hello' ) ), 3 myTableType( myScalarType( 3, sysdate+1, 'GoodBye' ) ) 4 ); 1 row created.

was turned into two INSERTs. One for the child table (the nested table), and one for the parent able T.LogMiner does not reproduce the single INSERT – equivalent SQL was produced. As we look closer however, we will notice that in the INSERT INTO T, we see Unsupported Type as one of the column values. Looking back at the original INSERT, we can see that the unsupported type is in fact our VARRAY. LogMiner is not capable of reproducing this particular construct.

This does not remove all of the usefulness of LogMiner with regards to objects. It does prevent us from using it to undo or redo transactions, since it cannot faithfully reproduce the necessary SQL. However, we can still use it to analyze historical trends, perform auditing, and the like. Perhaps of more interest is that it gives us the ability to see how Oracle physically implements object types under the covers. For example, look at the insert into T:

insert into tkyte.t ( a, b, SYS_NC0000300004$) values ...

It is pretty clear to us what A and B are. They are our INT and MyArrayType (VARRAY) columns. However, where is C and what is this SYS_NC0000300004$ column? Well, C is our nested table, and nested tables are actually physically stored as a parent/child table. C is not stored in T; it is stored in a wholly separate table. The column SYS_NC0000300004$ is actually a surrogate primary key on T, and is used as a foreign key in C_TBL– the nested table. If we look at the INSERT into the nested table:

insert into tkyte.c_tbl( nested_table_id, x, y, z ) values ...

Appendix A

1116

we can see that the NESTED_TABLE_ID was added to our nested table, and this column is, in fact, used to join to the T.SYS_NC0000300004$ column. Further, looking at the value that is put into both of these columns:

HEXTORAW('252cb5fad8784e2ca93eb432c2d35b7c')

we can see that Oracle is, by default, using a system-generated 16 byte RAW value to join C_TBL with T.Therefore, through LogMiner analysis, we can gain a better understanding of how various features in Oracle are implemented. Here, we have seen how a nested table type is really nothing more than a parent/child table with a surrogate key in the parent table, and a foreign key in the child table.

Chained or Migrated Rows LogMiner currently does not handle chained or migrated rows. A chained row is a row that spans more than one Oracle block. A migrated row is a row that started on one block when it was inserted, but due to an UPDATE, grew too large to fit on this block with the other rows that were there, and was therefore 'moved' to a new block. A migrated row retains its original row ID – the block it was originally on has a pointer to the new location for the row. A migrated row is a special type of chained row. It is a chained row where no data will be found on the first block, and all of the data will be found on the second block.

In order to see what LogMiner does with chained rows, we will artificially create one. We'll start with a table that has nine CHAR(2000) columns. I am using an 8 KB block size for my database so if all nine columns have values, they will be 18,000 bytes in size, which is too big to fit on a block. This row will have to be chained onto at least three blocks. The table we'll use to demonstrate this limitation is as follows:

tkyte@TKYTE816> create table t ( x int primary key, 2 a char(2000), 3 b char(2000), 4 c char(2000), 5 d char(2000), 6 e char(2000), 7 f char(2000), 8 g char(2000), 9 h char(2000), 10 i char(2000) );

Table created.

Now, to demonstrate the issue, we'll insert a row into T with a value for only column X and column A. The size of this row will be about 2,000 plus bytes. Since B, C, D, and so on are Null, they will consume no space whatsoever. This row will fit on a block. We will then update this row, and supply values for B, C, D, and E.Since CHARs are always blank padded, this will cause the size of the row to increase from 2,000 plus bytes to 10,000 plus bytes, forcing it to chain onto two blocks. We'll then update every column in the row, growing the row to 18 KB, forcing it to span three blocks. We can then dump the redo with LogMiner, and see what it does with it:

tkyte@TKYTE816> begin 2 sys.dbms_logmnr_d.build( 'miner_dictionary.dat', 3 'c:\temp' ); 4 end; 5 /


DBMS_LOGMNR

1117


System altered.

tkyte@TKYTE816> insert into t ( x, a ) values ( 1, 'non-chained' );

1 row created.


Commit complete.

tkyte@TKYTE816> update t set a = 'chained row', 2 b = 'x', c = 'x', 3 d = 'x', e = 'x' 4 where x = 1;

1 row updated.


Commit complete.

tkyte@TKYTE816> update t set a = 'chained row', 2 b = 'x', c = 'x', 3 d = 'x', e = 'x', 4 f = 'x', g = 'x', 5 h = 'x', i = 'x' 6 where x = 1;

1 row updated.


Commit complete.


System altered.

Now that we've created the exact case we want to analyze, we can dump it via LogMiner. Don't forget, we need to rebuild your data dictionary file after we create table T, or the output will be unreadable!

tkyte@TKYTE816> declare 2 l_name v$archived_log.name%type; 3 begin 4 5 select name into l_name 6 from v$archived_log 7 where completion_time = ( select max(completion_time) 8 from v$archived_log ); 9 10 sys.dbms_logmnr.add_logfile( l_name, dbms_logmnr.NEW ); 11 end; 12 /

Appendix A

1118




tkyte@TKYTE816> select scn, sql_redo, sql_undo 2 from v$logmnr_contents 3 where sql_redo is not null or sql_undo is not null 4 /

SCN SQL_REDO SQL_UNDO---------------- --------------------------- --------------------------- 6442991118354 set transaction read write; 6442991118354 insert into TKYTE.T(X,A) va delete from TKYTE.T where X lues (1,'non-chained = 1 and A = 'non-chained '); ' and ROWID = 'AAAHdgAAGAAAACKAAA';

6442991118355 commit; 6442991118356 set transaction read write; 6442991118356 Unsupported (Chained Row) Unsupported (Chained Row) 6442991118356 Unsupported (Chained Row) Unsupported (Chained Row) 6442991118357 commit; 6442991118358 set transaction read write; 6442991118358 Unsupported (Chained Row) Unsupported (Chained Row) 6442991118358 Unsupported (Chained Row) Unsupported (Chained Row) 6442991118358 Unsupported (Chained Row) Unsupported (Chained Row) 6442991118359 commit;

12 rows selected.

As you can see, the original INSERT we did was reported by Log Miner as you would expect. The UPDATEhowever, since it caused the row to be chained, is not reported by LogMiner. Instead it reports Unsupported (Chained Row). It is interesting to note that it reports this twice for our first UPDATE, and three times for the second. LogMiner is reporting changes by database block. If your row is on two blocks, there will be two change entries in V$LOGMNR_CONTENTS. If your database block is on three blocks, then there will be three entries. You just need to be aware of the fact that LogMiner cannot faithfully reproduce the SQL to redo or undo actions against chained and migrated rows.

Other limits LogMiner has some other limitations similar to the above. In addition to the above it does not currently support:

❑ Analysis of IOTs.

❑ Analysis of clustered tables and indexes.

DBMS_LOGMNR

1119

V$LOGMNR_CONTENTSThe V$LOGMNR_CONTENTS table contains a row for every logical change to the database retrieved from the processed redo log files. We have already used this view many times, but have utilized a small fraction of the columns within it. The following table describes all of the columns available in this view with a more detailed description of what is available in them, than is available in the Oracle documentation:

PRIVATE COLUMN DESCRIPTION

SCN System Change Numbers associated with the transaction that made this change.

TIMESTAMP Date when redo record was generated. Timestamps cannot be used to infer ordering of redo records. Since the SCN is assigned upon COMMIT, only the SCN can be used to infer ordering of redo records. Ordering by timestamp in a multi-user system will result in the wrong order.

THREAD# Identifies the thread that generated the redo record.

LOG_ID Identifies the log file within the V$LOGMNR_FILES table that contains the redo record. This is a foreign key to the V$LOGMNR_FILES view.

XIDUSN Transaction ID (XID) Undo Segment Number (USN). The transaction identifier is constructed from the XIDUSN, XIDSLOT, and XIDSQN, and is used to identify the transaction that generated the change. These three fields taken together uniquely identify the transaction.

XIDSLOT Transaction ID slot number. Identifies the transaction table entry number.

XIDSQN Transaction ID sequence number.

RBASQN Uniquely identifies the log that contained this redo record, among a group of redo logs. A RBA (Redo Block Address) is composed of the RBASQN,RBABLK, and RBABYTE fields.

RBABLK The block number within the log file.

RBABYTE The byte offset within the block.

UBAFIL UBA (Undo Block Address) file number identifying the file containing the undo block. The UBA is constructed from the UBAFIL, UBABLK, UBASQN,and UBAREC, and is used to identify the undo generated for the change.

UBABLK UBA block number.

UBAREC UBA record index.

UBASQN UBA undo block sequence number.

ABS_FILE# Data block absolute file number. The ABS_FILE#, together with the REL_FILE#, DATA_BLOCK#, DATA_OBJ#, DATA_DOBJ, identify the block changed by the transaction.

REL_FILE# Data block relative file number. The file number is relative to the tablespace of the object.

Table continued on following page

Appendix A

1120


DATA_BLOCK# Data block number.

DATA_OBJ# Data block object number.

DATA_DOBJ# Data block data object number identifying the object within the tablespace.

SEG_OWNER Name of the user owning the object.

SEG_NAME Name of the structure the segment was allocated for (in other words, table name, cluster name, and so on). Partitioned tables will have a segment name constructed of two parts; the table name followed by a comma-separated partition name (for example, (TableName,PartitionName)).

SEG_TYPE The type of the segment in numeric form.

SEG_TYPE_NAME The type of segment in string form (in other words, TABLE, INDEX, and so on) Only the type, TABLE, will be supported in the initial release. Other segment types will be reported as UNSUPPORTED.

TABLE_SPACE_NAME Name of the tablespace.

ROW_ID Row ID.

SESSION# Identifies session that generated the redo. A Null value will be reported if the session number is not available from the redo log.

SERIAL# Serial number of the session, which generated the redo. The SESSION#and SERIAL# can be used to uniquely identify the Oracle session. A Null value will be reported if the session number is not available from the redo log.

USERNAME Name of the user initiating the operation that generated the redo record. The user name will always be Null if the archive auditing option is not enabled. This auditing is enabled via the init.ora parameter TRANSACTION_AUDITING.

SESSION_INFO String containing login username, client information, OS username, machine name, OS terminal, OS PID, OS program name.

ROLLBACK A value of 1 (True) identifies operations and SQL statements that were generated as a result of a rollback request, 0 (False) otherwise.

OPERATION Type of SQL operation. Only INSERT, DELETE, UPDATE, COMMIT, and BEGIN_TRANSACTION will be reported. All other operations will be reported as UNSUPPORTED or INTERNAL_OPERATION.

SQL_REDO, SQL_UNDO The SQL_REDO and SQL_UNDO columns contain SQL-compliant statements that represent the logical redo and undo operations decoded from one or more archive log records. A Null value indicates that no valid SQL statement can be generated for this redo record. Some redo records may not be translatable. In this case, the SQL_REDO and SQL_UNDO will be Null, and the STATUS column will contain the string UNSUPPORTED.

DBMS_LOGMNR

1121


RS_ID RS_ID (Record Set ID). uniquely identifies the set of records used to generate a SQL statement (a set may be a single record). It can be used to determine when multiple records generate a single SQL statement. The RS_ID will be identical for all records within the set. The SQL statement will appear only in the last row the record set. The SQL_REDO and SQL_UNDO columns for all other rows, within the set, will be Null. Note that the RS_ID/SSN pair together provide a unique SQL identifier for every SQL statement generated (see SSN).

SSN The SSN (SQL Sequence Number) can be used to identify multiple rows, with valid SQL_REDO statements, that are generated from a single redo record (in other words, array inserts, direct loads). All such rows will have the same RS_ID, but a unique SSN. The SSN is an incrementing value starting at 1 for each new RS_ID.

CSF CSF (Continuation SQL Flag) set to 1 (True) indicates that either a LogMiner-generated REDO_SQL or UNDO_SQL statement is larger than the maximum size of the VARCHAR2 (currently 4000 characters) data type. SQL statements exceeding this limit will span multiple rows.

The next row entry will contain the remainder of the SQL statement. The RS_ID, SSN pair will be identical for all continued rows corresponding to the same SQL statement. The last of the continued rows will have CSF set to 0 (False) to indicate the end of the SQL continuation.

STATUS Indicates the status of the translation. Null value indicates a successful translation, UNSUPPORTED will indicate that this version of LogMiner does not support the SQL translation, READ_FAILURE will indicate an internal operating system failure to read from the log file, TRANSLATION_ERROR will indicate that LogMiner was unable to complete the translation (this may be due to a corrupted log or an out of date dictionary file).

PH1_NAME Placeholder column name. Placeholder columns are generic columns that can be assigned to specify database table columns, via an optional LogMiner mapping file.

PH1_REDO Placeholder column redo value.

PH1_UNDO Placeholder column undo value.

PH2_NAME Placeholder column name.












Appendix A

1122

Summary LogMiner is not a tool you will use every day – I cannot anticipate any application that would actually use it as part of its processing. It is an easy way to see what the database does however, and is an excellent exploration tool in this regard. We have seen how LogMiner can be useful in finding out 'who did what and when' after the fact – this is the use of LogMiner I've seen more often than others. You have the errant program that is doing something it wasn't supposed to, or you have a privileged person doing things they shouldn't be (and not owning up to it). If auditing wasn't turned on, you have no other way to go back in time, and see what happened. In a pinch, this tool can be used to undo an errant transaction as well, given that it supplies you the SQL UNDO and REDO statements. In general, you'll find that LogMiner is a good 'stealth' tool. It won't be on the top ten list of executed procedures, but when you need it, it's good to know it is there.

DBMS_OBFUSCATION_TOOLKIT

1123


In this section, we’ll take a look at encryption of data. We will discuss the supplied DBMS_OBFUSCATION_TOOLKIT package for Oracle 8.1.6 and 8.1.7. We will look at another implementation (a wrapper) that could be placed on top of this package, increasing its functionality. We will discuss some of the caveats with regards to using this package and perhaps most importantly, we’ll touch on the subject of key management.

In Oracle 8.1.6, database packages for encryption were introduced. These packages have been enhanced for Oracle 8.1.7 to include support wider ranging encryption key sizes and MD5 hashing. In Oracle 8.1.6, support for single DES (Data Encryption Standard) encryption using a 56-bit key is provided. In Oracle 8.1.7, support for both single and triple DES encryption is provided, allowing for the use of 56, 112, or 168-bit keys. DES is implemented using a symmetric key cipher. What this means simply, is that the same key used to encrypt data, is used to decrypt data. DES encrypts data in 64-bit (8 byte) blocks using a 56-bit key. We’ll see below how this 8-byte fact affects us when using the encryption routines. The DES algorithm ignores 8 bits of the 64-bit key that is supplied. However, developers must supply a 64-bit (8 byte) key to the algorithm. Triple DES (3DES) is a far stronger cipher than DES. The resulting encrypted data is much harder to break using an exhaustive search. It would take 2**112 attempts using two-key (16 byte key) 3DES, or 2**168 attempts using three-key (24 byte key) 3DES, as opposed to 2**56 attempts with single-key DES.

To quote the executive summary of rfc1321 (for a full description of rfc1321, the website is http://www.ietf.org/rfc.html), the new MD5:

... takes as input a message of arbitrary length and produces as output a 128-bit “fingerprint” or “message digest” of the input. It is conjectured that it is computationally infeasible to produce two messages having the same message digest, or to produce any message having a given prespecified target message digest. The MD5 algorithm is intended for digital signature applications, where a large file must be “compressed” in a secure manner before being encrypted with a private (secret) key under a public-key cryptosystem such as RSA.

Appendix A

1124

In essence, MD5 is a way to verify data integrity, and is much more reliable than checksum and many other commonly used methods.

In order to run the DES3 and MD5 examples below, you will need access to an Oracle 8.1.7 database. The DES examples require the use of Oracle 8.1.6, or higher.

The encryption and MD5 routines have the following constraints that make them a bit unwieldy to use ‘out of the box’ in the DBMS_OBFUSCATION_TOOLKIT package. They are:

❑ The data being encrypted must have a length that is evenly divisible by 8. A 9-byte VARCHAR2field for example, must be padded out to 16 bytes. Any attempt to encrypt or decrypt a piece of data that does not have a length evenly divisible by 8, will fail with an error.

❑ The key used to encrypt the data must be 8 bytes long for DESEncrypt, and either 16 or 24 bytes for DES3Decrypt.

❑ There are different routines to be called depending on whether you are using 56 bit encryption (DESENCRYPT and DESDECRYPT) versus 112/168bit encryption (DES3ENCRYPTand DES3DECRYPT). I personally find it nicer to have one set of routines for all three.

❑ The encryption routines in Oracle 8.1.6 are procedures and therefore, they are not callable from SQL (procedures cannot be called from SQL).

❑ The ‘out of the box’ encryption routines support encryption of up to 32 KB of data. They do not encrypt/decrypt LOBs.

❑ The encryption routines in Oracle 8.1.7 include functions. However, these functions are overloaded in such a way (see the example below) that also makes them not callable from SQL.

❑ The MD5 routines are likewise overloaded in such a way as to make them not callable from SQL.

I find the first constraint, that the data length must be a multiple of 8, to be the hardest to satisfy in an application. Typically, I just have some data, like a salary amount or some other sensitive data, and want to encrypt it. I do not really want to be bothered with ensuring it is a multiple of 8 bytes in length. Fortunately, we can easily implement our own encryption wrapper package to hide this, and most all of the other issues. The fact that the key must be 8, 16 or, 24 bytes in length is something that you must do yourself, however.

What I intend to do here is to create a wrapper package, that is installable in 8.1.6 and later versions, which provides support for all of the encryption functionality, and adds support for:

❑ Calling the functions from SQL

❑ Single function calls regardless of the key length

❑ Encryption/decryption of LOBs callable from both PL/SQL and SQL

❑ Installing successfully regardless of which version of the database (8.1.6 or 8.1.7) you are using. That is, it is not dependent on DES3Encrypt/Decrypt, and MD5 support.


1125

The Wrapper We’ll start with the package specification. Here we will define an API that provides functions to encrypt and decrypt VARCHAR, RAW, BLOB, and CLOB data. The algorithm used (single key DES or 3DES with 16 or 24 byte keys) will be decided, based on the key length. In our API, the length of the key you send in implies the algorithm.

The API is set up so that the key may be passed in with each call, or it may optionally be set for the package by calling SETKEY. The advantage of using SETKEY is that a certain amount of work must be performed to look at the key length and figure out which algorithm to use. If you set the key once and call the routines many times, we can avoid performing this iterative work over and over. Another detail about the key we must use is that if you are working with RAW or BLOB data, you must use a RAW key. If you want to use a VARCHAR2 as the key for RAW/BLOB data, you must cast it to be a RAW using the UTL_RAW package discussed in a later section of this appendix. On the other hand, if you are working with VARCHAR2 and CLOB data, the key must be a VARCHAR2.

In addition to providing a layer on top of encryption, this package provides access to the MD5 CHECKSUM routines if installed (version 8.1.7 and up).

This wrapper package adds a couple of new possible errors to the documented set of DBMS_OBFUSCATION_TOOLKIT errors (which this package will simply propagate). The following ‘new’ errors will only occur when using version 8.1.6:

❑ PLS-00302: component ‘MD5’ must be declared

❑ PLS-00302: component ‘DES3ENCRYPT’ must be declared

❑ PLS-00302: component ‘THREEKEYMODE’ must be declared

You will get these errors if you attempt to use the 8.1.7 functionality of DES3 encryption or MD5 hashing, in the 8.1.6 database.

Here is our suggested wrapper package specification. Explanation of the procedures and functions listed below follow the code:

create or replace package crypt_pkgas

function encryptString( p_data in varchar2, p_key in varchar2 default NULL ) return varchar2; function decryptString( p_data in varchar2, p_key in varchar2 default NULL ) return varchar2;

function encryptRaw( p_data in raw, p_key in raw default NULL ) return raw; function decryptRaw( p_data in raw, p_key in raw default NULL ) return raw;

function encryptLob( p_data in clob, p_key in varchar2 default NULL ) return clob; function encryptLob( p_data in blob, p_key in raw default NULL ) return blob; function decryptLob( p_data in clob, p_key in varchar2 default NULL ) return clob; function decryptLob( p_data in blob,

Appendix A

1126

p_key in raw default NULL ) return blob;

subtype checksum_str is varchar2(16); subtype checksum_raw is raw(16);

function md5str( p_data in varchar2 ) return checksum_str;function md5raw( p_data in raw ) return checksum_raw;function md5lob( p_data in clob ) return checksum_str;function md5lob( p_data in blob ) return checksum_raw;

procedure setKey( p_key in varchar2 );

end;/

The functions ENCRYPTSTRING and DECRYPTSTRING are used to encrypt/decrypt any STRING, DATE,or NUMBER data up to 32 KB in size. 32 KB is the maximum size of a PL/SQL variable, and is considerably larger than the maximum size that can be stored in a database table, where the limit is 4,000 bytes. These functions are callable from SQL directly so you’ll be able to encrypt data in the database using an INSERT or UPDATE statement, and retrieve decrypted data using a simple SELECT.The KEY parameter is optional. If you have set a key via the SETKEY procedure, we do not need to pass it with each and every call.

Next we have ENCRYPTRAW and DECRYPTRAW. These functions are to the RAW data type what the previous two functions are to VARCHAR2s. Note how we purposely avoided function overloading the encrypt/decrypt routines for RAW and VARCHAR2 data by naming them differently. We did this because of the following issue:

tkyte@TKYTE816> create or replace package overloaded 2 as 3 function foo( x in varchar2 ) return number; 4 function foo( x in raw ) return number; 5 end; 6 /

Package created.

tkyte@TKYTE816> select overloaded.foo( ‘hello’ ) from dual; select overloaded.foo( ‘hello’ ) from dual * ERROR at line 1: ORA-06553: PLS-307: too many declarations of ‘FOO’ match this call

tkyte@TKYTE816> select overloaded.foo( hextoraw( ‘aa’ ) ) from dual; select overloaded.foo( hextoraw( ‘aa’ ) ) from dual * ERROR at line 1: ORA-06553: PLS-307: too many declarations of ‘FOO’ match this call

The database does not distinguish between RAW and VARCHAR2 in the signature of an overloaded function. We would have no way to call these functions from SQL. Even if we used different parameter names for the inputs to these routines (as DBMS_OBFUSCATION_TOOLKIT currently does), they cannot be called from SQL because named parameter notation cannot be used in SQL. The only viable solution to this conundrum is to use functions with unique names to identify the routine we really want.


1127

Next we have the ENCRYPTLOB and DECRYPTLOB functions. These are overloaded functions designed to work with either CLOBs or BLOBs. Oracle is able to successfully overload, based on these types, so we’ll take advantage of this fact. Since we are limited to encrypting at most 32 KB of data by the DBMS_OBFUSCATION_TOOLKIT routine, these wrapper APIs will implement an algorithm that encrypts 32 KB chunks of a LOB. The resulting LOB will be a series of encrypted 32 KB chunks of data. The decrypt wrapper we implement understands how the data was packed by the LOB encryption routines, and will decrypt the pieces for us and re-assemble them back into our original LOB.

Next, we have the routines for the MD5 CHECKSUMs. To better define what these routines return, we’ve set up the subtypes:

subtype checksum_str is varchar2(16); subtype checksum_raw is raw(16);

and defined our routines to return these types. You can declare variables of this type yourself:

tkyte@TKYTE816> declare 2 checksum_variable crypt_pkg.checksum_str; 3 begin 4 null; 5 end; 6 /


This saves you from having to guess how big the checksum return types are. We’ve provided for four different CHECKSUM routines, one each for VARCHAR2 (includes the DATE and NUMBER types), RAW,CLOB, and BLOB data. It should be noted that the MD5 checksum will only be computed on the first 32 KB of the CLOB or BLOB data, as this is the largest variable PL/SQL can work with.

The implementation of the package below will not only give us an easier to use encryption package, but it also shows a couple of useful concepts. Firstly, it shows how you can easily create your own wrapper layer to provide for a more customized interface to the database packages. In this case, we are working around some perceived limitations of the DBMS_OBFUSCATION_TOOLKIT package. Secondly, it shows one method for developing a package that is protected from enhancements in the supplied database packages over time. We would like to provide a single wrapper package that works both in 8.1.6 and 8.1.7, but which provides total access to the 8.1.7 functionality. If we used static SQL to access the DESENCRYPT, DES3DECRYPT, and MD5 routines, we would need a different package for 8.1.6 because MD5 and the DES3 functions do not exist in 8.1.6. The dynamic invocation we’ll utilize below allows us to develop a package that can be used by both versions of the database. It also reduces the amount of code has to be written.

Here is the implementation of the CRYPT_PKG with explanations of what is taking place intermixed with the code:

create or replace package body crypt_pkgas-- package globals g_charkey varchar2(48); g_stringFunction varchar2(1); g_rawFunction varchar2(1); g_stringWhich varchar2(75); g_rawWhich varchar2(75); g_chunkSize CONSTANT number default 32000;

Appendix A

1128

The package begins with a few global variables. They are:

❑ G_CHARKEY – This stores the RAW or VARCHAR2 key for use by the encryption routines. It is 48 bytes long to support holding a 24 byte RAW key (which will be doubled in length due to the hexadecimal conversion caused by placing a RAW string in a VARCHAR2 variable).

❑ G_STRINGFUNCTION and G_RAWFUNCTION – Contains either Null or the string ‘3’ after a call to SETKEY. We will dynamically add this string to the routine name at run-time so we either run DESENCRYPT or DES3ENCRYPT, depending on the key size. In short, this is used to construct the appropriate function name we need to call.

❑ G_STRINGWHICH and G_RAWWHICH – Used with DES3EN/DECRYPT only. Adds the fourth optional parameter to force 3 key mode, when doing 3DES in 3 key mode. Whereas the string function variables above tell us whether to call DESENCRYPT or DES3ENCRYPT, this tells us what value we need to pass in for the key mode (two key or three key).

❑ G_CHUNKSIZE – A constant that controls the size of the LOB chunk we’ll encrypt/decrypt. It also controls the maximum amount of data sent to the MD5 checksum routines when working with LOBs. It is crucial that this number is a multiple of 8 – the implementation below counts on it!

Continuing on, we have six small ‘private’ routines to implement. These are helper functions used by the other routines in the package:

function padstr( p_str in varchar2 ) return varchar2 as l_len number default length(p_str); begin return to_char(l_len,’fm00000009’) || rpad(p_str, ( trunc(l_len/8)+sign(mod(l_len,8)) )*8, chr(0)); end;

function padraw( p_raw in raw ) return raw as l_len number default utl_raw.length(p_raw);begin return utl_raw.concat( utl_raw.cast_to_raw(to_char(l_len,’fm00000009’)), p_raw, utl_raw.cast_to_raw( rpad(chr(0), (8-mod(l_len,8))*sign(mod(l_len,8)), chr(0))));

end;

It you recall from the description of the DES encryption algorithm, it was stated that ‘DES encrypts data in 64-bit (8 byte) blocks...’ A side effect of this is that the DBMS_OBFUSCATION_TOOLKIT package works only on data whose length is a multiple of 8. If you have a string that is 7 bytes long, it must be padded to 8 bytes. A 9 byte string must be padded out to 16 bytes. The above two routines encode and pad out strings and RAW data. They encode the string or RAW by placing the original length into the string/RAW itself. Then they pad out the string with binary zeros (CHR(0)) to make it a multiple of 8 bytes in length. For example, the string Hello World will be encoded as follows:


1129

tkyte@TKYTE816> select length(padstr), padstr, dump(padstr) dump 2 from 3 ( select to_char(l_len,’fm00000009’) || 4 rpad(p_str, 5 (trunc(l_len/8)+sign(mod(l_len,8)) )*8, 6 chr(0)) padstr 7 from ( select length( ‘Hello World’ ) l_len, 8 ‘Hello World’ p_str 9 from dual 10 ) 11 ) 12 /

LENGTH(PADSTR) PADSTR DUMP -------------- ------------------------- ------------------------------ 24 00000011Hello World Typ=1 Len=24: 48,48,48,48,48,4 8,49,49,72,101,108,108,111,32, 87,111,114,108,100,0,0,0,0,0

The final length of the encoded string is 24 bytes (LENGTH(PADSDTR)), and the original length was 11 (this is visible in the first 8 characters of the PADSTR column). Looking at the DUMP column, which shows the ASCII values of the bytes in the string, we can see it ends with 5 binary zeroes. We needed to pad out 5 bytes to make the 11 byte Hello World string a multiple of 8. Next, we have the routines that ‘undo’ the padding we did above:

function unpadstr( p_str in varchar2 ) return varchar2 isbegin return substr( p_str, 9, to_number(substr(p_str,1,8)) ); end;

function unpadraw( p_raw in raw ) return raw isbegin return utl_raw.substr( p_raw, 9, to_number( utl_raw.cast_to_varchar2(utl_raw.substr(p_raw,1,8)) ) ); end;

They are straightforward enough. They assume the first 8 bytes of the string or RAW is the original length of the string, and return the SUBSTR of this encoded data appropriately.

Continuing on, we have the last of our internal helper routines:

procedure wa( p_clob in out clob, p_buffer in varchar2 )isbegin dbms_lob.writeappend(p_clob,length(p_buffer),p_buffer);end;

procedure wa( p_blob in out blob, p_buffer in raw )isbegin dbms_lob.writeappend(p_blob,utl_raw.length(p_buffer),p_buffer); end;

Appendix A

1130

These simply make it easier to call DBMS_LOB.WRITEAPPEND, by shortening the name to WA, and by passing in the length of the buffer to write for us, which in our case is always the current length of the buffer.

Now we hit our first externally callable routine SETKEY:

procedure setKey( p_key in varchar2 ) asbegin if ( g_charkey = p_key OR p_key is NULL ) then return; end if; g_charkey := p_key;

if ( length(g_charkey) not in ( 8, 16, 24, 16, 32, 48 ) ) then raise_application_error( -20001, ‘Key must be 8, 16, or 24 bytes’ ); end if;

select decode(length(g_charkey),8,’’,’3’), decode(length(g_charkey),8,’’,16,’’, 24,’, which=>dbms_obfuscation_toolkit.ThreeKeyMode’), decode(length(g_charkey),16,’’,’3’), decode(length(g_charkey),16,’’,32,’’, 48,’, which=>dbms_obfuscation_toolkit.ThreeKeyMode’) into g_stringFunction, g_stringWhich, g_rawFunction, g_rawWhich from dual; end;

This routine is used whether you call it or not. The remaining externally callable routines below will call SETKEY regardless of whether you do or not. This routine will compare your key P_KEY to the one in the global variable G_CHARKEY. If they compare, or no key was provided, this routine simply returns. It has no work to perform. If the P_KEY is different from G_CHARKEY however, this routine will continue. The first thing it does is a sanity check to verify that the key is a valid multiple of 8. The key must be 8, 16 or 24 bytes in length. Since this routine may be passed RAW data, which causes each byte to be expanded into a 2 byte hexadecimal code, 16, 32, and 48 are valid lengths as well. This check does not fully guarantee the key will work however. For example, you could send a 4 byte RAW key that will appear as 8 bytes to us. You will get a run-time error from DBMS_OBFUSCATION_TOOLKIT later in that case.

The SELECT with a DECODE is a used to set up the remaining global variables. Since we cannot tell the difference between a RAW and VARCHAR2 string at this point, we set up all four possible variables. The key thing to note about this piece of code is that if the length of the key is 8 bytes (16 bytes when RAW), then the FUNCTION variable will be set to a Null string. If the key length is 16 or 24 bytes (32 or 48 bytes when RAW), the FUNCTION variable will be set to the string ‘3’. This is what will cause us to call DESENCRYPT or DES3Encrypt later. The other thing to notice here is the setting of the WHICH global variable. This is used to set the optional parameter to the DES3ENCRYPT routine. If the key length is 8 or 16 bytes (16 or 32 bytes RAW), we set this string to Null – we do not pass a parameter. If the key length is 24 bytes (48 bytes RAW), we set this string to pass THREEKEYMODE to the ENCRYPT/DECRYPTroutines to instruct them to use this larger key.


1131

Now we are ready to see the functions that do the actual work for us:

function encryptString( p_data in varchar2, p_key in varchar2 default NULL ) return varchar2 as l_encrypted long; begin setkey(p_key); execute immediate ‘begin dbms_obfuscation_toolkit.des’ || g_StringFunction || ‘encrypt ( input_string => :1, key_string => :2, encrypted_string => :3’ || g_stringWhich || ‘ ); end;’ using IN padstr(p_data), IN g_charkey, IN OUT l_encrypted;

return l_encrypted;end;

function encryptRaw( p_data in raw, p_key in raw default NULL ) return raw as l_encrypted long raw; begin setkey(p_key); execute immediate ‘begin dbms_obfuscation_toolkit.des’ || g_RawFunction || ‘encrypt ( input => :1, key => :2, encrypted_data => :3’ || g_rawWhich || ‘ ); end;’ using IN padraw( p_data ), IN hextoraw(g_charkey), IN OUT l_encrypted;

return l_encrypted;end;

The ENCRYPTSTRING and ENCRYPTRAW functions work in a similar manner to each other. They both dynamically call either DESENCRYPT or DES3ENCRYPT. This dynamic call not only reduces the amount of code we have to write as it avoids the IF THEN ELSE we would have use to statically call either routine, but it also makes it so the package can be installed in 8.1.6 or 8.1.7, without change. Since we do not statically reference DBMS_OBFUSCATION_TOOLKIT, we can compile against either version. This dynamic invocation is a technique that is useful any time you are not certain what might or might not be installed in the database. I’ve used it in the past when writing utility routines that needed to be installed in 7.3, 8.0, and 8.1. Over time, additional functionality was added to the core packages, and when the code was running in 8.1, we wanted to take advantage of it. When we were running in 7.3, the code would still function; it just wouldn’t be able to benefit from the newer functionality. The same concept applies here. When installed in an 8.1.7 database, the above code can, and will call DES3ENCRYPT.When installed in 8.1.6, any attempt to invoke the DES3ENCRYPT will result in a run-time error (instead of preventing you from installing this package). The calls to DESENCRYPT will function as expected in 8.1.6.

These functions work simply by creating a dynamic string using the FUNCTION and WHICH we set in the SETKEY routine. We will either add the number 3 to the procedure name, or not. We will add the optional fourth parameter to DES3ENCRYPT when we want three key mode. Then we execute the string,

Appendix A

1132

send the data and key to be encrypted, and receive the encrypted data as output. Notice how we bind the PADSTR or PADRAW of the original data. The data that is encrypted is the encoded string, which is padded out to the proper length.

Now for the inverse of the above two functions:

function decryptString( p_data in varchar2, p_key in varchar2 default NULL ) return varchar2 as l_string long; begin setkey(p_key); execute immediate

‘begin dbms_obfuscation_toolkit.des’ || g_StringFunction || ‘decrypt ( input_string => :1, key_string => :2, decrypted_string => :3’ || g_stringWhich || ‘ ); end;’ using IN p_data, IN g_charkey, IN OUT l_string;

return unpadstr( l_string ); end;

function decryptRaw( p_data in raw, p_key in raw default NULL ) return raw as l_string long raw; begin setkey(p_key); execute immediate

‘begin dbms_obfuscation_toolkit.des’ || g_RawFunction || ‘decrypt ( input => :1, key => :2, decrypted_data => :3 ‘ || g_rawWhich || ‘ ); end;’ using IN p_data, IN hextoraw(g_charkey), IN OUT l_string;

return unpadraw( l_string ); end;

DECRYPTSTRING and DECRYPTRAW work in a similar manner as the ENCRYPT routines above functionally. The only difference is they call DECRYPT instead of ENCRYPT in the DBMS_OBFUSCATION_TOOLKIT package, and call UNPAD to decode the string or RAW data.

Now onto the routines for encrypting LOBs:


1133

function encryptLob( p_data in clob, p_key in varchar2 ) return clob as l_clob clob; l_offset number default 1; l_len number default dbms_lob.getlength(p_data);begin setkey(p_key); dbms_lob.createtemporary( l_clob, TRUE ); while ( l_offset <= l_len ) loop wa( l_clob, encryptString( dbms_lob.substr( p_data, g_chunkSize, l_offset ) ) ); l_offset := l_offset + g_chunksize; end loop; return l_clob;end;

function encryptLob( p_data in blob, p_key in raw ) return blob as l_blob blob; l_offset number default 1; l_len number default dbms_lob.getlength(p_data);begin setkey(p_key); dbms_lob.createtemporary( l_blob, TRUE ); while ( l_offset <= l_len ) loop wa( l_blob, encryptRaw( dbms_lob.substr( p_data, g_chunkSize, l_offset ) ) ); l_offset := l_offset + g_chunksize; end loop; return l_blob;end;

These are overloaded procedures for BLOBs and CLOBs. They work by creating a temporary LOB to write the encrypted data into. Since we change the length of a string/RAW data when encrypted to preserve its original length and pad it out, doing this ‘in place’ using the existing LOB would not be possible. For example, if you had a 64 KB LOB, we would take the first 32 KB, and make it ‘larger’ than 32 KB. Now we would need to slide over the last 32 KB of the existing LOB to make room for this larger chunk of data. Also, it would make it not possible to call these functions from SQL, since the LOBlocator would have to be IN/OUT, and IN/OUT parameters would preclude this from being called from SQL. So, we simply copy the encrypted data into a new LOB which the caller can use anywhere, even in an INSERT or UPDATE statement.

The algorithm used to encrypt and encode the LOB data is as follows. We start at byte 1 (L_OFFSET) and encrypt G_CHUNKSIZE bytes of data. This is appended to the temporary LOB we created. We add G_CHUNKSIZE to the offset, and continue looping until we have processed the entire LOB. At the end, we return the temporary LOB to the caller.

Appendix A

1134

Next for the decryption routines for LOB data:

function decryptLob( p_data in clob, p_key in varchar2 default NULL ) return clob as l_clob clob; l_offset number default 1; l_len number default dbms_lob.getlength(p_data);begin setkey(p_key); dbms_lob.createtemporary( l_clob, TRUE ); loop exit when l_offset > l_len; wa( l_clob, decryptString( dbms_lob.substr( p_data, g_chunksize+8, l_offset ) ) ); l_offset := l_offset + 8 + g_chunksize; end loop; return l_clob;end;

function decryptLob( p_data in blob, p_key in raw default NULL ) return blob as l_blob blob; l_offset number default 1; l_len number default dbms_lob.getlength(p_data);begin setkey(p_key); dbms_lob.createtemporary( l_blob, TRUE ); loop exit when l_offset > l_len; wa( l_blob, decryptRaw( dbms_lob.substr( p_data, g_chunksize+8, l_offset ) ) ); l_offset := l_offset + 8 + g_chunksize; end loop; return l_blob;end;

Once again, for the same reasons as before, we utilize a temporary LOB to perform the decryption. This time however, there is one additional reason for the temporary LOB. If we did not use a temporary LOBto decrypt the data into, we would actually be decrypting the data in the DATABASE. Subsequent SELECTs would see already decrypted data if we didn’t copy it into a new LOB! Here, the temporary LOB usage is even more important than before.

The logic employed here is to loop over the chunks in the LOB as before. We start at offset 1 (the first byte) in the LOB and SUBSTR off G_CHUNKSIZE+8 bytes. The additional 8 bytes caters for the 8 bytes the PADSTR/PADRAW functions added to the data when we encoded it. So, all we do is walk through the LOB G_CHUNKSIZE+8 bytes at a time, decrypting the data, and appending it to the temporary LOB. This is what gets returned to the client.

And now for the last part of the CRYPT_PKG, the interface to the MD5 routines:


1135

function md5str( p_data in varchar2 ) return checksum_stris l_checksum_str checksum_str;begin execute immediate ‘begin :x := dbms_obfuscation_toolkit.md5( input_string => :y ); end;’ using OUT l_checksum_str, IN p_data;

return l_checksum_str; end;

function md5raw( p_data in raw ) return checksum_rawis l_checksum_raw checksum_raw;begin execute immediate ‘begin :x := dbms_obfuscation_toolkit.md5( input => :y ); end;’ using OUT l_checksum_raw, IN p_data;

return l_checksum_raw; end;

function md5lob( p_data in clob ) return checksum_stris l_checksum_str checksum_str;begin execute immediate ‘begin :x := dbms_obfuscation_toolkit.md5( input_string => :y ); end;’ using OUT l_checksum_str, IN dbms_lob.substr(p_data,g_chunksize,1);

return l_checksum_str; end;

function md5lob( p_data in blob ) return checksum_rawis l_checksum_raw checksum_raw;begin execute immediate ‘begin :x := dbms_obfuscation_toolkit.md5( input => :y ); end;’ using OUT l_checksum_raw, IN dbms_lob.substr(p_data,g_chunksize,1);

return l_checksum_raw; end;

end;/

The MD5 routines act as a passthrough to the native DBMS_OBFUSCATION_TOOLKIT routines. The one thing they do differently is that they are not overloaded, allowing them to be called directly from SQL. You should note that the MD5 LOB routines only compute the CHECKSUM based on the first G_CHUNKSIZE bytes of data. This is due to the limitation of PL/SQL in regards to variable sizes.

Now we will briefly test out and demonstrate the functionality of this package. The following examples were executed in an Oracle 8.1.7 database. If executed in 8.1.6, you should expect the DES3 and MD5examples to fail at run-time:

Appendix A

1136

tkyte@DEV817> declare 2 l_str_data varchar2(25) := ‘hello world’; 3 l_str_enc varchar2(50); 4 l_str_decoded varchar2(25); 5 6 l_raw_data raw(25) := utl_raw.cast_to_raw(‘Goodbye’); 7 l_raw_enc raw(50); 8 l_raw_decoded raw(25); 9 10 begin 11 crypt_pkg.setkey( ‘MagicKey’ ); 12 13 l_str_enc := crypt_pkg.encryptString( l_str_data ); 14 l_str_decoded := crypt_pkg.decryptString( l_str_enc ); 15 16 dbms_output.put_line( ‘Encoded In hex = ‘ || 17 utl_raw.cast_to_raw(l_str_enc) ); 18 dbms_output.put_line( ‘Decoded = ‘ || l_str_decoded ); 19 20 crypt_pkg.setkey( utl_raw.cast_to_raw(‘MagicKey’) ); 21 22 l_raw_enc := crypt_pkg.encryptRaw( l_raw_data ); 23 l_raw_decoded := crypt_pkg.decryptRaw( l_raw_enc ); 24 25 dbms_output.put_line( ‘Encoded = ‘ || l_raw_enc ); 26 dbms_output.put_line( ‘Decoded = ‘ || 27 utl_raw.cast_to_varchar2(l_raw_decoded) ); 28 end; 29 / Encoded In hex = 7004DB310AC6A8F210F8467278518CF988DF554B299B35EF Decoded = hello world Encoded = E3CC4E04EF3951178DEB9AFAE9C99096 Decoded = Goodbye


This shows the basic functionality of the ENCRYPT and DECRYPT routines. Here, I am calling them procedurally – below we will do it in SQL. I test against both string and RAW data in this example. On line 11 of the code, I call SETKEY to set the encryption key, to be used for encoding and decoding of the VARCHAR2 data elements, to the string MAGICKEY. This saves me from having to repeatedly pass this string into these routines. Then I encrypt the string into L_STR_ENC. I then decrypt this string just to make sure everything is working as expected. On lines 16-18 I print out the results. Since the encrypted data can contain various characters that drive terminal emulators crazy, I print out the encrypted string using UTL_RAW.CAST_TO_RAW on line 17. This has the effect of just changing the data type of the VARCHAR2 into a RAW, as mentioned previously. The underlying data is not changed at all. Since RAWdata is implicitly converted to a string of hexadecimal digits, we can use this as a convenient way to dump data to the screen in hexadecimal.

On lines 20 through 27 I do the same thing to RAW data. I must call SETKEY once again, this time with 8 bytes of RAW data. For convenience sake, I use UTL_RAW.CAST_TO_RAW to change a VARCHAR2 key into a RAW key. I could have used HEXTORAW and passed a string of hexadecimal characters as well. I then encrypt the data, and decrypt the encrypted data. When I print it out, I just print the encrypted data (it’ll display in hexadecimal), and I cast the data I decrypted back to VARCHAR2 so we can see that it worked. The output confirms the package functions.


1137

Next, we’ll look at how this might work in SQL. We’ll test out the triple DES encryption in two key mode this time:

tkyte@DEV817> drop table t;

Table dropped.

tkyte@DEV817> create table t 2 ( id int primary key, data varchar2(255) );

Table created.

tkyte@DEV817> insert into t values 2 ( 1, crypt_pkg.encryptString( ‘This is row 1’, ‘MagicKeyIsLonger’ ) );

1 row created.

tkyte@DEV817> insert into t values 2 ( 2, crypt_pkg.encryptString( ‘This is row 2’, ‘MagicKeyIsLonger’ ) );

1 row created.

tkyte@DEV817> select utl_raw.cast_to_raw(data) encrypted_in_hex, 2 crypt_pkg.decryptString(data,’MagicKeyIsLonger’) decrypted 3 from t 4 /

ENCRYPTED_IN_HEX DECRYPTED ------------------------------------------------ ------------- 0B9A809515519FA6A34F150941B318DA441FBB0C790E9481 This is row 1 0B9A809515519FA6A34F150941B318DA20A936F9848ADC13 This is row 2

So, simply by using a 16 byte key as input to the CRYPT_PKG.ENCRYPTSTRING routine, we automatically switched over to the DES3ENCRYPT routine, within the DBMS_OBFUSCATION_TOOLKITpackage. This example shows how easy it is to use the CRYPT_PKG in SQL. All of the functions are callable from SQL and can be used anywhere where SUBSTR, for example, could be used. The CRYPT_PKG could be used in the SET clause of an UPDATE, the VALUES clause of an INSERT, the SELECT clause of a SQL query, and even in the WHERE clause of any statement if you like.

Now we will look at how this package may be used on LOBs and demonstrate the MD5 routines as well. We’ll use a 50 KB CLOB as our test case. First we must load the LOB into the database:

tkyte@DEV817> create table demo ( id int, theClob clob );

Table created.

tkyte@DEV817> create or replace directory my_files as 2 ‘/d01/home/tkyte’;

Directory created.

tkyte@DEV817> declare 2 l_clob clob; 3 l_bfile bfile;

Appendix A

1138

4 begin 5 insert into demo values ( 1, empty_clob() ) 6 returning theclob into l_clob; 7 8 l_bfile := bfilename( ‘MY_FILES’, ‘htp.sql’ ); 9 dbms_lob.fileopen( l_bfile ); 10 11 dbms_lob.loadfromfile( l_clob, l_bfile, 12 dbms_lob.getlength( l_bfile ) ); 13 14 dbms_lob.fileclose( l_bfile ); 15 end; 16 /


The above procedure has loaded some data into the CLOB. Now we would like to perform some operations on it. Again, we will use SQL, as this is a fairly natural way to interact with the data. We’ll start by computing a CHECKSUM based on the first 32 KB of the CLOB:

tkyte@DEV817> select dbms_lob.getlength(theclob) lob_len, 2 utl_raw.cast_to_raw( crypt_pkg.md5lob(theclob) ) md5_checksum 3 from demo;

LOB_LEN MD5_CHECKSUM---------- -------------------------------- 50601 307D19748889C2DEAD879F89AD45D1BA

Again, we use the UTL_RAW.CAST_TO_RAW to convert the VARCHAR2 returned from the MD5 routines into a hexadecimal string for display. The VARCHAR2 string will most likely contain data that is ‘unprintable’ on your terminal, or it may contain embedded newlines, tabs, and other control characters. The above code shows how easy it is to use the MD5 routines – just send it some data and it will compute the CHECKSUM.

Next, we want to see how one might encrypt and decrypt a LOB. We’ll do it with a simple UPDATE.Notice that this time our encryption key is 24 bytes long. We will be using the DES3ENCRYPT routine with the optional which => ThreeKeyMode parameter set. This gives us 3 key, triple DES encryption:

tkyte@DEV817> update demo 2 set theClob = crypt_pkg.encryptLob( theClob, 3 ‘MagicKeyIsLongerEvenMore’ ) 4 where id = 1;

1 row updated.

tkyte@DEV817> select dbms_lob.getlength(theclob) lob_len, 2 utl_raw.cast_to_raw( crypt_pkg.md5lob(theclob) ) md5_checksum 3 from demo;

LOB_LEN MD5_CHECKSUM---------- -------------------------------- 50624 FCBD33DA2336C83685B1A62956CA2D16


1139

Here we can see by the fact that the length has changed from 50,601 to 50,624 bytes, and that the MD5CHECKSUM is different, that we have in fact modified the data. What we did, if you recall from the algorithm above, is to take the first 32,000 bytes of the CLOB, added 8 bytes onto the front of this 32,000 bytes as part of the string encryption, and encrypted it. Then, we retrieved the remaining 18,601 bytes. We padded this out to 18,608 bytes (divisible by 8 evenly), and added 8 bytes to remember the original length. This gives us our expanded length of 50,624 bytes.

Lastly, we will look at how to retrieve the CLOB decrypted from the database:

tkyte@DEV817> select dbms_lob.substr( 2 crypt_pkg.decryptLob(theClob), 100, 1 ) data 3 from demo 4 where id = 1;

DATA----------------------------------------set define off create or replace package htp as /* STRUCTURE tags */ procedure htmlOpen; procedure

An interesting thing to note here is that I did not pass in the encryption key. Since we save this key in the package state, it is not necessary here. The package will remember it from call to call, but not session to session. I could send the key, but I do not need to. The key is stored in a package body global variable, so it is not visible to anything other than the functions in the package body, and cannot be seen by other sessions.

Caveats Currently, there exists a situation with the DBMS_OBFUSCATION_TOOLKIT whereby data encrypted on a ‘little endian’ system cannot be decrypted using the same key on a ‘big endian’ system. ‘Endian’ has to do with he ordering of bytes in a multi-byte number. Intel platforms (NT, many Linuxes, and Solaris x86 run on Intel) have a little endian byte ordering. Sparc and Risc typically have a big endian. Data that is encrypted on Windows NT using a key of ‘12345678’ cannot be decrypted on Sparc Solaris using the same key. The following example demonstrates the issue (and shows the workaround). On Windows NT:

tkyte@TKYTE816> create table anothert ( encrypted_data varchar2(25) );

Table created.

tkyte@TKYTE816> insert into anothert values 2 ( crypt_pkg.encryptString( ‘hello world’, ‘12345678’ ) );

1 row created.

tkyte@TKYTE816> select crypt_pkg.decryptstring( encrypted_data ) from anothert;

CRYPT_PKG.DECRYPTSTRING(ENCRYPTED_DATA) -------------------------------------------------------------------------hello world

tkyte@TKYTE816> host exp userid=tom/kyte tables=anothert

Appendix A

1140

I ftp this EXPDAT.DMP file to my Sparc Solaris machine, and load the data into it. Then, when I attempt to query it out I receive:

ops$tkyte@DEV816> select 2 crypt_pkg.decryptstring( encrypted_data, ‘12345678’ ) 3 from t; crypt_pkg.decryptstring( encrypted_data, ‘12345678’ ) *ERROR at line 2: ORA-06502: PL/SQL: numeric or value error: character to number conversionORA-06512: at “OPS$TKYTE.CRYPT_PKG”, line 84 ORA-06512: at “OPS$TKYTE.CRYPT_PKG”, line 215 ORA-06512: at line 1

ops$tkyte@DEV816> select 2 crypt_pkg.decryptstring( encrypted_data, ‘43218765’ ) 3 from t;

CRYPT_PKG.DECRYPTSTRING(ENCRYPTED_DATA,’43218765’) ------------------------------------------------------------------------helloworld

The error above is coming from my wrapper package. I am taking the first 8 bytes of data in the string, and assuming that it is a number. Since the key could not successfully decrypt the data, the first 8 bytes is in fact not my length field – it is some garbage set of characters.

Apparently, our 8 byte (or 16 or 24 byte) key is addressed internally as a series of 4 byte integers. We must reverse the bytes in every 4 byte group in our key, in order to decrypt data on one system that was encrypted on another with a different byte order. Therefore, if I use the key ‘12345678’ on Windows NT (Intel), I must use the key ‘43218765’ on Sparc Solaris. We take the first 4 bytes and reverse them, then take the next 4 bytes and reverse them (and so on for larger keys).

This is an important fact to keep in mind if you either move the data from NT to Sparc Solaris for example, or use database links to query the data out. You must be prepared to physically reorder the bytes to decrypt successfully. This particular issue is corrected in the patch release of Oracle 8.1.7.1 and up, and the need for the byte swapping is removed.

Key Management I would like to briefly talk about key management here for a moment. Encryption is only part of the solution to making data ‘secure’. The primary reason people quote for encrypting data in the database is to make it so the DBA, who can query any table, cannot make sense of the data in the table. For example, you are an online web site taking customer orders. People are giving you their credit card numbers. You are storing them in the database. You would like to assure that neither the DBA, who must be able to backup your database, nor the malicious hacker that breaks into your database, could read this highly sensitive information. If you stored it in clear text, it would be easy for someone to see it, if they gained DBA access to your database. If it is stored encrypted this would not be the case.

The encrypted data is only as secure as the key you use to encrypt the data. The key that is used is the magic component here. If the key is discovered, the data might as well not be encrypted (as evidenced by the fact that we can simply select to decrypt data if we were given the key).


1141

Therefore key generation and key protection are two things you must give a lot of thought to. There are many paths you could follow. What follows are some ideas, and concepts that you can use, but each has its own specific issues.

The Client Application Manages and Stores Keys One approach is to keep the keys out of the database, perhaps even on another machine (just don’t lose them – it’ll take you a couple hundred CPU years to guess what they might be!). In this scenario, the client application, be it software in a middle tier application server, or a client-server application, manages the keys on its system. The client software determines if the user accessing the data is permitted to decrypt it or not, and sends the key to the database.

If you choose to do this, to transmit the key over the network, we must add yet one more layer of encryption and this is data stream encryption for the Net8 traffic. Bind variables and string literals are transmitted unencrypted by default. In this situation, since the keys are so critical, you would need to make use of a technology such as the ASO (Advanced Security Option). This Net8 option provides full data stream encryption so that no one can ‘sniff’ your keys from the network.

As long as the key is securely maintained by the client application (this is up to you to ensure), and you use ASO, this would be a viable solution.

Store the Keys in the Same Database Here, you would store the keys in the database itself with the data. This is not a perfect solution, as you now know that a DBA with enough time on their hands (or a hacker who has gotten into a DBA account) could possibly discover your keys and the data that was encrypted by them. What you need to do in a case like this, is to make it as hard as possible to put the keys together with the data. This is a hard thing to do because both are in the same database.

One approach is to never directly relate the key table to the data table. For example, you have a table with CUSTOMER_ID, CREDIT_CARD, and other data. The CUSTOMER_ID is immutable; it is the primary key (and we all know you never update a primary key). You might set up another table:

ID number primary key, DATA varchar2(255)

This is the table in which we will store our keys, one key per customer ID. We will provide a packaged function that will return the key if and only if the appropriate user in the appropriate environment is executing it (similar to the concept behind FGAC; you can only get to the data if you have set up the correct application context).

This package would provide two basic functions:

❑ Function 1: The addition of a new customer – In this case, the function would perform some operation on the customer ID to ‘scramble’ it (convert it into another string). This function would be deterministic so that given the same customer ID we would always get the same output. Some ideas for how to mangle this customer ID, or any string in fact, follow below. It would also generate a random key for this customer. Some ideas on how to generate that key follow. It would then use dynamic SQL to insert the row into the key table (which is not going to be named KEY_TABLE or anything obvious like that).

Appendix A

1142

❑ Function 2: The retrieval of a key for a customer – This function would take a customer ID, run it through the same deterministic function as above, and then using dynamic SQL, it would look up the key for this customer, and return it. It will only perform these functions if the current user is executing in the appropriate environment.

The reason for the use of dynamic SQL is that it may become obvious to people that this package is the one performing the key management. A user may be able to query ALL_DEPENDENCIES to find all of the tables this package statically references. By using dynamic SQL, there will be no correlation between this package, and the key table. We are not preventing a really smart person from deducing the key table here, just making it as hard as we possibly can.

As for how to scramble the customer ID (or any set of immutable data related to this row, the primary key is a good candidate as long as you never update it), we can use many algorithms. If I was using Oracle 8.1.7, I may send this piece of information concatenated with some constant value (commonly referred to as a ‘salt’) to the MD5 routines, to generate a 16 byte CHECKSUM. I would use this as my key. Using Oracle 8.1.6, I might do the same sort of operation, but use DBMS_UTILITY.GET_HASH_VALUEwith a very large hash table size. I could use an XOR algorithm after reversing the bytes in the CUSTOMER_ID. Any algorithm that would be hard to guess given the resulting output data would suffice.

By now, you should be saying, ‘Ahh but the DBA can just read the code out, see the algorithm, and figure this all out.’ Not if you wrap the code. Wrapping PL/SQL code is very straightforward (see the WRAP utility that is documented in the PL/SQL User’s Guide and Reference). This will take your source code and ‘obfuscate’ it. You will load the wrapped version of the code into the database. Now no one can read your code. There are no ‘reverse wrap’ tools available. Just make sure to maintain a good copy of your algorithm somewhere safe. Since there are no reverse wrap, tools you will not be able to resurrect your code from the database if you need to.

Now, to generate a key for this customer, we need a randomizer of sorts. There are many ways we could do this. We could use the same basic routines we used to obfuscate the CUSTOMER_ID. We could use a random number generator (such as DBMS_RANDOM or one we develop ourselves). The goal is to generate something that cannot be ‘guessed’, based on anything else.

Personally speaking, this would be my preferred choice, storing the keys in the database. If I let the client application manage them, there is the risk of the client application ‘losing’ the keys due to a media failure, or some other system catastrophe. If I use the next method, storing them in the file system, I stand the same sort of risk. Only if I store the keys in the database can I be ensured that the encrypted data can in fact be decrypted – my database is always in ‘sync’, and backup and recovery is assured.

Store the Keys in the File System with the Database You could also store the keys used to encrypt the data in files within the file system and, using a C external procedure, access them. I suggest using a C external procedure because the goal here is to prevent the DBA from ‘seeing’ them, and quite often the DBA has access to the Oracle software account. UTL_FILE, BFILES, and Java stored procedures doing I/O, run as the Oracle software account user. If the DBA can become the Oracle software owner, they can see the files. If they can see the files, they can see the keys. Using an external procedure written in C, the EXTPROC service can run under a wholly different account by running the listener for the EXTPROC service as another user. In this fashion, the Oracle account cannot ‘see’ the keys. Only via the EXTPROC listener can I gain access to them. This just adds a layer of assuredness to your solution. See Chapter 18 on C-Based External Procedures for more information regarding this approach.


1143

Summary We have spent quite a bit of time looking at the DBMS_OBFUSCATION_TOOLKIT package in this section. Here, we have seen how to effectively create a wrapper package that provides the functionality in the way we would like to (feel free to write your own wrappers if you don’t like my implementation). We’ve learned also how to use dynamic SQL to create packages that can be installed into databases with different capabilities (8.1.6 versus 8.1.7 encryption capabilities in this case). We investigated a cross-platform issue that accompanies the DMBS_OBFUSCATION_TOOLKIT package, that of byte ordering in the keys. We learned how to solve this issue, if it occurs, by rearranging the bytes in the key itself. An interesting enhancement to the CRYPT_PKG above would be to have it automatically detect if you were on a little or big endian system, and swap the bytes in the key on one of them for you, so you would not even be aware of these differences. This would be an excellent idea given that in version 8.1.7.1, the need to reverse the bytes goes away and you could just remove the code that does this, and be fully functional again.

Lastly, we looked at the very important area of key management. I spent a lot of time working on a nice wrapper package to make encryption and decryption look easy. The hard part is still left up to you – how to secure the keys. You have to consider the fact that if you believe your keys have been compromised, you would have to come up with a new set of keys, and need to decrypt/encrypt all your existing data to protect it. Planning ahead will avoid you from having to do such things.

Appendix A

1144

DBMS_OUTPUT

The DBMS_OUTPUT package is one that people frequently misunderstand. They misunderstand how it works, what it does, and it’s limits. In this section I will address these misunderstandings. I will also address some alternative implementations that give you DBMS_OUTPUT-like functionality, but without some of the limits found in the native package.

DBMS_OUTPUT is a simple package designed to give the appearance that PL/SQL has the ability to perform simple screen I/O operations. It is designed so that it appears PL/SQL can print Hello Worldon your screen for example. You’ve seen me use it many hundreds of times in this book. An example is:

ops$tkyte@DEV816> exec dbms_output.put_line( ‘Hello World’ ); Hello World


What you didn’t see is that I had to issue a SQL*PLUS (or SVRMGRL) command in order to make this work. We can turn this screen I/O on and off like this:

ops$tkyte@DEV816> set serveroutput off ops$tkyte@DEV816> exec dbms_output.put_line( ‘Hello World’ );


ops$tkyte@DEV816> set serveroutput on ops$tkyte@DEV816> exec dbms_output.put_line( ‘Hello World’ ); Hello World


DBMS_OUTPUT

1145

In reality, PL/SQL has no capability to perform screen I/O (that’s why I said it was designed to give PL/SQL the appearance of being able to do this). In fact, it is SQL*PLUS that is doing the screen I/O – it is impossible for PL/SQL to write to our terminal. PL/SQL is being executed in a totally separate process, typically running on a different machine elsewhere in the network. SQL*PLUS, SVRMGRL, and other tools however, can write to our screens quite easily. You will notice that if you use DBMS_OUTPUT in your Java or Pro*C programs (or any program) the DBMS_OUTPUT data goes into the ‘bit bucket’, and never gets displayed. This is because your application would be responsible for displaying the output.

How DBMS_OUTPUT Works DBMS_OUTPUT is a package with a few entry points. The ones you will use the most are:

❑ PUT – Puts a string, NUMBER, or DATE into the output buffer, without adding a newline.

❑ PUT_LINE – Puts a STRING, NUMBER, or DATE into the output buffer and adds a newline.

❑ NEW_LINE – Puts a newline into the output stream.

❑ ENABLE/DISABLE – Enables or disables the buffering of data in the package. Effectively turns DBMS_OUTPUT on and off procedurally.

These procedures write to an internal buffer; a PL/SQL table stored in the package body of DBMS_OUTPUT. The limit the total length of a line (the sum of all bytes put into the buffer by you, without calling either PUT_LINE or NEW_LINE to terminate that line) is set to 255 bytes. All of the output your procedure generates is buffered in this table, and will not be visible to you in SQL*PLUS until after your procedure completes execution. PL/SQL is not writing to a terminal anywhere, it is simply stuffing data into a PL/SQL table.

As your procedure makes calls to DBMS_OUTPUT.PUT_LINE, the DBMS_OUTPUT package stores this data into an array (PL/SQL table), and returns control to your procedure. It is not until you are done that you will see any output. Even then, you will only see output if the client you are using is aware of DBMS_OUTPUT, and goes out of its way to print it out. SQL*PLUS for example, will issue calls to DBMS_OUTPUT.GET_LINES to get some of the DBMS_OUTPUT buffer, and print it on your screen. If you run a stored procedure from your Java/JDBC application, and expect to see the DBMS_OUTPUT output appear with the rest of your System.out.println data, you will be disappointed. Unless the client application makes a conscious effect to retrieve and print the data, it is just going into the bit bucket. We will demonstrate how to do this from Java/JDBC later in this chapter.

This fact that the output is buffered until the procedure completes, is the number one point of confusion with regards to DBMS_OUTPUT. People see DBMS_OUTPUT and read about it, and then they try to use it to monitor a long running process. That is, they’ll stick DBMS_OUTPUT.PUT_LINE calls all over their code, and run the procedure in SQL*PLUS. They wait for the output to start coming to the screen, and are very disappointed when it does not (because it cannot). Without an understanding of how it is implemented, it is not clear why the data doesn’t start appearing. Once you understand that PL/SQL (and Java and C external routines) running in the database cannot perform screen I/O, and that DBMS_OUTPUT is really just buffering the data in a big array, it becomes clear. This is when you should go back to the section on DBMS_APPLICATION_INFO, and read about the long operations interface! DBMS_APPLICATION_INFO is the tool you want to use to monitor long running processes, not DBMS_OUTPUT.

Appendix A

1146

So, what is DBMS_OUTPUT useful for then? It is great for printing out simple reports and making utilities. See Chapter 23 on Invoker and Definer Rights for a PRINT_TABLE procedure that uses DBMS_OUTPUT to generate output like this:

SQL> exec print_table( ‘select * from all_users where username = user’ ); USERNAME : OPS$TKYTE USER_ID : 334 CREATED : 02-oct-2000 10:02:12 -----------------


It prints the data down the screen instead of wrapping it across. Great for printing that really wide row, which would consume lots of horizontal space, and wrap on your screen, making it pretty unreadable.

Now that we know that DBMS_OUTPUT works by putting data into a PL/SQL table, we can look further at the implementation. When we enable DBMS_OUTPUT, either by calling DBMS_OUTPUT.ENABLE, or by using SET SERVEROUTPUT ON, we are not only enabling the capture of the data, but also we are setting a maximum limit on how much data we will capture. By default, if I issue:

SQL> set serveroutput on

I have enabled 20,000 bytes of DBMS_OUTPUT buffer. If I exceed this, I will receive:

begin*ERROR at line 1: ORA-20000: ORU-10027: buffer overflow, limit of 20000 bytes ORA-06512: at “SYS.DBMS_OUTPUT”, line 106 ORA-06512: at “SYS.DBMS_OUTPUT”, line 65 ORA-06512: at line 3

I can increase this limit via a call to SET SERVEROUTPUT (or DBMS_OUTPUT.ENABLE):

SQL> set serveroutput on size 1000000

SQL> set serveroutput on size 1000001 SP2-0547: size option 1000001 out of range (2000 through 1000000)

As you can see from the error message however, the limit is 20,000 bytes through 1,000,000 bytes. The limit of the number of bytes you can put into the buffer is somewhat less than the amount you set, perhaps much less. DBMS_OUTPUT has a simple packing algorithm it uses to place the data into the PL/SQL table. It does not put the i’th row of your output into the i’th array element, rather it densely packs the array. Array element #1 might have your first five lines of output encoded into it. In order to do this (to encode many lines into one line), they necessarily introduce some overhead. This overhead, the data they use to remember where your data is, and how big it is, is included in the byte count limit. So, even if you SET SERVEROUTPUT ON SIZE 1000000, you will get somewhat less than one million bytes of output.

Can you figure out how many bytes you will get? Sometimes yes and sometimes no. If you have a fixed size output line, every line is the same length, then the answer is yes. We can compute the number of

DBMS_OUTPUT

1147

bytes you will get exactly. If your data is of varying width, then no, we cannot calculate the number of bytes you will be able to output before you actually output it. Below, I explain the algorithm Oracle uses to pack this data.

We know that Oracle stores the data in an array. The maximum total number of lines in this array is set based upon your SET SERVEROUTPUT ON SIZE setting. The DBMS_OUTPUT array will never have more than IDXLIMIT lines where IDXLIMIT is computed as:

idxlimit := trunc((xxxxxx+499) / 500);

So, if you SET SERVEROUTPUT ON SIZE 1000000, DBMS_OUTPUT will use 2,000 array elements at most. DBMS_OUTPUT will store at most 504 bytes of data in each array element, and typically less. DBMS_OUTPUT packs the data into a row in the array, in the following format:

their_buffer(1) = ‘<sp>NNNyour data here<sp>NNNyour data here...’; their_buffer(2) = ‘<sp>NNNyour data here<sp>NNNyour data here...’;

So, for each line of your output, there is a 4-byte overhead for a space, and a 3-digit number. Each line in the DBMS_OUTPUT buffer will not exceed 504 bytes, and DBMS_OUTPUT will not wrap your data from line to line. So, for example, if you use the maximum line length and always write 255 bytes per line, DBMS_OUTPUT will be able to pack one line per array element above. This is because (255+4) * 2 = 518, 518 is bigger than 504, and DBMS_OUTPUT will not split your line between two of its array elements. Two lines will simply not fit in one of DBMS_OUTPUT’s lines. Therefore, even though you asked for a buffer of 1,000,000 bytes, you will only get 510,000 – a little more then half of what you asked for. The 510,000 comes from the fact you are printing lines of 255 bytes, and they will allow for a maximum of 2,000 lines (remember IDXLIMIT from above); 255*2000 = 510,000. On the other hand, if you used a fixed line size of 248 bytes, they will get two lines for each of their lines, resulting in you being able to print out 248 * 2 * 2000 = 992,000 – a little more than 99 percent of what you asked for. In fact, this is the best you can hope for with DBMS_OUTPUT – 992,000 bytes of your data. It is impossible to get more printed out.

As I said previously, with a fixed size line, it is very easy to determine the number of lines you will be able to print. If you give me a number, say 79, 80, or 81 bytes per line, I can simply determine:

[email protected]> select trunc(504/(79+4)) * 79 * 2000 from dual;

TRUNC(504/(79+4))*79*2000------------------------- 948000


TRUNC(504/(80+4))*80*2000------------------------- 960000


TRUNC(504/(81+4))*81*2000------------------------- 810000

Appendix A

1148

As you can see, the amount of data we can output varies widely, depending on the size of our output line!

The trouble with varying length output is that the amount of output we can produce is unpredictable. It depends on how you do the output, and the mix of line sizes DBMS_OUTPUT receives. If you output the same lines, just in a different order, you may be able to print more or less lines. This is a direct result of the packing algorithm.

This is one of the most confusing aspects of DBMS_OUTPUT. You might run your procedure once and have it produce a report of 700,000 bytes successfully, and run it then tomorrow and have it fail with ORA-20000: ORU-10027: buffer overflow at 650,000 bytes of output. This is simply due to the way DBMS_OUTPUT packs the data in the buffer. Further on in this section, we will look at some alternatives to DBMS_OUTPUT that remove this ambiguity.

A reasonable question to ask is, ‘Why do they do this packing?’ The reason is that when DBMS_OUTPUTwas introduced in version 7.0, PL/SQL table memory allocation was very different. If you allocated a slot in a PL/SQL table, enough storage for the maximum array element size was allocated immediately. This means that since DBMS_OUTPUT uses a VARCHAR2(500), 500 bytes would be allocated for a DBMS_OUTPUT.PUT_LINE( ‘hello world’ ) – the same as for the output of a really big string. 2,000 lines of output would take 1,000,000 bytes of data, even if you printed out hello world 2,000 times, something that should actually take about 22 KB. So, this packing was implemented in order to prevent this over-allocation of memory in the PGA for the buffering array. In the latest releases of Oracle (8.0 and up) this is no longer the case. Array elements are dynamically sized and this packing isn’t technically necessary any longer. So, you might say this is a legacy side effect from code written in prior releases.

The last thing about how DBMS_OUTPUT works I would like to mention has to do with the trimming of leading blanks on output lines. It is a mistaken belief that this is a DBMS_OUTPUT ‘feature’. It is actually a SQL*PLUS ‘feature’ (although I know of many who disagree with the ‘feature’ tag on this one). To see what I mean, we can run a small test:

[email protected]> exec dbms_output.put_line( ‘ hello world’ ); hello world


When I call DBMS_OUTPUT with ‘ hello world’, the leading blanks are trimmed away. It is assumed that DBMS_OUTPUT is doing this but really it isn’t. It is SQL*PLUS doing the trimming. The simple solution to this is to use the extended syntax on the SET SERVEROUTPUT command. The full syntax is of that command is:

set serveroutput {ON|OFF} [SIZE n] [FORMAT {WRAPPED|WORD_WRAPPED|TRUNCATED}]

The formats have the following meanings:

❑ WRAPPED – SQL*PLUS wraps the server output within the line size specified by SETLINESIZE, beginning new lines when required.

❑ WORD_WRAPPED – Each line of server output is wrapped within the line size specified by SETLINESIZE. Lines are broken on word boundaries. SQL*PLUS left-justifies each line, skipping all leading whitespace. This is the default.

DBMS_OUTPUT

1149

❑ TRUNCATED – When enabled, each line of server output is truncated to the line size specified by SET LINESIZE.

It is easiest just to see the effect of each format in action, to understand what each does:

SQL>set linesize 20 SQL>set serveroutput on format wrapped SQL>exec dbms_output.put_line( ‘ Hello World !!!!!’ ); Hello World !!!!!


SQL>set serveroutput on format word_wrappedSQL>exec dbms_output.put_line( ‘ Hello World !!!!!’ ); Hello World !!!!!


SQL>set serveroutput on format truncated SQL>exec dbms_output.put_line( ‘ Hello World !!!!!’ ); Hello World


DBMS_OUTPUT and Other Environments By default, tools such as SQL*PLUS and SVRMGRL are DBMS_OUTPUT-aware. Most other environments are not. For example, your Java/JDBC program is definitely not DBMS_OUTPUT-aware. In this section, we’ll see how to make Java/JDBC DBMS_OUTPUT-aware. The same principles used below apply equally to any programming environment. The methods I use with Java can be easily applied to Pro*C, OCI, VB, or any number of these environments.

We’ll start with a small PL/SQL routine that generates some output data:

scott@TKYTE816> create or replace 2 procedure emp_report 3 as 4 begin 5 dbms_output.put_line 6 ( rpad( ‘Empno’, 7 ) || 7 rpad(‘Ename’,12) || 8 rpad(‘Job’,11) ); 9 10 dbms_output.put_line 11 ( rpad( ‘-’, 5, ‘-’ ) || 12 rpad(‘ -’,12,’-’) || 13 rpad(‘ -’,11,’-’) ); 14

Appendix A

1150

15 for x in ( select * from emp ) 16 loop 17 dbms_output.put_line 18 ( to_char( x.empno, ‘9999’ ) || ‘ ‘ || 19 rpad( x.ename, 12 ) || 20 rpad( x.job, 11 ) ); 21 end loop; 22 end; 23 /

Procedure created.

scott@TKYTE816> set serveroutput on format wrapped scott@TKYTE816> exec emp_reportEmpno Ename Job ----- ---------- --------- 7369 SMITH CLERK 7499 ALLEN SALESMAN ... 7934 MILLER CLERK


Now, we’ll set up a class to allow Java/JDBC to easily perform DBMS_OUTPUT for us:

import java.sql.*;

class DbmsOutput{/* * Our instance variables. It is always best to * use callable or prepared statements, and prepare (parse) * them once per program execution, rather then once per * execution in the program. The cost of reparsing is * very high. Also, make sure to use BIND VARIABLES! * * We use three statements in this class. One to enable * DBMS_OUTPUT, equivalent to SET SERVEROUTPUT on in SQL*PLUS, * another to disable it, like SET SERVEROUTPUT OFF. * The last is to ‘dump’ or display the results from DBMS_OUTPUT * using system.out. * */ private CallableStatement enable_stmt; private CallableStatement disable_stmt; private CallableStatement show_stmt;

/* * Our constructor simply prepares the three * statements we plan on executing. * * The statement we prepare for SHOW is a block of * code to return a string of DBMS_OUTPUT output. Normally, * you might bind to a PL/SQL table type, but the JDBC drivers

DBMS_OUTPUT

1151

* don’t support PL/SQL table types. Hence, we get the output * and concatenate it into a string. We will retrieve at least * one line of output, so we may exceed your MAXBYTES parameter * below. If you set MAXBYTES to 10, and the first line is 100 * bytes long, you will get the 100 bytes. MAXBYTES will stop us * from getting yet another line, but it will not chunk up a line. * */ public DbmsOutput( Connection conn ) throws SQLException { enable_stmt = conn.prepareCall( “begin dbms_output.enable(:1); end;” ); disable_stmt = conn.prepareCall( “begin dbms_output.disable; end;” );

show_stmt = conn.prepareCall( “declare “ + “ l_line varchar2(255); “ + “ l_done number; “ + “ l_buffer long; “ + “begin “ + “ loop “ + “ exit when length(l_buffer)+255 > :maxbytes OR l_done = 1; “ + “ dbms_output.get_line( l_line, l_done ); “ + “ l_buffer := l_buffer || l_line || chr(10); “ + “ end loop; “ + “ :done := l_done; “ + “ :buffer := l_buffer; “ + “end;” ); }

/* * ENABLE simply sets your size and executes * the DBMS_OUTPUT.ENABLE call * */ public void enable( int size ) throws SQLException { enable_stmt.setInt( 1, size ); enable_stmt.executeUpdate(); }

/* * DISABLE only has to execute the DBMS_OUTPUT.DISABLE call */ public void disable() throws SQLException { disable_stmt.executeUpdate();}

/* * SHOW does most of the work. It loops over * all of the DBMS_OUTPUT data, fetching it, in this * case, 32,000 bytes at a time (give or take 255 bytes). * It will print this output on STDOUT by default (just * reset what System.out is to change or redirect this * output). */

Appendix A

1152

public void show() throws SQLException {int done = 0;

show_stmt.registerOutParameter( 2, java.sql.Types.INTEGER ); show_stmt.registerOutParameter( 3, java.sql.Types.VARCHAR );

for(;;) { show_stmt.setInt( 1, 32000 ); show_stmt.executeUpdate(); System.out.print( show_stmt.getString(3) ); if ( (done = show_stmt.getInt(2)) == 1 ) break; } }

/* * CLOSE closes the callable statements associated with * the DbmsOutput class. Call this if you allocate a DbmsOutput * statement on the stack and it is going to go out of scope, * just as you would with any callable statement, resultset, * and so on. */ public void close() throws SQLException { enable_stmt.close(); disable_stmt.close(); show_stmt.close(); }}

In order to demonstrate its use, I’ve set up the following small Java/JDBC test program. Here dbserver is the name of the database server and ora8i is the service name of the instance:

import java.sql.*;

class test {

public static void main (String args []) throws SQLException{ DriverManager.registerDriver (new oracle.jdbc.driver.OracleDriver());

Connection conn = DriverManager.getConnection (“jdbc:oracle:thin:@dbserver:1521:ora8i”, “scott”, “tiger”); conn.setAutoCommit (false);

Statement stmt = conn.createStatement();

DbmsOutput dbmsOutput = new DbmsOutput( conn );

dbmsOutput.enable( 1000000 );

DBMS_OUTPUT

1153

stmt.execute ( “begin emp_report; end;” ); stmt.close();

dbmsOutput.show();

dbmsOutput.close(); conn.close(); }}

Now we will test it, by first compiling it, and then running it:

$ javac test.java

$ java test Empno Ename Job----- ---------- --------- 7369 SMITH CLERK 7499 ALLEN SALESMAN 7521 WARD SALESMAN ...

So, this shows how to teach Java to do DBMS_OUTPUT for us. Just as SQL*PLUS does, you’ll have to call DbmsOutput.show() after executing any statement that might cause some output to be displayed. After we execute an INSERT, UPDATE, DELETE, or stored procedure call, SQL*PLUS is calling DBMS_OUTPUT.GET_LINES to get the output. Your Java (or C, or VB) application would call SHOW to display the results.

Getting Around the Limits DBMS_OUTPUT has two major limitations that I’ve found:

❑ The length of a ‘line’ is limited to 255 bytes. You must inject a new line at least every 255 bytes.

❑ The total output you can produce is limited to between 200,000 bytes (if you output 1 byte per line) and 992,000 bytes (if you output 248 bytes per line). This is sufficient for many operations, but a show stopper for others, especially since the amount of output you can generate is a function of the lengths of the strings, and the order in which you print them.

So, what can we do? I’ll suggest three alternatives to get around these various limits. The next two sections demonstrate these alternatives.

Using A Small Wrapper Function or Another Package Sometimes the 255 byte line limit is just a nuisance. You want to print some debug statements, and the thing you are printing is 500 characters long. You just want to print it, and the format of it is not as relevant as just being able to see it. In this case, we can write a small wrapper routine. I have one permanently installed in all of my database instances, in part to get around the 255 bytes per line, and in part because DBMS_OUTPUT.PUT_LINE is 20 characters long, which is a lot of typing. I use a procedure P frequently. P is simply:

Appendix A

1154

procedure p( p_string in varchar2 ) is l_string long default p_string;begin loop exit when l_string is null; dbms_output.put_line( substr( l_string, 1, 248) ); l_string := substr( l_string, 251 ); end loop; end;

It does not word-wrap output, it does nothing fancy. It simply takes a string up to 32 KB in size, and prints it out. It will break my large strings into many strings of 248 bytes each (248 being the ‘best’ number we calculated above, giving us the maximum output), and output them. It will change the data (so it is not suitable for increasing the line width of a routine that is creating a flat file), and will cause my one line of data to be printed on perhaps many lines.

All it does is solve a simple problem. It removes the error message:

[email protected]> exec dbms_output.put_line( rpad(‘*’,256,’*’) ) BEGIN dbms_output.put_line( rpad(‘*’,256,’*’) ); END;

*ERROR at line 1: ORA-20000: ORU-10028: line length overflow, limit of 255 bytes per line ORA-06512: at “SYS.DBMS_OUTPUT”, line 99 ORA-06512: at “SYS.DBMS_OUTPUT”, line 65 ORA-06512: at line 1

from occurring when I am just printing some debug, or a report.

A more robust method of getting around this limit, especially useful if you are creating a flat file data dump, is to not use DBMS_OUTPUT at all, but rather to use UTL_FILE and write directly to a file. UTL_FILE has a 32 KB limit per line of output, and does not have a byte limit on the size of a file. Using UTL_FILE, you can only create a file on the server so it is not appropriate if you were using SQL*PLUS on a network-connected client, and spooling to a local file on the client. If your goal was to create a flat file for data loading, and creating the file on the server is OK, UTL_FILE would be the correct approach.

So, this covers two of the three alternatives, now for the last one.

Creating DBMS_OUTPUT Functionality This is a general-purpose solution that works well in all environments. What we will do here is reinvent the wheel, only we’ll invent a ‘better’ wheel. We will create a DBMS_OUTPUT-like package that:

❑ Has a 4,000 byte per line limit (this is a SQL limit unfortunately, not a PL/SQL limit).

❑ No limit on the number of output lines.

❑ Can be spooled on the client, like DBMS_OUTPUT.

❑ SQL*PLUS will not blank-trim the front of the string in any mode.

❑ Can be fetched into a resultset on a client using a cursor (the output will be available via a query).

DBMS_OUTPUT

1155

We’ll begin by creating a SQL type. This type will be our DBMS_OUTPUT buffer. Since it is a SQL type, we can SELECT * from it easily. Since virtually everything can do a SELECT *, any tool should be able to display our output easily.

[email protected]> create or replace type my_dbms_output_type 2 as table of varchar2(4000) 3 /

Type created.

Now we move on to the specification for our DBMS_OUTPUT-like package. This package is set up much like the real DBMS_OUTPUT. It does not have the routines GET_LINE and GET_LINES. These will not be needed, given our implementation. The routines PUT, PUT_LINE, and NEW_LINE work just like their counterparts in DBMS_OUTPUT. The functions GET, FLUSH, and GET_AND_FLUSH are new – they have no counterpart in DBMS_OUTPUT. These routines will be used to retrieve the output once the stored procedure has executed. The function GET will simply return the buffered data, but it will not ‘erase’ it. You can call GET over and over to retrieve the same buffer (DBMS_OUTPUT always flushes the buffer). The function FLUSH allows you to reset the buffer, in other words empty it out. The function GET_AND_FLUSH, as you might guess, returns the buffer, and clears it out – the next calls to this package will function against an empty buffer:

tkyte@TKYTE816> create or replace package my_dbms_output 2 as

3 procedure enable;

4 procedure disable;

5

6 procedure put( s in varchar2 ); 7 procedure put_line( s in varchar2 ); 8 procedure new_line; 9 10 function get return my_dbms_output_type; 11 procedure flush; 12 function get_and_flush return my_dbms_output_type; 13 end;

14 /

Package created.

We will be using some of the methods we discussed Chapter 20 on Using Object Relational Features,specifically the capability of being able to SELECT * from PLSQL_FUNCTION, which is how our DBMS_OUTPUT package will work. The functions you are most interested in are the ENABLE, DISABLE,PUT, PUT_LINE, and NEW_LINE routines. These work more or less like their DBMS_OUTPUTcounterparts, the major difference being that ENABLE takes no parameters, and that MY_DBMS_OUTPUTis enabled by default (whereas DBMS_OUTPUT is disabled by default). You are limited by the amount of RAM you can allocate on your system (so beware!). Next, we implement the package body. The implementation of this package is very straightforward. We have a package global variable that is our output buffer. We add lines of text to it and extend this variable when necessary. To flush it, we assign an empty table to it. As it is so straightforward, it is presented here without further comment:

Appendix A

1156

tkyte@TKYTE816> create or replace package body my_dbms_output 2 as 3 4 g_data my_dbms_output_type := my_dbms_output_type(); 5 g_enabled boolean default TRUE; 6 7 procedure enable 8 is 9 begin 10 g_enabled := TRUE; 11 end; 12 13 procedure disable 14 is 15 begin 16 g_enabled := FALSE; 17 end; 18 19 procedure put( s in varchar2 ) 20 is 21 begin 22 if ( NOT g_enabled ) then return; end if; 23 if ( g_data.count <> 0 ) then 24 g_data(g_data.last) := g_data(g_data.last) || s; 25 else 26 g_data.extend; 27 g_data(1) := s; 28 end if; 29 end; 30 31 procedure put_line( s in varchar2 ) 32 is 33 begin 34 if ( NOT g_enabled ) then return; end if; 35 put( s ); 36 g_data.extend; 37 end; 38 39 procedure new_line 40 is 41 begin 42 if ( NOT g_enabled ) then return; end if; 43 put( null ); 44 g_data.extend; 45 end; 46 47 48 procedure flush 49 is 50 l_empty my_dbms_output_type := my_dbms_output_type(); 51 begin 52 g_data := l_empty; 53 end; 54 55 function get return my_dbms_output_type 56 is

DBMS_OUTPUT

1157

57 begin 58 return g_data; 59 end;

60 61 function get_and_flush return my_dbms_output_type 62 is 63 l_data my_dbms_output_type := g_data; 64 l_empty my_dbms_output_type := my_dbms_output_type(); 65 begin 66 g_data := l_empty; 67 return l_data; 68 end;

69 end;

70 /


Now, in order to make this package useful, we need some method of getting at the buffer easily. You can call MY_DBMS_OUTPUT.GET or GET_AND_FLUSH, and retrieve the object type yourself, or you can use one of the two views below. The first view, MY_DBMS_OUTPUT_PEEK, provides a SQL interface to the GET routine. It allows you to query the output buffer over and over again, in effect, allowing you to ‘peek’ into the buffer without resetting it. The second view, MY_DBMS_OUTPUT_VIEW, allows you to query the buffer once – any subsequent calls to PUT, PUT_LINE, NEW_LINE, GET, or GET_AND_FLUSHwill work on an empty output buffer. A SELECT * FROM MY_DBMS_OUTPUT_VIEW is similar to calling DBMS_OUTPUT.GET_LINES. It resets everything:

tkyte@TKYTE816> create or replace 2 view my_dbms_output_peek ( text ) 3 as

4 select * 5 from TABLE ( cast( my_dbms_output.get() 6 as my_dbms_output_type ) ) 7 /

View created.

tkyte@TKYTE816> create or replace 2 view my_dbms_output_view ( text ) 3 as

4 select * 5 from TABLE ( cast( my_dbms_output.get_and_flush() 6 as my_dbms_output_type ) ) 7 /

View created.

Now we are ready to demonstrate how this works. We will run a procedure to generate some data into the buffer, and then see how to display and interact with it:

Appendix A

1158

tkyte@TKYTE816> begin 2 my_dbms_output.put_line( ‘hello’ ); 3 my_dbms_output.put( ‘Hey ‘ ); 4 my_dbms_output.put( ‘there ‘ ); 5 my_dbms_output.new_line; 6 7 for i in 1 .. 20 8 loop 9 my_dbms_output.put_line( rpad( ‘ ‘, i, ‘ ‘ ) || i ); 10 end loop; 11 end; 12 /


tkyte@TKYTE816> select * 2 from my_dbms_output_peek 3 /

TEXT------------------------------------------------------------------helloHey there 1 2 ... 19 20

23 rows selected.

The interesting thing to note here is that SQL*PLUS, not being aware of MY_DBMS_OUTPUT, will not display the results automatically. You need to help it along, and execute a query to dump the results.

Since we are just using SQL to access the output, it should be easy for you to rewrite your own DbmsOutput Java/JDBC class. It will be a simple ResultSet object, nothing more. As a last comment on this snippet of code, the output buffer is still there waiting for us:

tkyte@TKYTE816> select * 2 from my_dbms_output_peek 3 /

TEXT------------------------------------helloHey there 1 2 ... 19 20

23 rows selected.

DBMS_OUTPUT

1159

and not only is it waiting for us, we also can WHERE on it, sort it, join it, and so on (like any table could be):

tkyte@TKYTE816> select * 2 from my_dbms_output_peek 3 where text like ‘%1%’ 4 /

TEXT-------------------------------------- 1 10 11 ... 18 19

11 rows selected.

Now, if this is not the desired behavior (to be able to query and re-query this data) we would SELECTfrom MY_DBMS_OUTPUT_VIEW instead:

tkyte@TKYTE816> select * 2 from my_dbms_output_view 3 /

TEXT---------------------------------helloHey there 1 ... 19 20

23 rows selected.

tkyte@TKYTE816> select * 2 from my_dbms_output_view 3 /

no rows selected

In this fashion, we get to see the data only once.

This new implementation of DBMS_OUTPUT raises the 255 bytes per line limit to 4,000 bytes per line, and effectively removes the size limitation on the number of total output bytes (you are still limited by available RAM on your server though). It introduces some new functionality (you can query your output, sort it, and so on) as well. It removes the SQL*PLUS default feature of blank-trimming. Lastly, unlike UTL_FILE, the results of MY_DBMS_OUTPUT can be spooled to a client file in the same way DBMS_OUTPUT output could, making it a viable replacement for client-side functions.

Appendix A

1160

You might ask why I used an object type instead of a temporary table in this implementation. The answer is one of code, and overhead. The amount of code to manage the temporary table, to have at least an additional column to remember the proper order of the data, as compared to this simple implementation, is large. Also, a temporary table incurs some amount of I/O activities and overhead. Lastly, it would be hard to implement the ‘flushing view’ effect I have above, whereby we empty the output buffer automatically simply by selecting from it. In short, using the object type lead to a lighter weight implementation. If I planned on using this to generate tens of MB of output, I might very well reconsider my choice of buffering mechanisms, and use a temporary table. For moderate amounts of data, this implementation works well.

Summary In this section, we have covered how the DBMS_OUTPUT package is actually implemented. Now that you know how it works, you will not become a victim of the side effects of this. You are now able to anticipate that you won’t get the buffer size you asked for, and that the size of this output buffer will seem arbitrary at times. You’ll be aware that it is not possible to produce an output line that exceeds 255 bytes without a newline. You know that you cannot see DBMS_OUTPUT output until after the procedure or statement completes execution, and even then, only if the environment you are using to query the database supports DBMS_OUTPUT.

In addition to gaining an understanding of how DBMS_OUTPUT works, we have also seen how to solve many of the major limitations, typically by using other features to accomplish our goals. Solutions such as UTL_FILE to produce flat files and simple functions like P to not only save on typing, but also to print larger lines. In the extreme case, we looked implementing your own equivalent functionality that does not suffer from some of the limits.

DBMS_OUTPUT is a good example of how something seemingly trivial can, in fact, be a very complex piece of software with unintended side effects. When you read through the DBMS_OUTPUTdocumentation in Oracle’s Supplied PL/SQL Packages Reference guide, it sounds so simple and straightforward. Then issues like the total number of output bytes you can generate, and so on, crop up. Knowledge of how the package works helps us avoid these issues, either by just being aware that they are there, or by using alternative methods to implement our applications.

DBMS_PROFILER

1161

DBMS_PROFILER

The profiler is a long awaited (by me anyway) feature. It provides us with a source code profiler for our PL/SQL applications. In the past, you would tune your PL/SQL applications using SQL_TRACE and TKPROF. This would help you identify and tune your long running SQL, but trying to discover where the bottlenecks in 5,000 lines of PL/SQL code are (which you might not have even written yourself) was pretty near impossible. You typically ended up instrumenting the code with lots of calls to DBMS_UTILITY.GET_TIME to measure elapsed time, in an attempt to find out what was slow.

Well, you no longer have to do this – we have the DBMS_PROFILER package. I am going to demonstrate how I use it. I myself use little of its total functionality – I just use it to find the really bad areas, and go straight there. I use it in a very simplistic fashion. It is set up and designed to do much more than presented here, however.

The statistics gathering takes place in database tables. These tables are set up to hold the statistics for many different runs of the code. This is OK for some people, but I like to just keep the last run or two, in there only. Any more than that and it gets too confusing. Sometimes, too much information is just that – too much information.

Your DBA may have to install the profiler in your database. The procedure for installing this package is simple:

❑ cd [ORACLE_HOME]/rdbms/admin.

❑ using SVRMGRL you would connect as SYS or INTERNAL.

❑ Run profload.sql.

Appendix A

1162

In order to actually use the profiler after that, you will need to have the profiling tables installed. You can install these once per database, but I recommend that developers have their own copy. Fortunately, the DBMS_PROFILER package is built with invoker rights and unqualified table names, so that we can install the tables in each schema, and the profiler package will use them correctly. The reason you each want your own tables, is so that you only see the results of your profiling runs, not those of your co-workers. In order to get a copy of the profiling tables in your schema, you would run [ORACLE_HOME]\rdbms\admin\proftab.sql in SQL*PLUS. After you run proftab.sql, you'll need to run profrep.sql as well. This script creates views and packages to operate on the profiler tables in order to generate reports. This script is found in [ORACLE_HOME]\plsql\demo\profrep.sql. You should run this in your schema as well, after creating the tables.

I like to keep a small script around to reset these tables, and clear them out every so often. After I've done a run or two and have analyzed the results, I run this script. I have the following in a script I call profreset.sql:

-- uses deletes because of foreign key constraints delete from plsql_profiler_data;delete from plsql_profiler_units;delete from plsql_profiler_runs;

Now we are ready to start profiling. I'm going to demonstrate the use of this package by running two different implementations of a factorial algorithm. One is recursive, and the other is iterative. We'll use the profiler to see which one is faster, and what components of the code are 'slow' in each implementation. The test driver for this is simply:

tkyte@TKYTE816> @profreset tkyte@TKYTE816> create or replace 2 function fact_recursive( n int ) return number 3 as 4 begin 5 if ( n = 1 ) 6 then 7 return 1; 8 else 9 return n * fact_recursive(n-1); 10 end if; 11 end; 12 /

Function created.

tkyte@TKYTE816> create or replace 2 function fact_iterative( n int ) return number 3 as 4 l_result number default 1; 5 begin 6 for i in 2 .. n 7 loop 8 l_result := l_result * i; 9 end loop; 10 return l_result; 11 end; 12 /

DBMS_PROFILER

1163

Function created.

tkyte@TKYTE816> set serveroutput on

tkyte@TKYTE816> exec dbms_profiler.start_profiler( 'factorial recursive' )


tkyte@TKYTE816> begin 2 for i in 1 .. 50 loop 3 dbms_output.put_line( fact_recursive(50) ); 4 end loop; 5 end; 6 / 30414093201713378043612608166064768844300000000000000000000000000...30414093201713378043612608166064768844300000000000000000000000000


tkyte@TKYTE816> exec dbms_profiler.stop_profiler


tkyte@TKYTE816> exec dbms_profiler.start_profiler( 'factorial iterative' )


tkyte@TKYTE816> begin 2 for i in 1 .. 50 loop 3 dbms_output.put_line( fact_iterative(50) ); 4 end loop; 5 end; 6 / 30414093201713378043612608166064768844300000000000000000000000000...30414093201713378043612608166064768844300000000000000000000000000


tkyte@TKYTE816> exec dbms_profiler.stop_profiler


In order to collect statistics for a profiler run, we must call START_PROFILER. We name each run with some meaningful name, and then start running the code. I ran the factorial routines 50 times each before ending the statistics collection for a given run. Now, we are ready to analyze the results.

In [ORACLE_HOME]/plsql/demo there is a script profsum.sql. Don't run it– some of the queries in that script can take a considerable amount of time to execute (could take hours), and the amount of data it produces is very large. Below is the modified profsum.sql I use myself. It provides much of the same information, but the queries execute quickly, and many of the really detailed reports are cut out. Also, some of the queries would include the timings for the STOP_PROFILER call in it, and others would not, skewing the observations from query to query. I've adjusted all of the queries to not include the timings of the profiler package itself.

Appendix A

1164

My profsum.sql is the following, and this of course is available for download at http://www.apress.com:

set echo off set linesize 5000 set trimspool on set serveroutput on set termout off

column owner format a11 column unit_name format a14 column text format a21 word_wrappedcolumn runid format 9999 column secs format 999.99 column hsecs format 999.99 column grand_total format 9999.99 column run_comment format a11 word_wrapped column line# format 99999 column pct format 999.9 column unit_owner format a11

spool profsum.out

/* Clean out rollup results, and recreate. */ update plsql_profiler_units set total_time = 0;

execute prof_report_utilities.rollup_all_runs;

prompt = prompt = prompt ==================== prompt Total timeselect grand_total/1000000000 as grand_total from plsql_profiler_grand_total;

prompt = prompt = prompt ==================== prompt Total time spent on each runselect runid, substr(run_comment,1, 30) as run_comment, run_total_time/1000000000 as secs from (select a.runid, sum(a.total_time) run_total_time, b.run_comment from plsql_profiler_units a, plsql_profiler_runs b where a.runid = b.runid group by a.runid, b.run_comment ) where run_total_time > 0 order by runid asc;

prompt = prompt = prompt ==================== prompt Percentage of time in each module, for each run separately

DBMS_PROFILER

1165

select p1.runid, substr(p2.run_comment, 1, 20) as run_comment, p1.unit_owner, decode(p1.unit_name, '', '<anonymous>', substr(p1.unit_name,1, 20)) as unit_name, p1.total_time/1000000000 as secs, TO_CHAR(100*p1.total_time/p2.run_total_time, '999.9') as percentage from plsql_profiler_units p1, (select a.runid, sum(a.total_time) run_total_time, b.run_comment from plsql_profiler_units a, plsql_profiler_runs b where a.runid = b.runid group by a.runid, b.run_comment ) p2 where p1.runid=p2.runid and p1.total_time > 0 and p2.run_total_time > 0 and (p1.total_time/p2.run_total_time) >= .01 order by p1.runid asc, p1.total_time desc;

column secs form 9.99 prompt = prompt = prompt ==================== prompt Percentage of time in each module, summarized across runsselect p1.unit_owner, decode(p1.unit_name, '', '<anonymous>', substr(p1.unit_name,1, 25)) as unit_name, p1.total_time/1000000000 as secs, TO_CHAR(100*p1.total_time/p2.grand_total, '99999.99') as percentage from plsql_profiler_units_cross_run p1, plsql_profiler_grand_total p2 order by p1.total_time DESC;

prompt = prompt = prompt ==================== prompt Lines taking more than 1% of the total time, each run separateselect p1.runid as runid, p1.total_time/10000000 as Hsecs, p1.total_time/p4.grand_total*100 as pct, substr(p2.unit_owner, 1, 20) as owner, decode(p2.unit_name, '', '<anonymous>', substr(p2.unit_name,1, 20)) as unit_name, p1.line#, ( select p3.text from all_source p3 where p3.owner = p2.unit_owner and p3.line = p1.line# and p3.name=p2.unit_name and p3.type not in ( 'PACKAGE', 'TYPE' )) text from plsql_profiler_data p1, plsql_profiler_units p2, plsql_profiler_grand_total p4 where (p1.total_time >= p4.grand_total/100) AND p1.runID = p2.runid and p2.unit_number=p1.unit_number order by p1.total_time desc; prompt = prompt = prompt ==================== prompt Most popular lines (more than 1%), summarize across all runs

Appendix A

1166

select p1.total_time/10000000 as hsecs, p1.total_time/p4.grand_total*100 as pct, substr(p1.unit_owner, 1, 20) as unit_owner, decode(p1.unit_name, '', '<anonymous>', substr(p1.unit_name,1, 20)) as unit_name, p1.line#, ( select p3.text from all_source p3 where (p3.line = p1.line#) and (p3.owner = p1.unit_owner) AND (p3.name = p1.unit_name) and (p3.type not in ( 'PACKAGE', 'TYPE' ) ) ) text from plsql_profiler_lines_cross_run p1, plsql_profiler_grand_total p4 where (p1.total_time >= p4.grand_total/100) order by p1.total_time desc;

execute prof_report_utilities.rollup_all_runs;

prompt = prompt = prompt ==================== prompt Number of lines actually executed in different units (by unit_name)

select p1.unit_owner, p1.unit_name, count( decode( p1.total_occur, 0, null, 0)) as lines_executed , count(p1.line#) as lines_present, count( decode( p1.total_occur, 0, null, 0))/count(p1.line#) *100 as pct from plsql_profiler_lines_cross_run p1 where (p1.unit_type in ( 'PACKAGE BODY', 'TYPE BODY', 'PROCEDURE', 'FUNCTION' ) ) group by p1.unit_owner, p1.unit_name;

prompt = prompt = prompt ==================== prompt Number of lines actually executed for all unitsselect count(p1.line#) as lines_executed from plsql_profiler_lines_cross_run p1 where (p1.unit_type in ( 'PACKAGE BODY', 'TYPE BODY', 'PROCEDURE', 'FUNCTION' ) ) AND p1.total_occur > 0;

prompt = prompt = prompt ==================== prompt Total number of lines in all unitsselect count(p1.line#) as lines_present from plsql_profiler_lines_cross_run p1 where (p1.unit_type in ( 'PACKAGE BODY', 'TYPE BODY', 'PROCEDURE', 'FUNCTION' ) );

spool off set termout on edit profsum.out set linesize 131

DBMS_PROFILER

1167

I have gone out of my way to make that report fit into an 80-column screen, you could be more generous with some of the column formats if you don't use Telnet frequently.

Now, let's look at the output of our factorial run, the results of running the above profsum.sql script:

Total time

GRAND_TOTAL----------- 5.57

This tells us the grand total of our run times across both runs was 5.57 seconds. Next, we'll see a breakdown by run:

Total time spent on each run

RUNID RUN_COMMENT SECS ----- ----------- ------- 17 factorial 3.26 recursive

18 factorial 2.31 iterative

This shows us already that the recursive routine is not nearly as efficient as the iterative version, it took almost 50 percent longer to execute. Next, we will look at the amount of time spent in each module (package or procedure) in both runs, and the raw percentage of time by run:

Percentage of time in each module, for each run separately

RUNID RUN_COMMENT UNIT_OWNER UNIT_NAME SECS PERCEN ----- ----------- ----------- -------------- ------- ------ 17 factorial TKYTE FACT_RECURSIVE 1.87 57.5 recursive

17 factorial SYS DBMS_OUTPUT 1.20 36.9 recursive

17 factorial <anonymous> <anonymous> .08 2.5 recursive

17 factorial <anonymous> <anonymous> .06 1.9 recursive

18 factorial SYS DBMS_OUTPUT 1.24 53.6 iterative

18 factorial TKYTE FACT_ITERATIVE .89 38.5 iterative

18 factorial <anonymous> <anonymous> .08 3.4 iterative

18 factorial <anonymous> <anonymous> .06 2.7 iterative

8 rows selected.

Appendix A

1168

In this example, we see that in the recursive implementation, 57 percent of our run-time is spent in our routine, 37 percent in DBMS_OUTPUT, and the rest in miscellaneous routines. In our second run, the percentages are quite different. Our code is only 38 percent of the total run-time, and that it 38 percent of a smaller number! This already shows that the second implementation is superior to the first. More telling is the SECS column. Here, we can see that the recursive routine took 1.87 seconds, whereas the iterative routine took only .89. If we ignore DBMS_OUTPUT for a moment, we see that the iterative routine is two times faster than the recursive implementation.

It should be noted that you might not get exactly the same percentages (or even close) on your system. If you do not have SERVEROUTPUT ON in SQL*PLUS for example, DBMS_OUTPUT might not even show up on your system. If you run on a slower or faster machine, the numbers will be very different. For example, when I ran this on my Sparc Solaris machine, the GRAND_TOTAL time was about 1.0 seconds, and the percentages spent in each section of code were slightly different. Overall, the end result was much the same, percentage-wise.

Now we can look at the time spent in each module summarized across the runs. This will tell us what piece of code we spend most of our time in:

Percentage of time in each module, summarized across runs

UNIT_OWNER UNIT_NAME SECS PERCENTAG ----------- -------------- ----- --------- SYS DBMS_OUTPUT 2.44 43.82 TKYTE FACT_RECURSIVE 1.87 33.61 TKYTE FACT_ITERATIVE .89 16.00 <anonymous> <anonymous> .33 5.88 SYS DBMS_PROFILER .04 .69

Here, is it obvious we could cut our run-time almost in half by removing the single call to DBMS_OUTPUT. In fact, if you simply SET SERVEROUTPUT OFF, effectively disabling DBMS_OUTPUT, and rerun the test, you should find that it drops down to 3 percent or less of the total run-time. Currently however, it is taking the largest amount of time. What is more interesting than this, is that 33 percent of the total time is in the recursive routine and only 16 percent in the iterative – the iterative routine is much faster.

Now, let's look at some more details:

Lines taking more than 1% of the total time, each run separate

RUNID HSECS PCT OWNER UNIT_NAME LINE TEXT ----- ------- ------ ----- -------------- ---- --------------------- 17 142.47 25.6 TKYTE FACT_RECURSIVE 8 return n*fact_recursive(n-1); 18 68.00 12.2 TKYTE FACT_ITERATIVE 7 l_result := l_result * i; 17 43.29 7.8 TKYTE FACT_RECURSIVE 4 if ( n = 1 ) 17 19.58 3.5 SYS DBMS_OUTPUT 116 a3 a0 51 a5 1c 6e 81 b0 18 19.29 3.5 TKYTE FACT_ITERATIVE 5 for i in 2 .. n 18 17.66 3.2 SYS DBMS_OUTPUT 116 a3 a0 51 a5 1c 6e 81 b0 17 14.76 2.7 SYS DBMS_OUTPUT 118 1c 51 81 b0 a3 a0 1c 51 18 14.49 2.6 SYS DBMS_OUTPUT 118 1c 51 81 b0 a3 a0 1c 51 18 13.41 2.4 SYS DBMS_OUTPUT 142 :2 a0 a5 b b4 2e d b7 19 17 13.22 2.4 SYS DBMS_OUTPUT 142 :2 a0 a5 b b4 2e d b7 19 18 10.62 1.9 SYS DBMS_OUTPUT 166 6e b4 2e d :2 a0 7e 51 b4

DBMS_PROFILER

1169

17 10.46 1.9 SYS DBMS_OUTPUT 166 6e b4 2e d :2 a0 7e 51 b4 17 8.11 1.5 SYS DBMS_OUTPUT 72 1TO_CHAR: 18 8.09 1.5 SYS DBMS_OUTPUT 144 8f a0 b0 3d b4 55 6a :3 a0 18 8.02 1.4 SYS DBMS_OUTPUT 72 1TO_CHAR: 17 8.00 1.4 SYS DBMS_OUTPUT 144 8f a0 b0 3d b4 55 6a :3 a0 17 7.52 1.4 <ano> <anonymous> 3 18 7.22 1.3 <ano> <anonymous> 3 18 6.65 1.2 SYS DBMS_OUTPUT 141 a0 b0 3d b4 55 6a :3 a0 7e 18 6.21 1.1 <ano> <anonymous> 1 17 6.13 1.1 <ano> <anonymous> 1 18 5.77 1.0 SYS DBMS_OUTPUT 81 1ORU-10028:: line length

22 rows selected.

Here, I am printing out the run-time in hundreds of seconds instead of seconds, and showing the percentages as well. No surprises here – we would expect that line 8 in the recursive routine and line 7 in the iterative routine would be the big ones. Here, we can see that they are. This part of the report gives you specific lines in the code to zero in on and fix. Notice the strange looking lines of code from DBMS_OUTPUT. This is what wrapped PL/SQL looks like in the database. It is just a bytecode representation of the actual source, designed to obscure it from prying eyes like yours and mine.

Now, the next report is similar to the one above, but it aggregates results across runs, whereas the numbers above show percentages within a run:

Most popular lines (more than 1%), summarize across all runs

HSECS PCT OWNER UNIT_NAME LINE TEXT ------- ------ ----- -------------- ---- --------------------- 142.47 25.6 TKYTE FACT_RECURSIVE 8 return n * fact_recursive(n-1); 68.00 12.2 TKYTE FACT_ITERATIVE 7 l_result := l_result * i; 43.29 7.8 TKYTE FACT_RECURSIVE 4 if ( n = 1 ) 37.24 6.7 SYS DBMS_OUTPUT 116 a3 a0 51 a5 1c 6e 81 b0 29.26 5.3 SYS DBMS_OUTPUT 118 1c 51 81 b0 a3 a0 1c 51 26.63 4.8 SYS DBMS_OUTPUT 142 :2 a0 a5 b b4 2e d b7 19 21.08 3.8 SYS DBMS_OUTPUT 166 6e b4 2e d :2 a0 7e 51 b4 19.29 3.5 TKYTE FACT_ITERATIVE 5 for i in 2 .. n 16.88 3.0 <ano> <anonymous> 1 16.13 2.9 SYS DBMS_OUTPUT 72 1TO_CHAR: 16.09 2.9 SYS DBMS_OUTPUT 144 8f a0 b0 3d b4 55 6a :3 a0 14.74 2.6 <ano> <anonymous> 3 11.28 2.0 SYS DBMS_OUTPUT 81 1ORU-10028:: line length overflow, 10.17 1.8 SYS DBMS_OUTPUT 147 4f 9a 8f a0 b0 3d b4 55 9.52 1.7 SYS DBMS_OUTPUT 73 1DATE: 8.54 1.5 SYS DBMS_OUTPUT 117 a3 a0 1c 51 81 b0 a3 a0 7.36 1.3 SYS DBMS_OUTPUT 141 a0 b0 3d b4 55 6a :3 a0 7e 6.25 1.1 SYS DBMS_OUTPUT 96 1WHILE: 6.19 1.1 SYS DBMS_OUTPUT 65 1499: 5.77 1.0 SYS DBMS_OUTPUT 145 7e a0 b4 2e d a0 57 b3

20 rows selected.

Lastly, we'll take a look at some of the code coverage statistics. This is useful not only for profiling and performance tuning, but testing as well. This tells you how many of the statements in the code we have executed, and shows the percentage of the code that has been 'covered':

Appendix A

1170

Number of lines actually executed in different units (by unit_name)

UNIT_OWNER UNIT_NAME LINES_EXECUTED LINES_PRESENT PCT ----------- -------------- -------------- ------------- ------ SYS DBMS_OUTPUT 51 88 58.0 SYS DBMS_PROFILER 9 62 14.5 TKYTE FACT_ITERATIVE 4 4 100.0 TKYTE FACT_RECURSIVE 3 3 100.0

======================Number of lines actually executed for all units

LINES_EXECUTED -------------- 67

======================Total number of lines in all units

LINES_PRESENT ------------- 157

This shows that of the 88 statements in the DBMS_OUTPUT package, we executed 51 of them. It is interesting to note how DBMS_PROFILER counts lines or statements here. It claims that FACT_ITERATIVE has 4 lines of code, but if we look at the source code:

function fact_iterative( n int ) return number as l_result number default 1; begin for i in 2 .. n loop l_result := l_result * i; end loop; return l_result; end;

I don't see four of anything clearly. DBMS_PROFILER is counting statements, and not really lines of code. Here, the four statements are:

... l_result number default 1; ... for i in 2 .. n ... l_result := l_result * i; ... return l_result; ...

DBMS_PROFILER

1171

Everything else, while necessary to compile and execute the code, was not really executable code, and therefore, not statements. DBMS_PROFILER can be used to tell us how many executable statements we have in our code, and how many of them we actually executed.

Caveats The only caveats I have with regards to DBMS_PROFILER, are the amount of data it generates, and the amount of your time it can consume.

The small test case we did above generated some 500-plus rows of observations in the PLSQL_PROFILER_DATA table. This table contains eleven number columns, and so it is not very 'wide', but it grows rapidly. Every statement executed will cause a row to be added to this table. You will need to monitor the space you need for this table, and make sure you clear it out every now and again. Typically, this is not a serious issue, but I have seen this table getting filled with thousands of rows for extremely complex PL/SQL routines (hundreds of thousands of rows).

The amount of your time it can consume, is a more insidious problem. No matter how much you tune, there will always be a line of code that consumes the most amount of time. If you remove this line of code from the top of the list, another one is just waiting to take its place. You will never get a report from DBMS_PROFILER that says, 'everything ran so fast, I won't even generate a report.' In order to use this tool effectively, you have to set some goals for yourself – give yourself a finish line. Either set a time boundary (I will tune this routine to the best of my ability for two hours), or a performance metric boundary (when the run-time is N units long, I will stop). Otherwise, you will find yourself (as I have from time to time) spending an inordinate amount of time fine-tuning a routine that just cannot get any faster.

DBMS_PROFILER is a nice tool with lots of detailed information. It is far too easy to get bogged down in the details.

Summary In this section we have covered the uses of the DBMS_PROFILER package. One of its two main uses is source code profiling to detect where in the code time is being spent, or to compare two different algorithms. The other major use is as a code coverage tool, to report back the percentage of executable statements your test routines actually exercised in the application. While 100 percent code coverage does not assure you of bug free code – it certainly brings you a step closer though.

We also developed a report, based on the example profiler report provided by Oracle. This report extracts the basic information you need in order to use the DBMS_PROFILER tool successfully. It avoids the great detail you can go into, providing you with the aggregate view of what happened in your application, and more details on the most expensive parts. It may be the only report you really need to use with this tool in order to identify bottlenecks and tune your application.

Appendix A

1172

DBMS_UTILITY

The DBMS_UTILITY package is a collection of miscellaneous procedures. It is where many, standalone procedures are placed. The DBMS_UTILITY package is installed in the database by default, and has EXECUTE granted to PUBLIC. The procedures in this package are not related to one another as they typically are in the other packages. For example, all of the entry points in the UTL_FILE package have a common goal and meaning – to perform I/O on a file. The entry points in DBMS_UTILITY are pretty much independent of one another.

In this section, we will look at many of these functions, and the important caveats and issues will be pointed out.

COMPILE_SCHEMAThe goal of the COMPILE_SCHEMA procedure is to attempt to make valid all invalid procedures, packages, triggers, views, types, and so on in a schema. This procedure works in Oracle 8.1.6 by using the SYS.ORDER_OBJECT_BY_DEPENDENCY view. This view returns objects in the order they depend on each other. In Oracle 8.1.7 and higher, this view is no longer used (why this is relevant will be shown below). If we compile the objects in the order that this view returns them, then at the end, all objects that can be valid, should be valid. This procedure runs the ALTER COMPILE command as the user who invoked the procedure (invoker rights).

It should be noted that COMPILE_SCHEMA demands you pass in a case-sensitive username. If you call:

scott@TKYTE816> exec DBMS_UTILITY.compile_schema( 'scott' );

It is probable that nothing will happen, unless you have a lowercase user named scott. You must pass in SCOTT.

DBMS_UTILITY

1173

There is however another issue with COMPILE_SCHEMA in 8.1 versions of the database prior to 8.1.6.2 (that is all 8.1.5, 8.1.6.0, and 8.1.6.1 versions). If you have a Java-enabled database, this will introduce some recursive dependencies into your system. This will cause COMPILE_SCHEMA to raise the error:

scott@TKYTE816> exec dbms_utility.compile_schema( user ); BEGIN dbms_utility.compile_schema( user ); END;

*ERROR at line 1: ORA-01436: CONNECT BY loop in user data ORA-06512: at "SYS.DBMS_UTILITY", line 195 ORA-06512: at line 1

This is coming from the SYS.ORDER_OBJECT_BY_DEPENDENCY view, and is the reason why Oracle 8.1.7 and up do not use this view. If you encounter this error, we can create our own COMPILE_SCHEMAprocedure that behaves exactly as the real COMPILE_SCHEMA. We can do this by compiling the objects in any order we feel like it. It is a common misconception that we must compile objects in some specific order – we can in fact do them in any arbitrary order, and still end up with the same outcome we would have, if we ordered by dependency. The logic is:

1. Pick any invalid object from a schema that we have not yet tried to compile.

2. Compile it.

3. Go back to step one until there are no more invalid objects that we have not yet tried to compile.

It is that simple – we need no special ordering. This is because a side effect of compiling an invalid object is that all invalid objects it depends on will be compiled in order to validate this one. We just have to keep compiling objects until we have no more invalid ones (well, we might have invalid ones, but that would be because they cannot be successfully compiled no matter what). What we might discover is that we need only to compile a single procedure to get 10 or 20 other objects compiled. As long as we don't attempt to manually recompile those 10 or 20 other objects (as this would invalidate the first object again) we are OK.

Since the implementation of this procedure is somewhat interesting, we'll demonstrate it here. We need to rely on an invoker rights routine to do the actual ALTER COMPILE command. However, we need access to the DBA_OBJECTS table to find the 'next' invalid object, and report on the status of the just-compiled object. We do not necessarily want the invoker of the routine to have to have access to DBA_OBJECTS. In order to achieve this, we will use a mixture of invoker rights routines and definer rights routines. We need to make sure that the top-level routine, the one called by the end user, is the invoker rights routine however, to ensure that roles are enabled.

Here is my implementation of a COMPILE_SCHEMA.

The user who runs this script must have had SELECT granted to them on the SYS.DBA_OBJECTSview directly (refer to Chapter 23, Invoker and Definer Rights for details on why this is).

Appendix A

1174

Since this is a SQL*PLUS script, with some SQL*PLUS directives in it, I'll show the script here this time, not the results of actually running the script. I am using a SQL*PLUS substitution variable to fill in the schema name as we compile objects. I am doing this because of the invoker rights routine (the need to fully qualify objects if they should always access the same table, regardless of who is running it), and the fact that I personally do not like to rely on public synonyms. The script will be given to you in pieces below with commentary in between:

column u new_val uname select user u from dual;

drop table compile_schema_tmp/

create global temporary table compile_schema_tmp( object_name varchar2(30), object_type varchar2(30), constraint compile_schema_tmp_pk primary key(object_name,object_type) )on commit preserve rows /

grant all on compile_schema_tmp to public /

We start the script by getting the currently logged in user's username into a SQL*PLUS substitution variable. We will use this later in our CREATE OR REPLACE procedures. We need to do this because our procedure is going to run as an invoker rights routine, and needs to access the table we just created above. If you recall in Chapter 23 on Invoker and Definer Rights, we discussed how references to tables in the procedure are be done using the default schema of the person running the procedure. Well, we only have one temporary table that all users will use, and it will be owned by whoever installs this package. Therefore, we need to hard code the username into the PL/SQL routine. The temporary table is used by our procedures to 'remember' what objects we have attempted to compile. We need to use ON COMMITPRESERVE ROWS because of the fact that we are going to do DDL in our procedure (the ALTERCOMPILE command is DDL), and DDL commits. Next, we can start in on the procedures we need:

create or replace procedure get_next_object_to_compile( p_username in varchar2, p_cmd out varchar2, p_obj out varchar2, p_typ out varchar2 ) asbegin select 'alter ' || object_type || ' ' || p_username || '.' || object_name || decode( object_type, 'PACKAGE BODY', ' compile body', ' compile' ), object_name, object_type into p_cmd, p_obj, p_typ from dba_objects a where owner = upper(p_username) and status = 'INVALID' and object_type <> 'UNDEFINED'

DBMS_UTILITY

1175

and not exists ( select null from compile_schema_tmp b where a.object_name = b.object_name and a.object_type = b.object_type ) and rownum = 1;

insert into compile_schema_tmp ( object_name, object_type ) values ( p_obj, p_typ ); end;/

This is a definer rights procedure that accesses the DBA_OBJECTS view for us. This will return 'some' invalid object to be compiled, as long as we have not yet attempted to compile it. It just finds the first one. As we retrieve them, we 'remember' them in our temporary table. Note that this routine will throw the exception NO_DATA_FOUND when there are no objects left to be compiled in the requested schema – we'll use this fact in our next routine to stop processing. Next, we have our invoker rights routine that will actually do the compilation. This also shows why we needed the COLUMN U NEW_VAL UNAMEdirective above– we need to physically insert the owner of the temporary table in here to avoid having to use a synonym. Since we do this dynamically upon compiling the procedure, it makes it better than a synonym:

create or replace procedure compile_schema( p_username in varchar2 ) authid current_useras l_cmd varchar2(512); l_obj dba_objects.object_name%type; l_typ dba_objects.object_type%type; begin delete from &uname..compile_schema_tmp;

loop get_next_object_to_compile( p_username, l_cmd, l_obj, l_typ );

dbms_output.put_line( l_cmd ); begin execute immediate l_cmd; dbms_output.put_line( 'Successful' ); exception when others then dbms_output.put_line( sqlerrm ); end; dbms_output.put_line( chr(9) ); end loop;

exception – get_next_object_to_compile raises this when done when no_data_found then NULL; end;/

grant execute on compile_schema to public /

Appendix A

1176

And that's it. Now you can go into any schema that is able to compile some objects, and execute:

scott@TKYTE816> exec tkyte.compile_schema('scott') alter PROCEDURE scott.ANALYZE_MY_TABLES compile Successful

alter PROCEDURE scott.CUST_LIST compile ORA-24344: success with compilation error

alter TYPE scott.EMP_MASTER compile ORA-24344: success with compilation error

alter PROCEDURE scott.FOO compile Successful

alter PACKAGE scott.LOADLOBS compile Successful

alter PROCEDURE scott.P compile Successful

alter PROCEDURE scott.RUN_BY_JOBS compile Successful


So, this shows me the objects it attempted to compile, and the outcome. According to the above, we compile seven objects, two of which failed, and five of which succeeded. We compiled them in any order – the order was simply not relevant. This procedure should work in all situations.

ANALYZE_SCHEMA The ANALYZE_SCHEMA routine does pretty much what it sounds like it would do – it performs an ANALYZE to collect statistics for the objects in a user's schema. It is recommended that you never do this on either SYS or SYSTEM. This is especially for SYS, as the recursive SQL generated by Oracle over the years was optimized for execution using the rule-based optimizer. Having statistics on SYS-owned tables will cause your database to operate slower than it should. You may use this procedure to analyze application schemas you have yourself developed.

The ANALYZE_SCHEMA procedure accepts five arguments:

❑ SCHEMA – The schema to be analyzed.

❑ METHOD – ESTIMATE, COMPUTE, or DELETE. If ESTIMATE, then either ESTIMATE_ROWS or ESTIMATE_PERCENT must be non-zero.

❑ ESTIMATE_ROWS – Number of rows to estimate.

❑ ESTIMATE_PERCENT – Percentage of rows to estimate. If ESTIMATE_ROWS is specified, then this parameter is ignored.

❑ METHOD_OPT [ FOR TABLE ] [ FOR ALL [INDEXED] COLUMNS] [SIZE n] [ FOR ALLINDEXES] – These options are the same options as you use with the ANALYZE command itself. You will find these options fully documented in the Oracle8i SQL Reference manual under the ANALYZE commands, for clause.

DBMS_UTILITY

1177

So, for example, to analyze all of the objects in SCOTT's schema, we can do the following. We start by first deleting and then collecting statistics:

scott@TKYTE816> exec dbms_utility.analyze_schema(user,'delete');


scott@TKYTE816> select table_name, num_rows, last_analyzed 2 from user_tables;

TABLE_NAME NUM_ROWS LAST_ANAL ------------------------------ ---------- --------- BONUSCREATE$JAVA$LOB$TABLEDEPT...

12 rows selected.

scott@TKYTE816> exec dbms_utility.analyze_schema(user,'compute');


scott@TKYTE816> select table_name, num_rows, last_analyzed 2 from user_tables;

TABLE_NAME NUM_ROWS LAST_ANAL ------------------------------ ---------- --------- BONUS 0 03-FEB-01 CREATE$JAVA$LOB$TABLE 58 03-FEB-01 DEPT 4 03-FEB-01 ...

12 rows selected.

This simple shows that the ANALYZE COMPUTE actually did its job – the NUM_ROWS and LAST_ANALYZED columns got filled in.

In general, the ANALYZE_SCHEMA procedure is as straightforward as it sounds. If you have the need to specifically analyze certain objects in certain ways, it will not apply. This procedure does the same sort of analysis to each object type, and does not have exceptions. For example, if you are a large data warehouse, and you make use of histograms on specific columns, or sets of columns on certain tables only, ANALYZE_SCHEMA is not what you want. You can use ANALYZE_SCHEMA to get histograms either for every column or none of the columns – not just certain columns. Once you go beyond the 'simple' with regards to analyzing objects, ANALYZE_SCHEMA will not be useful any more. This routine works well for small to medium sized applications, where small to medium is a measure of the amount of data you have. If you have large volumes of data, you will want to analyze in parallel or use special options to analyze on various tables. This will exclude ANALYZE_SCHEMA from being of use to you.

If you do use ANALYZE_SCHEMA, you should be aware of the following two issues. The first has to do with ANALYZE_SCHEMA against a schema that is changing. The second is with respect to objects ANALYZE_SCHEMA does not analyze. We will look at both of these caveats in turn.

Appendix A

1178

ANALYZE_SCHEMA with a Changing Schema Suppose you start an ANALYZE_SCHEMA in the SCOTT schema. You've added some large tables so it will take a while. In another session, you drop or add some objects to SCOTT's schema. The object you drop hasn't been reached by ANALYZE_SCHEMA yet. When it does, you will receive the somewhat misleading message:

scott@TKYTE816> exec dbms_utility.analyze_schema(user,'compute'); BEGIN dbms_utility.analyze_schema(user,'compute'); END;

*ERROR at line 1: ORA-20000: You have insufficient privileges for an object in this schema. ORA-06512: at "SYS.DBMS_UTILITY", line 258 ORA-06512: at line 1

Obviously, you have all of the privileges you need; you own the objects after all. The error here is that a table, which it is trying to analyze no longer exists, when it gets round to analyzing it. Instead of recognizing that the table does not exist anymore, it assumes it does, and the error must be that you do not have sufficient privileges to analyze it. Currently, there is nothing you can do about this other than:

❑ Restart the ANALYZE_SCHEMA.

❑ Do not drop objects while ANALYZE_SCHEMA is executing.

The other thing to be aware of is that objects added to the schema after the ANALYZE_SCHEMA begins will not be analyzed – it will not see them yet. This is fairly harmless, as the ANALYZE_SCHEMA will run to completion successfully.

ANALYZE_SCHEMA does not Analyze Everything There is an open issue with respect to ANALYZE_SCHEMA. It will not analyze an index-organized table that has an overflow segment (see Chapter 7, Indexes, for more information regarding IOTs and overflows). For example if you run the following code:

scott@TKYTE816> drop table t;

Table dropped.

scott@TKYTE816> create table t ( x int primary key, y date ) 2 organization index 3 OVERFLOW TABLESPACE TOOLS 4 /

Table created.

scott@TKYTE816> execute dbms_utility.analyze_schema('SCOTT','COMPUTE')


scott@TKYTE816> select table_name, num_rows, last_analyzed 2 from user_tables 3 where table_name = 'T'; TABLE_NAME NUM_ROWS LAST_ANAL ------------------------------ ---------- --------- T

DBMS_UTILITY

1179

it did not get analyzed. However, if you leave off the OVERFLOW clause:

scott@TKYTE816> drop table t;

Table dropped.

scott@TKYTE816> create table t ( x int primary key, y date ) 2 organization index 3 /

Table created.

scott@TKYTE816> execute dbms_utility.analyze_schema('SCOTT','COMPUTE')


scott@TKYTE816> select table_name, num_rows, last_analyzed 2 from user_tables 3 where table_name = 'T';

TABLE_NAME NUM_ROWS LAST_ANAL ------------------------------ ---------- --------- T 0 03-FEB-01

it does. This does not mean you should leave the OVERFLOW off of your IOTs, but rather that you will have to manually analyze these objects.

ANALYZE_DATABASE This will be an exceptionally short section. Do not use this procedure. It is not realistic on a database of any size, and has a nasty side effect of analyzing the data dictionary (these are SYS owned objects, and we should never analyze these). Do not use it. Simply ignore its existence.

FORMAT_ERROR_STACK FORMAT_ERROR_STACK is a function that, at first glance, would appear to be very useful, but in retrospect, is not at all. In actuality, FORMAT_ERROR_STACK is simply a less functional implementation of SQLERRM (SQL ERRor Message). A simple demonstration will help you to understand what I mean:

scott@TKYTE816> create or replace procedure p1 2 as 3 begin 4 raise program_error; 5 end; 6 /

Procedure created.

Appendix A

1180

scott@TKYTE816> create or replace procedure p2 2 as 3 begin 4 p1; 5 end; 6 /

Procedure created.

scott@TKYTE816> create or replace procedure p3 2 as 3 begin 4 p2; 5 end; 6 /

Procedure created.

scott@TKYTE816> exec p3 BEGIN p3; END;

*ERROR at line 1: ORA-06501: PL/SQL: program error ORA-06512: at "SCOTT.P1", line 4 ORA-06512: at "SCOTT.P2", line 4 ORA-06512: at "SCOTT.P3", line 4 ORA-06512: at line 1

If we have an error, and we do not catch it in an exception handle, the entire error stack is displayed for us, and would be available to use in a Pro*C, OCI, JDBC, and so on, program. You would expect that the DBMS_UTILITY.FORMAT_ERROR_STACK routine would return similar information. You will find however that it loses this important information:

scott@TKYTE816> create or replace procedure p3 2 as 3 begin 4 p2; 5 exception 6 when others then 7 dbms_output.put_line( dbms_utility.format_error_stack ); 8 end; 9 / Procedure created.

scott@TKYTE816> exec p3 ORA-06501: PL/SQL: program error


As you can see, we actually lost the error stack by calling FORMAT_ERROR_STACK! This routine returns the same information SQLERRM would return:

DBMS_UTILITY

1181

scott@TKYTE816> create or replace procedure p3 2 as 3 begin 4 p2; 5 exception 6 when others then 7 dbms_output.put_line( sqlerrm ); 8 end; 9 / Procedure created.

scott@TKYTE816> exec p3 ORA-06501: PL/SQL: program error


Before, I said FORMAT_ERROR_STACK was a less functional SQLERRM. This is because SQLERRM can, not only return the current error message, but it can also return any error message:

scott@TKYTE816> exec dbms_output.put_line( sqlerrm(-1) ); ORA-00001: unique constraint (.) violated


Unfortunately, there simply is no way currently to get the real error stack in PL/SQL. You must let fatal errors propagate up to the calling client routine, in order to get the actual line number of the code that raised the error in the first place.

FORMAT_CALL_STACK Fortunately this function is truly useful compared to FORMAT_ERROR_STACK. This returns to us the current call stack. Using this, we can write some utility procedures such as MY_CALLER and WHO_AM_I.These routines call a procedure to determine what source code from which line number invoked it. This is very useful for debugging and logging purposes. Also, a procedure could modify its behavior based on who called it, or where it was called.

Before we introduce the code for MY_CALLER and WHO_AM_I, let us look at what the call stack provides for us, and what the output from these routines is destined to be. If we use the P1, P2, P3 example from above, and rewrite P1 to be:

scott@TKYTE816> create or replace procedure p1 2 as 3 l_owner varchar2(30); 4 l_name varchar2(30); 5 l_lineno number; 6 l_type varchar2(30); 7 begin 8 dbms_output.put_line( '----------------------' ); 9 dbms_output.put_line( dbms_utility.format_call_stack ); 10 dbms_output.put_line( '----------------------' ); 11 who_called_me( l_owner, l_name, l_lineno, l_type );

Appendix A

1182

12 dbms_output.put_line( l_type || ' ' || 13 l_owner || '.' || l_name || 14 '(' || l_lineno || ')' ); 15 dbms_output.put_line( '----------------------' ); 16 dbms_output.put_line( who_am_i ); 17 dbms_output.put_line( '----------------------' ); 18 raise program_error; 19 end; 20 /

Procedure created.

we will receive output such as:

scott@TKYTE816> exec p3 --------------------------- PL/SQL Call Stack ----- object line object handle number name 2f191e0 9 procedure SCOTT.P1 39f0a9c 4 procedure SCOTT.P2 3aae318 4 procedure SCOTT.P3 3a3461c 1 anonymous block ----------------------PROCEDURE SCOTT.P2(4) ----------------------SCOTT.P1(16)----------------------BEGIN p3; END;

*ERROR at line 1: ORA-06501: PL/SQL: program error ORA-06512: at "SCOTT.P2", line 8 ORA-06512: at "SCOTT.P3", line 4 ORA-06512: at line 1

So, we can see the entire call stack in P1. This shows that P1 was called by P2, P2 was called by P3, and P3 was called by an anonymous block. Additionally, we can procedurally retrieve the fact that our caller in P1 was the procedure SCOTT.P2, and that they called us from line 4. Lastly, we can see simply that we are the procedure SCOTT.P1.

So, now that we see what the call stack looks like, and what kind of output we would like to get, we can present the code to do it:

tkyte@TKYTE816> create or replace function my_caller return varchar2 2 3 as 4 owner varchar2(30); 5 name varchar2(30); 6 lineno number; 7 caller_t varchar2(30); 8 call_stack varchar2(4096) default dbms_utility.format_call_stack;

DBMS_UTILITY

1183

9 n number; 10 found_stack BOOLEAN default FALSE; 11 line varchar2(255); 12 cnt number := 0; 13 begin 14 15 loop 16 n := instr( call_stack, chr(10) ); 17 exit when ( cnt = 3 or n is NULL or n = 0 ); 18 19 line := substr( call_stack, 1, n-1 ); 20 call_stack := substr( call_stack, n+1 ); 21 22 if ( NOT found_stack ) then 23 if ( line like '%handle%number%name%' ) then 24 found_stack := TRUE; 25 end if; 26 else 27 cnt := cnt + 1; 28 -- cnt = 1 is ME 29 -- cnt = 2 is MY Caller 30 -- cnt = 3 is Their Caller 31 if ( cnt = 3 ) then 32 lineno := to_number(substr( line, 13, 6 )); 33 line := substr( line, 21 ); 34 if ( line like 'pr%' ) then 35 n := length( 'procedure ' ); 36 elsif ( line like 'fun%' ) then 37 n := length( 'function ' ); 38 elsif ( line like 'package body%' ) then 39 n := length( 'package body ' ); 40 elsif ( line like 'pack%' ) then 41 n := length( 'package ' ); 42 elsif ( line like 'anonymous block%' ) then 43 n := length( 'anonymous block ' ); 44 else -- must be a trigger 45 n := 0; 46 end if; 47 if ( n <> 0 ) then 48 caller_t := ltrim(rtrim(upper(substr(line,1,n-1)))); 49 line := substr( line, n ); 50 else 51 caller_t := 'TRIGGER'; 52 line := ltrim( line ); 53 end if; 54 n := instr( line, '.' ); 55 owner := ltrim(rtrim(substr( line, 1, n-1 ))); 56 name := ltrim(rtrim(substr( line, n+1 ))); 57 end if; 58 end if; 59 end loop; 60 return owner || '.' || name; 61 end; 62 /

Appendix A

1184

Function created.

tkyte@TKYTE816> create or replace function who_am_i return varchar2 2 as 3 begin 4 return my_caller; 5 end; 6 /

Function created.

Now that you have these routines, you can do some interesting things. It has been used to

❑ Perform auditing – The audit routines log not only the user that performed some operation, but also the code that did it as well.

❑ Perform debugging – For example, if you litter your code with calls to DBMS_APPLICATION_INFO.SET_CLIENT_INFO(WHO_AM_I), you can query V$SESSION in another session to see where in your code you are currently. See the earlier section of this appendix on DBMS_APPLICATION_INFO for details on this package.

GET_TIMEThis function returns a ticker that measures time in hundredths of a second. You cannot use GET_TIMEto tell what time it is, a function it's name may imply, but rather you can use this to measure elapsed time. A common way to do this is:

scott@TKYTE816> declare 2 l_start number; 3 n number := 0; 4 begin 5 6 l_start := dbms_utility.get_time; 7 8 for x in 1 .. 100000 9 loop 10 n := n+1; 11 end loop; 12 13 dbms_output.put_line( ' it took ' || 14 round( (dbms_utility.get_time-l_start)/100, 2 ) || 15 ' seconds...' ); 16 end; 17 / it took .12 seconds...


DBMS_UTILITY

1185

so you can use GET_TIME to measure elapsed time in hundredths of a second. You should realize however, that GET_TIME will wrap around to zero and start counting again if your database is up long enough. Now, on most platforms this time to wrap is well over a year in length. The counter is a 32-bit integer, and this can hold hundredths of seconds for about 497 days. After that, the 32-bit integer will roll over to zero, and start over again. On some platforms, the operating system supplies this ticker in a smaller increment than hundredths of seconds. On these platforms the ticker may roll over sooner than 497 days. For example, on Sequent it is known that the timer will roll over every 71.58 minutes, since this operating system's ticker measures time in microseconds, leaving significantly less room in the 32-bit integer. On a 64-bit platform, the time may very well not roll over for many thousands of years.

A last note about GET_TIME. The same value that GET_TIME returns may be retrieved from a SELECT *FROM V$TIMER. The dynamic view and GET_TIME return the same values:

tkyte@TKYTE816> select hsecs, dbms_utility.get_time 2 from v$timer;

HSECS GET_TIME---------- ---------- 7944822 7944822

GET_PARAMETER_VALUE This API allows anyone to get the value of a specific init.ora parameter. Even if you have no access to V$PARAMETER, and cannot run the SHOW PARAMETER command, you can use this to get the value of an init.ora parameter. It works like this:

scott@TKYTE816> show parameter utl_file_dirORA-00942: table or view does not exist

scott@TKYTE816> select * from v$parameter where name = 'utl_file_dir' 2 / select * from v$parameter where name = 'utl_file_dir' * ERROR at line 1: ORA-00942: table or view does not exist

scott@TKYTE816> declare 2 intval number; 3 strval varchar2(512); 4 begin 5 if ( dbms_utility.get_parameter_value( 'utl_file_dir', 6 intval, 7 strval ) = 0) 8 then 9 dbms_output.put_line( 'Value = ' || intval ); 10 else 11 dbms_output.put_line( 'Value = ' || strval ); 12 end if; 13 end; 14 / Value = c:\temp\


Appendix A

1186

As you can see, even though SCOTT cannot query V$PARAMETER and the call to show parameter failed, he can still use this call to get the value. It should be noted that parameters set with True/False strings in the init.ora file will be reported back as returning a number type (this particular function will return 0), and a value of 1 indicates True while a value of 0 indicates False. Additionally, for multi-valued parameters, such as UTL_FILE_DIR, this routine only returns the first value. If I use an account that can do a SHOW PARAMETER in the same database:

tkyte@TKYTE816> show parameter utl_file_dir

NAME TYPE VALUE ------------------------------------ ------- ------------------------------ utl_file_dir string c:\temp, c:\oracle

I can see more values.

NAME_RESOLVE This routine will take the name of a:

❑ Top-level procedure

❑ Top-level function

❑ Database package name

❑ A synonym that points to a database package, or a top level procedure or function

and fully resolve the name for you. It can tell you if the object name you gave it is a procedure, function, or package, and what schema it belongs to. Here is a simple example:

scott@TKYTE816> declare 2 type vcArray is table of varchar2(30); 3 l_types vcArray := vcArray( null, null, null, null, 'synonym', 4 null, 'procedure', 'function', 5 'package' ); 6 7 l_schema varchar2(30); 8 l_part1 varchar2(30); 9 l_part2 varchar2(30); 10 l_dblink varchar2(30); 11 l_type number; 12 l_obj# number; 13 begin 14 dbms_utility.name_resolve( name => 'DBMS_UTILITY', 15 context => 1, 16 schema => l_schema, 17 part1 => l_part1, 18 part2 => l_part2, 19 dblink => l_dblink, 20 part1_type => l_type, 21 object_number => l_obj# ); 22 if l_obj# IS NULL

DBMS_UTILITY

1187

23 then 24 dbms_output.put_line('Object not found or not valid.'); 25 else 26 dbms_output.put( l_schema || '.' || nvl(l_part1,l_part2) ); 27 if l_part2 is not null and l_part1 is not null 28 then 29 dbms_output.put( '.' || l_part2 ); 30 end if; 31 32 dbms_output.put_line( ' is a ' || l_types( l_type ) || 33 ' with object id ' || l_obj# || 34 ' and dblink "' || l_dblink || '"' ); 35 end if; 36 end; 37 / SYS.DBMS_UTILITY is a package with object id 2408 and dblink ""


In this case, NAME_RESOLVE took our synonym DBMS_UTILITY, and figured out for us that this was in fact a database package that is owned by SYS.

It should be noted that NAME_RESOLVE works only on procedures, functions, packages, and synonyms that point to one of these three object types. It explicitly will not work on a database table for example. You will receive the following error:

declare*ERROR at line 1: ORA-06564: object emp does not exist ORA-06512: at "SYS.DBMS_UTILITY", line 68 ORA-06512: at line 9

if you attempt to use it on the EMP table in the SCOTT schema for example.

In addition to not being able to do tables, indexes, and other objects, NAME_RESOLVE does not function as documented when it comes to resolving synonyms that point to remote objects over a database link. It is documented that if you pass NAME_RESOLVE a synonym to a remote package/procedure, then the TYPE will be set to synonym, and they will tell us the name of the database link. This is an issue with the NAME_RESOLVE code (the documentation is correct, the procedure does not function as it should). Currently, NAME_RESOLVE will never return SYNONYM as the type. Rather, it will resolve the remote object and return its name and an object ID of –1. For example, I have a database link set up, and I create a synonym X for [email protected]. When I NAME_RESOLVE this, I receive:

SYS.DBMS_UTILITY is a package with object id -1 and dblink ""


I should have been told that X was a synonym and the DBLINK OUT parameter would have been filled in. As you can see however, the DBLINK is Null, and the only indication we have that this is not a local package, is the fact that the object ID is set to –1. You should not rely on this behavior persisting in future releases of Oracle. It has been determined as an issue in the NAME_RESOLVE implementation, and is not a documentation issue. The documentation is correct, the observed behavior is wrong. When this gets corrected, NAME_RESOLVE will function differently on remote objects. For this reason, you will want to either avoid using NAME_RESOLVE on remote objects or make sure to 'wrap' the NAME_RESOLVE routine in some function of your own. This will make it so that when, and if, the behavior changes, you can easily modify your code to provide yourself with the old functionality, if that is what you depend on.

Appendix A

1188

One last comment about NAME_RESOLVE. The parameters CONTEXT and OBJECT_NUMBER are under-documented and not documented, respectively. The CONTEXT parameter is documented briefly as:

.. . must be an integer between 0 and 8

In fact, it must be an integer between 1 and 7 or you'll receive:

declare*ERROR at line 1: ORA-20005: ORU-10034: context argument must be 1 or 2 or 3 or 4 or 5 or 6 or 7 ORA-06512: at "SYS.DBMS_UTILITY", line 66 ORA-06512: at line 14

And if it is anything other than 1, you will receive one of the two following error messages:

ORA-04047: object specified is incompatible with the flag specified

ORA-06564: object OBJECT-NAME does not exist

So, the only valid value for context is 1. The OBJECT_NUMBER parameter is not documented at all. This is the OBJECT_ID value found in DBA_OBJECTS, ALL_OBJECTS and USER_OBJECTS. For example, given our first example where the OBJECT_ID was shown to be 2048 I can query:

scott@TKYTE816> select owner, object_name 2 from all_objects 3 where object_id = 2408;

OWNER OBJECT_NAME------------------------------ ------------------------------ SYS DBMS_UTILITY

NAME_TOKENIZEThis utility routine simply takes a string that represents some object name, and breaks it into its component pieces for you. Objects are referenced via:

[schema].[object_name].[procedure|function]@[database link]

NAME_TOKENIZE simply takes a string in this form, and breaks it out into the three leading pieces and the last (database link) piece. Additionally, it tells us what byte it stopped parsing the object name at. Here is a small example showing what you might expect back from various object names you pass to it. Note that you do not have to use real object names (these tables and procedures do not have to exist), but you must use valid object identifiers. If you do not use a valid object identifier, NAME_TOKENIZE will raise an error. This makes NAME_TOKENIZE suitable as a method to discover whether a given string of characters will be a valid identifier or not:

scott@TKYTE816> declare 2 l_a varchar2(30); 3 l_b varchar2(30); 4 l_c varchar2(30); 5 l_dblink varchar2(30);

DBMS_UTILITY

1189

6 l_next number; 7 8 type vcArray is table of varchar2(255); 9 l_names vcArray := 10 vcArray( 'owner.pkg.proc@database_link', 11 'owner.tbl@database_link', 12 'tbl', 13 '"Owner".tbl', 14 'pkg.proc', 15 'owner.pkg.proc', 16 'proc', 17 'owner.pkg.proc@dblink with junk', 18 '123' ); 19 begin 20 for i in 1 .. l_names.count 21 loop 22 begin 23 dbms_utility.name_tokenize(name => l_names(i), 24 a => l_a, 25 b => l_b, 26 c => l_c, 27 dblink => l_dblink, 28 nextpos=> l_next ); 29 30 dbms_output.put_line( 'name ' || l_names(i) ); 31 dbms_output.put_line( 'A ' || l_a ); 32 dbms_output.put_line( 'B ' || l_b ); 33 dbms_output.put_line( 'C ' || l_c ); 34 dbms_output.put_line( 'dblink ' || l_dblink ); 35 dbms_output.put_line( 'next ' || l_next || ' ' || 36 length(l_names(i))); 37 dbms_output.put_line( '-----------------------' ); 38 exception 39 when others then 40 dbms_output.put_line( 'name ' || l_names(i) ); 41 dbms_output.put_line( sqlerrm ); 42 end; 43 end loop; 44 end; 45 / name owner.pkg.proc@database_link A OWNER B PKG C PROC dblink DATABASE_LINKnext 28 28

As you can see, this breaks out the various bits and pieces of our object name for us. Here the NEXT is set to the length of the string – parsing ended when we hit the end of the string in this case. Since we used every possible piece of the object name, all four components are filled in. Now for the remaining examples:

name owner.tbl@database_linkA OWNER B TBL Cdblink DATABASE_LINKnext 23 23

Appendix A

1190

-----------------------name tbl A TBL BCdblinknext 3 3 -----------------------

Notice here how B and C are left Null. Even though an object identifier is SCHEMA.OBJECT.PROCEDURE, NAME_TOKENIZE makes no attempt to put TBL into the B OUT parameter. It simply takes the first part it finds, and puts it in A, the next into B, and so on. A, B, and Cdo not represent specific pieces of the object name, just the first found, next found, and so on.

name "Owner".tbl A Owner B TBL Cdblinknext 11 11 -----------------------

Here is something interesting. In the previous examples, NAME_TOKENIZE uppercased everything. This is because identifiers are in uppercase unless you use quoted identifiers. Here, we used a quoted identifier. NAME_TOKENIZE will preserve this for us, and remove the quotes!

name pkg.proc A PKG B PROC Cdblinknext 8 8 -----------------------name owner.pkg.proc A OWNER B PKG C PROC dblinknext 14 14 -----------------------name proc A PROC BCdblinknext 4 4 -----------------------name owner.pkg.proc@dblink with junk A OWNER B PKG C PROC dblink DBLINK next 22 31 -----------------------

DBMS_UTILITY

1191

Here is an example where the parsing stopped before we ran out of string. NAME_TOKENIZE is telling us it stopped parsing at byte 22 out of 31. This is the space right before with junk. It simply ignores the remaining pieces of the string for us.

name 123 ORA-00931: missing identifier


And lastly, this shows if we use an invalid identifier, NAME_TOKENIZE will raise an exception. It checks all tokens for being valid identifiers before returning. This makes it useful as a tool to validate object names if you are building an application that will create objects in the Oracle database. For example, if you are building a data modeling tool, and would like to validate that the name the end user wants to use for a table or column name is valid, NAME_TOKENIZE will do the work for you.

COMMA_TO_TABLE, TABLE_TO_COMMA These two utilities either take a comma-delimited string of identifiers and parse them into a PL/SQL table (COMMA_TO_TABLE), or take a PL/SQL table of any type of string, and make a comma-delimited string of them (TABLE_TO_COMMA). I stress the word identifiers above, because COMMA_TO_TABLE uses NAME_TOKENIZE to parse the strings, hence as we saw in that section, we need to use valid Oracle identifiers (or quoted identifiers). This still limits us to 30 characters per element in our comma-delimited string however.

This utility is most useful for applications that want to store a list of table names in a single string for example, and have them easily converted to an array in PL/SQL at run-time. Otherwise, it is of limited use. If you need a general purpose COMMA_TO_TABLE routine that works with comma-delimited strings of data, see Chapter 20, Using Object Relational Features. In the SELECT * from PLSQL_FUNCTIONsection, I demonstrate how to do this.

Here is an example using this routine, demonstrating how it deals with long identifiers and invalid identifiers:

scott@TKYTE816> declare 2 type vcArray is table of varchar2(4000); 3 4 l_names vcArray := vcArray( 'emp,dept,bonus', 5 'a, b , c', 6 '123, 456, 789', 7 '"123", "456", "789"', 8 '"This is a long string, longer then 32 characters","b",c'); 9 l_tablen number; 10 l_tab dbms_utility.uncl_array; 11 begin 12 for i in 1 .. l_names.count 13 loop 14 dbms_output.put_line( chr(10) || 15 '[' || l_names(i) || ']' ); 16 begin 17 18 dbms_utility.comma_to_table( l_names(i), 19 l_tablen, l_tab ); 20

Appendix A

1192

21 for j in 1..l_tablen 22 loop 23 dbms_output.put_line( '[' || l_tab(j) || ']' ); 24 end loop; 25 26 l_names(i) := null; 27 dbms_utility.table_to_comma( l_tab, 28 l_tablen, l_names(i) ); 29 dbms_output.put_line( l_names(i) ); 30 exception 31 when others then 32 dbms_output.put_line( sqlerrm ); 33 end; 34 end loop; 35 end; 36 /

[emp,dept,bonus][emp][dept][bonus]emp,dept,bonus

So, that shows that it can take the string emp,dept,bonus, break it into a table, and put it back together again.

[a, b, c] [a][ b ] [ c] a, b, c

This example shows that if you have whitespace in the list, it will be preserved. You would have to use the TRIM function to remove leading and trailing white space if you do not want any.

[123, 456, 789] ORA-00931: missing identifier

This shows that to use this procedure on a comma-delimited string of numbers, we must go one step further as demonstrated below:

["123", "456", "789"] ["123"][ "456"] [ "789"] "123", "456", "789"

Here, it is able to extract the numbers from the string. Note however, how it not only retains the leading whitespace, but it also retains the quotes. It would be up to you to remove them if you so desire.

["This is a long string, longer than 32 characters","b",c] ORA-00972: identifier is too long


DBMS_UTILITY

1193

And this last example shows that if the identifier is too long (longer than 30 characters), it will raise an error as well. These routines are only useful for strings of 30 characters or less. While it is true that TABLE_TO_COMMA will take larger strings than 30 characters, COMMA_TO_TABLE will not be able to undo this work.

DB_VERSION and PORT_STRING The DB_VERSION routine was added in Oracle 8.0, in order to make it easier for applications to figure out what version of the database they were running in. We could have used this in our CRYPT_PKG (see the DBMS_OBFUSCATION_TOOLKIT section) for example, to tell users who attempted to use the DES3routines in an Oracle 8.1.5 database that it would not work, instead of just trying to execute the DES3routines and failing. It is a very simple interface as follows:

scott@TKYTE816> declare 2 l_version varchar2(255); 3 l_compatibility varchar2(255); 4 begin 5 dbms_utility.db_version( l_version, l_compatibility ); 6 dbms_output.put_line( l_version ); 7 dbms_output.put_line( l_compatibility ); 8 end; 9 / 8.1.6.0.08.1.6


And provides more version detail than the older function, PORT_STRING:

scott@TKYTE816> select dbms_utility.port_string from dual;

PORT_STRING---------------------------IBMPC/WIN_NT-8.1.0

Using the PORT_STRING, not only would you have to parse the string, but you also cannot tell if you are in version 8.1.5 versus 8.1.6 versus 8.1.7. DB_VERSION will be more useful for this. On the other hand, the PORT_STRING does tell you what operating system you are on.

GET_HASH_VALUE This function will take any string as input, and return a numeric HASH value for it. You could use this to build your own 'index by table' that was indexed by a string, or as we did in the DBMS_LOCK section, to facilitate the implementation of some other algorithm.

You should be aware that the algorithm used to implement GET_HASH_VALUE can, and has, changed from release to release, so you should not use this function to generate surrogate keys. If you find yourself storing the return value from this function in a table, you might be setting yourself up for a problem in a later release when the same inputs return a different hash value!

Appendix A

1194

This function takes three inputs:

❑ The string to hash.

❑ The 'base' number to be returned. If you want the numbers to range from 0 to some number, use 0 for your base.

❑ The size of the hash table. Optimally this number would be a power of two.

As a demonstration of using the GET_HASH_VALUE, we will implement a new type, HASHTABLETYPE, to add to the PL/SQL language a hash type. This is very similar to a PL/SQL table type that is indexed by a VARCHAR2 string instead of a number. Normally, PL/SQL table elements are referenced by subscripts (numbers). This new type of PL/SQL table will have elements that are referenced by arbitrary strings. This will allow us to declare variables of type HASHTABLETYPE and GET and PUT values into it. We can have as many of these table types as we like. Here is the specification for our package:

tkyte@TKYTE816> create or replace type myScalarType 2 as object 3 ( key varchar2(4000), 4 val varchar2(4000) 5 ); 6 /

Type created.

tkyte@TKYTE816> create or replace type myArrayType 2 as varray(10000) of myScalarType; 3 /

Type created.

tkyte@TKYTE816> create or replace type hashTableType 2 as object 3 ( 4 g_hash_size number, 5 g_hash_table myArrayType, 6 g_collision_cnt number, 7 8 static function new( p_hash_size in number ) 9 return hashTableType, 10 11 member procedure put( p_key in varchar2, 12 p_val in varchar2 ), 13 14 member function get( p_key in varchar2 ) 15 return varchar2, 16 17 member procedure print_stats 18 ); 19 /

Type created.

An interesting implementation detail here is the addition of the static member function NEW. This will allow us to create our own constructor. You should note that there is absolutely nothing special about the name NEW that I used. It is not a keyword or anything like that. What NEW will allow us to do is to declare a HASHTABLETYPE like this:

DBMS_UTILITY

1195

declare l_hashTable hashTableType := hashTableType.new( 1024 );

instead of like this:

declare l_hashTable hashTableType := hashTableType( 1024, myArrayType(), 0 );

It is my belief that the first syntax is in general, more readable and clearer than the second. The second syntax makes the end user aware of many of the implementation details (that we have an array type in there, that there is some variable G_COLLISION_CNT that must be set to zero, and so on). They neither need to know that nor do they really care.

So, now onto the type body itself:

scott@TKYTE816> create or replace type body hashTableType 2 as 3 4 -- Our 'friendly' constructor. 5 6 static function new( p_hash_size in number ) 7 return hashTableType 8 is 9 begin 10 return hashTableType( p_hash_size, myArrayType(), 0 ); 11 end; 12 13 member procedure put( p_key in varchar2, p_val in varchar2 ) 14 is 15 l_hash number := 16 dbms_utility.get_hash_value( p_key, 1, g_hash_size ); 17 begin 18 19 if ( p_key is null ) 20 then 21 raise_application_error( -20001, 'Cannot have NULL key' ); 22 end if; 23

This next piece of code looks to see if we need to 'grow' the table to hold this new, hashed value. If we do, we grow it out big enough to hold this index:

27 if ( l_hash > nvl( g_hash_table.count, 0 ) ) 28 then 29 g_hash_table.extend( l_hash-nvl(g_hash_table.count,0)+1 ); 30 end if; 31

Now, there is no guarantee that the index entry our key hashed to is empty. What we do upon detecting a collision is to try and put it in the next collection element. We search forward for up to 1,000 times to put it into the table. If we hit 1,000 collisions, we will fail. This would indicate that the table is not sized properly, if this is the case:

Appendix A

1196

35 for i in 0 .. 1000 36 loop 37 -- If we are going to go past the end of the 38 -- table, add another slot first. 39 if ( g_hash_table.count <= l_hash+i ) 40 then 41 g_hash_table.extend; 42 end if; 43

The next bit of logic says 'if no one is using this slot or our key is in this slot already, use it and return.' It looks a tad strange to check if the G_HASH_TABLE element is Null, or if the G_HASH_TABLE(L_HASH+I).KEY is Null. This just shows that a collection element may be Null, or it may contain an object that has Null attributes:

46 if ( g_hash_table(l_hash+i) is null OR 47 nvl(g_hash_table(l_hash+i).key,p_key) = p_key ) 48 then 49 g_hash_table(l_hash+i) := myScalarType(p_key,p_val); 50 return; 51 end if; 52 53 -- Else increment a collision count and continue 54 -- onto the next slot. 55 g_collision_cnt := g_collision_cnt+1; 56 end loop; 57 58 -- If we get here, the table was allocate too small. 59 -- Make it bigger. 60 raise_application_error( -20001, 'table overhashed' ); 61 end; 62 63 64 member function get( p_key in varchar2 ) return varchar2 65 is 66 l_hash number := 67 dbms_utility.get_hash_value( p_key, 1, g_hash_size ); 68 begin

When we go to retrieve a value, we look in the index element we think the value should be in, and then look ahead up to 1,000 entries in the event we had collisions. We short circuit this look-ahead search if we ever find an empty slot – we know our entry cannot be beyond that point:

71 for i in l_hash .. least(l_hash+1000, nvl(g_hash_table.count,0)) 72 loop 73 -- If we hit an EMPTY slot, we KNOW our value cannot 74 -- be in the table. We would have put it there. 75 if ( g_hash_table(i) is NULL ) 76 then 77 return NULL; 78 end if; 79 80 -- If we find our key, return the value.

DBMS_UTILITY

1197

81 if ( g_hash_table(i).key = p_key ) 82 then 83 return g_hash_table(i).val; 84 end if; 85 end loop; 86 87 -- Key is not in the table. Quit. 88 return null; 89 end; 90

The last routine is used to print out useful information, such as how many slots you've allocated versus used, and how many collisions we had. Note that collisions can be bigger than the table itself!

97 member procedure print_stats 98 is 99 l_used number default 0; 100 begin 101 for i in 1 .. nvl(g_hash_table.count,0) 102 loop 103 if ( g_hash_table(i) is not null ) 104 then 105 l_used := l_used + 1; 106 end if; 107 end loop; 108109 dbms_output.put_line( 'Table Extended To.....' || 110 g_hash_table.count ); 111 dbms_output.put_line( 'We are using..........' || 112 l_used ); 113 dbms_output.put_line( 'Collision count put...' || 114 g_collision_cnt ); 115 end; 116117 end; 118 /

Type body created.

As you can see, we simply used the GET_HASH_VALUE to turn the string into some number we could use to index into our table type, to get the value. Now we are ready to see how this new type can be used:

tkyte@TKYTE816> declare 2 l_hashTbl hashTableType := hashTableType.new(power(2,7)); 3 begin 4 for x in ( select username, created from all_users ) 5 loop 6 l_hashTbl.put( x.username, x.created ); 7 end loop; 8 9 for x in ( select username, to_char(created) created, 10 l_hashTbl.get(username) hash 11 from all_users )

Appendix A

1198

12 loop 13 if ( nvl( x.created, 'x') <> nvl(x.hash,'x') ) 14 then 15 raise program_error; 16 end if; 17 end loop; 18 19 l_hashTbl.print_stats; 20 end; 21 / Table Extended To.....120 We are using..........17 Collision count put...1


And that's it. We've just extended the PL/SQL language again, giving it a hash table using the built-in packages.

Summary This wraps up our overview of many of the procedures found in the DBMS_UTILITY package. Many, such as GET_TIME, GET_PARAMETER_VALUE, GET_HASH_VALUE, and FORMAT_CALL_STACK are in my list of 'frequently given answers'. This is to say, they are frequently the answer to many a question – people just weren't even aware they even existed.

UTL_FILE

1199

UTL_FILE

UTL_FILE is a package that is supplied to allow PL/SQL to read and create text files in the file system of the server. The keywords here are:

❑ Text Files – UTL_FILE can only read and create clear text files. Specifically, it cannot be used to read or create binary files. Special characters contained within arbitrary binary data will cause UTL_FILE to do the wrong thing.

❑ File System of the Server – UTL_FILE can only read and write to the file system of the database server. It cannot read or write to the file system the client is executing on if that client is not logged onto the server itself.

UTL_FILE is an appropriate tool for creating reports and flat file dumps of data from the database or for reading files of data to be loaded. In fact, if you refer to Chapter 9 on Data Loading in this book, we have a full blown example of using UTL_FILE to create flat file dumps in a format suitable for easy reloading. UTL_FILE is also a good choice for doing 'debugging'. If you refer to the Chapter 21 on Fine Grained Access Control we introduced the DEBUG package. This package makes heavy use of UTL_FILE to record messages in the file system.

UTL_FILE is an extremely useful package as long as you are aware of its limits. If not, you may attempt to use UTL_FILE in a way in which it will not work correctly (it might work in testing but not in production) leading to frustration. Like all tools, knowing its limits and how it works will be useful.

Appendix A

1200

We'll look at some issues frequently encountered when using UTL_FILE including:

❑ The UTL_FILE_DIR parameter in init.ora.

❑ Accessing mapped drives (network drives) in a Windows environment (there are no related issues in a Unix environment).

❑ Handling exceptions from UTL_FILE

❑ Using UTL_FILE to create static web pages on a recurring basis.

❑ The infamous 1023 byte limit.

❑ Getting a directory listing so you can process all files in a directory.

The UTL_FILE_DIR init.ora parameter This is a key part to using UTL_FILE, which always runs as the Oracle software owner – it is your dedicated server or shared server that is performing the I/O and these are running as 'Oracle'. Given that they run as Oracle and Oracle can read and write to its datafiles, configuration files, and so on – it would be a very bad thing if UTL_FILE permitted access to just any directory. The fact that we must explicitly set the directories we want to be able to write to in the init.ora is a great safety feature – it is not an encumbrance. Consider if UTL_FILE allowed you to write to any directory Oracle could – any user could then use UTL_FILE.FOPEN to rewrite your system datafile. That, to put it mildly, would be a bad thing. Therefore, your DBA must open up access to specific directories – explicitly. You cannot even specify a root directory and allow access to it and all directories underneath – you must explicitly list out each and every directory you want to read and write to with UTL_FILE.

It should be noted that this init.ora parameter is not changeable while the database is up and running. You must restart the instance in order for a directory entry to be added or removed.

The UTL_FILE_DIR init.ora parameter takes one of two forms:

utl_file_dir = (c:\temp,c:\temp2)

or

utl_file_dir = c:\temp utl_file_dir = c:\temp2

That is, you may either use a list of comma separated directories enclosed in parenthesis or you may list each directory on a line one after the other. The keywords here being 'one after the other'. If you have the following as the last lines in your init.ora file:

utl_file_dir = c:\temp timed_statistics=true utl_file_dir = c:\temp2

Only the last entry for UTL_FILE_DIR will be used. The first directory entry will effectively be ignored. This can be quite confusing as there will be no warning message or alert.log entries indicating the first UTL_FILE_DIR entry is ignored. All UTL_FILE_DIR entries must be contiguous in the init.ora file.

UTL_FILE

1201

One word of warning on the Windows Platform with regards to this init.ora parameter. If you decide to add the trailing \ to the UTL_FILE_dir parameter like this:

utl_file_dir = c:\temp\ utl_file_dir = c:\temp2

You will receive the following error message upon startup:

SVRMGR> startup LRM-00116: syntax error at 'c:\temputl_file_' following '=' LRM-00113: error when processing file 'C:\oracle\admin\tkyte816\pfile\init.ora' ORA-01078: failure in processing system parameters

That is because the \ is considered an escape character at the end of the line in the init.ora file. It would allow you normally to continue long entries on 2 lines. You must simply use two slashes:

utl_file_dir = c:\temp\\ utl_file_dir = c:\oracle

in order to avoid this concatenation.

Another closing note on this init.ora parameter. If you use a trailing slash in the init.ora, you must use the trailing slash in your fopen calls. If you omit the trailing slash in the init.ora, you should omit it in your fopen calls as well. The directory parameter to fopen should match in case and contents the value you put into the init.ora file.

Accessing Mapped Windows Drives This is a common area of confusion, especially to people used to working with Unix. On Unix, if you mount a device (for example, NFS mount a remote disk) – it is immediately visible to everyone on that machine – regardless of their session. Each user may have different access rights to it, but the mounted disk is an attribute of the system and not a specific session.

In Windows, this is very different. I may have many sessions executing on a given server and each will have its own set of 'disk drives' that it sees. It may very well be that when I log onto a machine – I see a network resource 'disk D:' that belongs physically to some other machine. That does not mean that every process running on that machine can see that disk. This is where the confusion comes in.

Many people log into the server and see 'disk D:'. They configure the init.ora to have UTL_FILE_dir =d:\reports in it ; a directory to write reports to using UTL_FILE. At runtime however they receive:

ERROR at line 1: ORA-06510: PL/SQL: unhandled user-defined exception ORA-06512: at "SYS.UTL_FILE", line 98 ORA-06512: at "SYS.UTL_FILE", line 157

Which if they used an exception handler (see below for the one I like to use) they would see something more informative like:

Appendix A

1202

ERROR at line 1: ORA-20001: INVALID_PATH: File location or filename was invalid.ORA-06512: at "TKYTE.CSV", line 51 ORA-06512: at line 2

Well, as far as they can tell – D:\reports is just fine. They use explorer and it is there. They use a DOS window and it is there. Only Oracle doesn't seem to be able to see it. This is because when the system started – D: drive didn't exist and furthermore, the account under which Oracle is running by default cannot see any network resources. Try as hard as you like, mounting that disk in any way possible, Oracle will not 'see' it.

When an Oracle instance is created the services that support it are setup to 'Log On As' the SYSTEM (or operating system) account, this account has very few privileges and no access to Windows NT Domains. To access another Windows NT machine the OracleServiceXXXX must be setup to logon to the appropriate Windows NT Domain as a user who has access to the required location for UTL_FILE.

To change the default logon for the Oracle services, go to (in Windows NT):

Control Panel | Services | OracleServiceXXXX | Startup | Log On As; (where XXXX is the instance name)

In Windows 2000, this would be:

Control Panel | Administrative Tools | Services | OracleServiceXXXX | Properties | Log On Tab; again XXXX is the instance name)

Choose the This Account radio button, and then complete the appropriate domain login information. Once the services have been setup as a user with the appropriate privileges, there are two options for setting UTL_FILE_DIR:

❑ Mapped Drive: To use a mapped drive, the user that the service starts as must have setup a drive to match UTL_FILE_DIR and be logged onto the server when UTL_FILE is in use.

❑ Universal Naming Convention: UNC is preferable to Mapped Drives because it does not require anyone to be logged on and UTL_FILE_DIR should be set to a name in the form \\<machine name>\<share name>\<path>.

You will of course need to stop and restart Oracle after changing the properties of the service.

Handling ExceptionsUTL_FILE throws exceptions when it encounters an error. Unfortunately, it uses user-defined exceptions – exceptions it has defined in its package specification. These exceptions, if not caught by name, produce the following less then useful error message:

ERROR at line 1: ORA-06510: PL/SQL: unhandled user-defined exception ORA-06512: at "SYS.UTL_FILE", line 98 ORA-06512: at "SYS.UTL_FILE", line 157

UTL_FILE

1203

This tells you nothing about the error itself. In order to solve this issue, we have to surround our calls to UTL_FILE with an exception block that catches each of the exceptions by name. I prefer to then turn these exceptions into RAISE_APPLICATION_ERROR exceptions. This allows me to assign an ORA- error code and supply a more meaningful error message. We used this in the preceding example to turn the above error message into:

ORA-20001: INVALID_PATH: File location or filename was invalid.

Which is much more useful. The block I always use for this is:

exception when utl_file.invalid_path then raise_application_error(-20001, 'INVALID_PATH: File location or filename was invalid.'); when utl_file.invalid_mode then raise_application_error(-20002, 'INVALID_MODE: The open_mode parameter in FOPEN was invalid.'); when utl_file.invalid_filehandle then raise_application_error(-20002, 'INVALID_FILEHANDLE: The file handle was invalid.'); when utl_file.invalid_operation then raise_application_error(-20003, 'INVALID_OPERATION: The file could not be opened or operated on as requested.'); when utl_file.read_error then raise_application_error(-20004, 'READ_ERROR: An operating system error occurred during the read operation.'); when utl_file.write_error then raise_application_error(-20005, 'WRITE_ERROR: An operating system error occurred during the write operation.'); when utl_file.internal_error then raise_application_error(-20006, 'INTERNAL_ERROR: An unspecified error in PL/SQL.'); end;

I actually keep this in a small file and read it into each routine that uses UTL_FILE to catch the exception and 'rename it' for me.

Dumping a Web Page to Disk This is such a frequently asked question; I thought I would include it here. The scenario is that you are using Oracle WebDB, Oracle Portal, or have some procedures that use the Web Toolkit (the htp packages). You would like to take a report one of these tools is capable of displaying and instead of dynamically generating that report for each and every user – you would like to create a static file with this report every X minutes or hours. This is in fact how I myself generate the home page of the web site I manage at work. Every 5 minutes we regenerate the home page with dynamic data instead of generating each time for the thousands of hits we get during that period of time. Cuts way down on the resources needed to get the home page served up. I do this for frequently accessed, dynamic pages where the underlying data is relatively static (slow changing).

Appendix A

1204

The following procedure is a generic procedure that I use for doing this:

create or replace procedure dump_page( p_dir in varchar2, p_fname in varchar2 ) is l_thePage htp.htbuf_arr; l_output utl_file.file_type; l_lines number default 99999999; begin l_output := utl_file.fopen( p_dir, p_fname, 'w', 32000 );

owa.get_page( l_thePage, l_lines );

for i in 1 .. l_lines loop utl_file.put( l_output, l_thePage(i) ); end loop;

utl_file.fclose( l_output ); end dump_page; /

It is that simple. We only have to open a file, get the HTML page, print each line, and close the file. If I call this after calling a WebDB procedure – it will save the output of that WebDB procedure to the file I name.

The only caveat here is that we will be running the WebDB procedure not from the web but directly. If any of the code in the WebDB procedures uses the CGI environment, then that procedure will fail – since the environment was not set up. We can solve that simply by using a small block of code to setup an environment for our WebDB routine:

declare nm owa.vc_arr; vl owa.vc_arr;begin nm(1) := 'SERVER_PORT'; vl(1) := '80'; owa.init_cgi_env( nm.count, nm, vl ); -- run your webdb procedure here dump_page( 'directory', 'filename' ); end;/

For example, if our WebDB code wanted to verify it was being run from the server on port 80 – we would need to provide the above environment for it. You would add any other environment variables that are relevant to your application in this block

Now all you need to do is refer to the DBMS_JOB section and schedule this block of code to be executed on whatever cycle you need.

UTL_FILE

1205

1023 Byte Limit Once upon a time, there was a 1023 byte limit to UTL_FILE. Each line written to a file could not exceed 1023 bytes. If it did, an exception was raised and UTL_FILE would fail. Fortunately, in Oracle 8.0.5, they introduced an overloaded version of FOPEN that allows us to specify the maximum line length up to 32 KB in size. 32 KB is the largest size a PL/SQL variable can ever be and is typically sufficient for most purposes.

Unfortunately, the documentation has this newly overloaded FOPEN function documented many pages after the original function. This leads to many people overlooking this ability. I still get many questions on this today with version 8.1.7. People did not see the overloaded version of FOPEN; they hit the limit and need to know how to get around it. The answer is simple – but you have to read through every UTL_FILE function to find it easily!

The solution to this particular problem is to use UTL_FILE in the way I did in the previous DUMP_PAGE routine above. The fourth parameter to UTL_FILE.FOPEN is the maximum length of the line of text you would like to produce. In my case above, I allow for up to 32,000 bytes per line.

Reading A Directory This is a missing piece of functionality in the UTL_FILE package. Frequently people want to setup a recurring job that will scan a directory for new input files and process them, perhaps loading the data into the database. Unfortunately, out of the box there is no way for PL/SQL to read a directory listing. We can however use a tiny bit of Java to give us this ability. The following example demonstrates how you might accomplish this.

First I create a user with the minimum set of privileges needs to perform this operation and to be able to list files in the /tmp directory. If you wish to read other directories, you would need to make more calls to dbms_java.grant_permission (see Chapter 19 on Java Stored Procedures for more information) or change the /tmp into * to provide the ability to list all directories.

SQL> connect system/manager

system@DEV816> drop user dirlist cascade; User dropped.

system@DEV816> grant create session, create table, create procedure 2 to dirlist identified by dirlist; Grant succeeded.

system@DEV816> begin 2 dbms_java.grant_permission 3 ('DIRLIST', 4 'java.io.FilePermission', 5 '/tmp', 6 'read' ); 7 end; 8 / PL/SQL procedure successfully completed.

Next, after connecting as this new user DirList, we set up a global temporary table in this schema (to hold the directory listing). This is how we will get the results from the Java stored procedure back to the caller – in the temporary table. We could have used other means (strings, arrays and such) as well.

Appendix A

1206

SQL> connect dirlist/dirlist Connected.

dirlist@DEV816> create global temporary table DIR_LIST 2 ( filename varchar2(255) ) 3 on commit delete rows 4 / Table created.

Now we create a Java stored procedure to do the directory listing. For ease of programming, I am using SQLJ to avoid having to code lots of JDBC calls:

dirlist@DEV816> create or replace 2 and compile java source named "DirList" 3 as 4 import java.io.*; 5 import java.sql.*; 6 7 public class DirList 8 { 9 public static void getList(String directory) 10 throws SQLException 11 { 12 File path = new File( directory ); 13 String[] list = path.list(); 14 String element; 15 16 for(int i = 0; i < list.length; i++) 17 { 18 element = list[i]; 19 #sql { INSERT INTO DIR_LIST (FILENAME) 20 VALUES (:element) }; 21 } 22 } 23 24 } 25 / Java created.

The next step is to create a 'mapping' function, the PL/SQL binding to Java. This will simply be:

dirlist@DEV816> create or replace 2 procedure get_dir_list( p_directory in varchar2 ) 3 as language java 4 name 'DirList.getList( java.lang.String )'; 5 / Procedure created.

Now we are ready to go:

dirlist@DEV816> exec get_dir_list( '\tmp' ); PL/SQL procedure successfully completed.

UTL_FILE

1207

dirlist@DEV816> select * from dir_list where rownum < 5;

FILENAME------------------lost+found.rpc_doorps_data.pcmcia

and that's it. Now we can list the contents of a directory into this temporary table. We can then apply filters to this easily using LIKE and sort the output if we like as well.

Summary UTL_FILE is an excellent utility you will most likely find use for in many of your applications. In this section, we covered the setup necessary for using UTL_FILE and described how it works. We've looked at some of the most common issues I see people running into with UTL_FILE such as accessing network drives in a Windows environment, hitting the 1023 byte limit, and handling exceptions. For each we presented the solutions. We also explored some utilities you might develop with UTL_FILE such as the UNLOADER described in Chapter 9 on Data Loading, the ability to read a directory presented here, or dumping a web page to disk as described above.

Appendix A

1208

UTL_HTTP

In this section, we’ll look at UTL_HTTP and what it can be used for. Additionally, I would like to introduce a new and improved UTL_HTTP built on the SocketType implemented in the UTL_TCPsection. It gives performance comparable to the native UTL_HTTP but provides many more features.

The UTL_HTTP package supplied with the database is relatively simplistic – it has two versions:

❑ UTL_HTTP.REQUEST: returns up to the first 2,000 bytes of the content of a URL as a function return value.

❑ UTL_HTTP.REQUEST_PIECES: returns a PL/SQL table of VARCHAR2(2000) elements. If you concatenated all of the pieces together – you would have the content of the page.

The UTL_HTTP package is missing the following functionality however:

❑ You cannot inspect the HTTP headers. This makes error reporting impossible. You cannot tell the difference between a Not Found and Unauthorized for example.

❑ You cannot POST information to a web server that requires POSTing data. You can only use the GET syntax. Additionally HEAD is not supported in the protocol.

❑ You cannot retrieve binary information using UTL_HTTP.

❑ The request pieces API is non intuitive, the use of CLOBs and BLOBs to return the data as a ‘stream’ would be much more intuitive (and give us access to binary data).

❑ It does not support cookies.

❑ It does not support basic authentication.

❑ It has no methods for URL encoding data.

UTL_HTTP

1209

One thing that UTL_HTTP does support that we will not in our re-implementation is SSL. Using the Oracle Wallet manager, it is possible to perform a HTTPS request (HTTPS is using SSL with HTTP). We will demonstrate using UTL_HTTP over SSL, but will not implement it in our own HTTP_PKG. Due to its size, the source code of the HTTP_PKG body will be omitted from this chapter – it is available in its entirety on the Apress website at http://www.apress.com.

UTL_HTTP Functionality We will look at the UTL_HTTP functionality first as we will support its syntax on our own HTTP_PKG.The simplest form of UTL_HTTP is as follows. In this example, myserver is the name I have given to the web server. Of course, you should try this example using a web server to which you have access:

ops$tkyte@DEV816> select utl_http.request( ‘http://myserver/’ ) from dual;

UTL_HTTP.REQUEST(‘HTTP://MYSERVER/’) -----------------------------------------------------------<HTML><HEAD><TITLE>Oracle Service Industries</TITLE> </HEAD><FRAMESET COLS=“130,*” border=0> <FRAME SRC=“navtest.html” NAME=“sidebar” frameborder=0> <FRAME SRC=“folder_home.html” NAME=“body” frameborder=“0” marginheight=“0” marginwidth=“0”></FRAMESET></BODY></HTML>

I can simply run UTL_HTTP.REQUEST and send it a URL. UTL_HTTP will connect to that web server and GET the requested page and then return the first 2,000 characters of it. As mentioned above, don’t try to use the URL I have given above, it is for my web server inside of Oracle. You won’t be able to get to it, the request will time out with an error message.

Most networks today are protected by firewalls. If the page I wanted was only available via a firewall proxy server; I can request that as well. A discussion of firewalls and proxy servers is beyond the scope of this book. However, if you know the hostname of your proxy server, you can retrieve a page from the internet via this method:

ops$tkyte@DEV816> select utl_http.request( ‘http://www.yahoo.com’, ‘www-proxy’) from dual;

UTL_HTTP.REQUEST(‘HTTP://WWW.YAHOO.COM’,’WWW-PROXY’) -------------------------------------------------------------------<html><head><title>Yahoo!</title><base href=http://www.yahoo.com/><meta http-equiv=“PICS-Label” content=‘(PICS-1.1 http://www.rsac.org/ratingsv01.html” l gen true for “http://www.yahoo.com” r (n 0 s 0 v 0 l 0))’></head><body><center><form action=http://search.yahoo.com/bin/search><map name=m><area coords=“0,0,52,52” href=r/a1><area coords=“53,0,121,52” href=r/p1><area coords=“122,0,191,52” href=r

The second parameter to UTL_HTTP.REQUEST and REQUEST_PIECES is the name of a proxy server. If your proxy server is not running on the standard port 80, we can add the port as follows (In the code this is myserver on port 8000):

Appendix A

1210

ops$tkyte@DEV816> select utl_http.request( ‘http://www.yahoo.com’, 2 ‘myserver:8000’ ) from dual 3 /

UTL_HTTP.REQUEST(‘HTTP://WWW.YAHOO.COM’,’MYSERVER:8000’) -------------------------------------------------------<html><head><title>Yahoo!</title><base href=http://www.yahoo.com/

So, by simply adding the :8000 to the proxy server name, we are able to connect to that proxy server. Now, let’s look at the REQUEST_PIECES interface:

ops$tkyte@DEV816> declare 2 pieces utl_http.html_pieces; 3 n number default 0; 4 l_start number default dbms_utility.get_time; 5 begin 6 pieces := 7 utl_http.request_pieces( url => ‘http://www.oracle.com/’, 8 max_pieces => 99999, 9 proxy => ‘www-proxy’ ); 10 for i in 1 .. pieces.count 11 loop 12 loop 13 exit when pieces(i) is null; 14 dbms_output.put_line( substr(pieces(i),1,255) ); 15 pieces(i) := substr( pieces(i), 256 ); 16 end loop; 17 end loop; 18 end; 19 /

<head><title>Oracle Corporation</title> <meta http-equiv=“Content-Type” content=“text/html; charset=iso-8859-1”><meta name=“description” content=“Oracle Corporation provides the software that powers the Internet. For more information about Oracle, pleas e call 650/506-7000.”>

The request pieces API cannot be called from SQL since it does not return a SQL type but rather a PL/SQL table type. So, REQUEST_PIECES is only useful inside of a PL/SQL block itself. In the above, we are requesting the web page http://www.oracle.com/ and we are requesting the first 99,999 chunks, which are each of the size 2,000 bytes. We are using the proxy server www-proxy. We must tell REQUEST_PIECES how many 2,000 byte chunks we are willing to accept, typically I set this to a very large number as I want the entire page back. If the information that you want from the page is always in the first 5000 bytes, you could request just 3 chunks to get it.

Adding SSL to UTL_HTTP UTL_HTTP also supports using SSL (Secure Sockets Layer). If you are not familiar with SSL and what it is used for, you can find a brief description at http://www.rsasecurity.com/rsalabs/faq/5-1-2.html. Both

UTL_HTTP

1211

the REQUEST and REQUEST_PIECES functions in UTL_HTTP support the retrieval of URLs that are protected by SSL. However, the available documentation on doing so can be described as sparse at best. SSL support is provided by using the last two parameters in the UTL_HTTP.REQUEST and UTL_HTTP.REQUEST_PIECES procedures. These parameters are the WALLET_PATH and WALLET_PASSWORD.

Oracle uses the wallet as a metaphor for how a person stores their security credentials; just like you would keep your driver’s license and credit cards in your wallet for identification purposes, the Oracle wallet stores the credentials needed by the SSL protocol. The WALLET_PATH is the directory where your wallet is stored on the database server machine. This wallet is password protected to prevent someone from using your credentials. This is the purpose of the WALLET_PASSWORD parameter, it is used to access the wallet. The password prevents people from copying the wallet directory and trying to impersonate you, as they will be unable to open and access the wallet. This is analogous to using a PIN for using an ATM machine. If someone steals your bank card, they need to have your PIN to get to your accounts.

The wallet, or the concept of a wallet is used not only by the Oracle database, but also in Web browsers. The important aspect is that when you connect to a site, e.g., http://www.amazon.com, how do you know that it is really Amazon.com? You have to get their certificate, which is digitally signed by someone. That someone is called a Certificate Authority or CA. How does my browser or database know to trust the CA that signed that certificate? For example, I could create a certificate for Amazon.com and sign it from hackerAttackers.com. My browser and database should not accept this certificate even though it is a legitimate X.509 certificate.

The answer to this trust issue is the wallet stores a set of trusted certificates. A trusted certificate is a certificate from a CA that you trust. The Oracle wallet comes with some common trusted certificates. You also have the ability to add certificates as necessary. Your browser does the same thing. If you ever connect to a site where your browser does not have the CA in its wallet, you will get a pop-up window that notifies you of this as well as a wizard that allows you to proceed or abort your connection.

Let’s see some examples of how to use SSL. First, we need to create a new wallet. You can invoke the OWM (Oracle Wallet Manager) program on UNIX, or launch it from the Windows START menu on Windows(it is in ORACLE HOME|NETWORK ADMINISTRATION). The screen you receive to do this will look like this:

Appendix A

1212

All you need to do is click on the NEW icon (the green ‘cube’) located on the left hand side of the display. It will prompt you for a password for this wallet, you are making up the password at this point so enter whatever password you would like. You may get a warning about a directory not existing, if you do, you should simply ignore it. It is the expected behavior if you have never created a wallet before. OWM at will then ask you to create a certificate request:

you do not need to this. The certificate request is so you can get a certificate for yourself. This would be used in SSL v.3 where the server needs the identification of the client. Most Web sites do not authenticate users via certificates, but rather by using a username and password. This is because an e - commerce site doesn’t care who is buying from them as long as they get the money. But you do care that you are sending the money (and credit card information) to the correct entity, so we use SSL v.2 to identify the server for example, Amazon.com, and to provide all the encryption of data. So, click NO in response to this and save the wallet by clicking on the SAVE WALLET (the yellow floppy disk) icon and we are ready to go.

Let’s go to Amazon.com first. Amazon’s certificate was signed by Secure Server Certificate Authority, RSA Data Security, Inc. This is one of the defaults in the Oracle wallet.

tkyte@TKYTE816> declare 2 l_output long; 3 4 l_url varchar2(255) default 5 ‘https://www.amazon.com/exec/obidos/flex-sign-in/’; 6 7 l_wallet_path varchar2(255) default 8 ‘file:C:\Documents and Settings\Thomas Kyte\ORACLE\WALLETS’; 9 10 11 begin 12 l_output := utl_http.request 13 ( url => l_url, 14 proxy => ‘www-proxy.us.oracle.com’, 15 wallet_path => l_wallet_path, 16 wallet_password => ‘oracle’ 17 ); 18 dbms_output.put_line(trim(substr(l_output,1,255))); 19 end; 20 /

<html><head><title>Amazon.com Error Page</title> </head><body bgcolor=“#FFFFFF” link=“#003399” alink=“#FF9933” vlink=“#996633” text=“#000000”>

UTL_HTTP

1213

<a name=“top”></a><table border=0 width=100% cellspacing=0 cellpadding=0> <tr


Don’t worry about getting this error page; this is accurate. The reason for receiving this error page is that there is no session information being passed. We are just testing that the connection worked here; we retrieved an SSL protected document.

Let’s try another site. How about E*Trade?

tkyte@TKYTE816> declare 2 l_output long; 3 4 l_url varchar2(255) default 5 ‘https://trading.etrade.com/’;

6 7 l_wallet_path varchar2(255) default 8 ‘file:C:\Documents and Settings\Thomas Kyte\ORACLE\WALLETS’;

9

10

11 begin 12 l_output := utl_http.request 13 ( url => l_url, 14 proxy => ‘www-proxy.us.oracle.com’, 15 wallet_path => l_wallet_path, 16 wallet_password => ‘oracle’ 17 ); 18 dbms_output.put_line(trim(substr(l_output,1,255))); 19 end;

20 /

declare

*

ERROR at line 1:

ORA-06510: PL/SQL: unhandled user-defined exception ORA-06512: at “SYS.UTL_HTTP”, line 174 ORA-06512: at line 12

That apparently does not work. E*Trade has a certificate signed by. www.verisign com/CPS Incorp.by Ref, which is not a default trusted certificate. In order to access this page, we’ll have to add that certificate to our Oracle wallet – assuming of course that we trust Verisign! Here is the trick. Go to the site (https://trading.etrade.com). Double-click the PADLOCK icon on the bottom right corner of the window (in Microsoft Internet Explorer). This will pop-up a window that looks similar to this one:

Appendix A

1214

Select the Certification Path tab on the top of this screen. This lists the certificate you are viewing here it is the one for E*Trade (trading.etrade.com), as well as who issued the certificate. We need to add the person who signed the certificate (the issuer) to our trusted certificates in the Oracle wallet. The issuer is www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD as depicted by the tree-like hierarchy.

Click the View Certificate button while www.verisign.com/CPS Incorp. by Ref. is highlighted. This shows information for the issuer’s certificate. Click the Details tab and you should see:

UTL_HTTP

1215

Now we need to Click the Copy to File button. Save the file locally as a Base-64 encoded X.509 (CER) file. The following screen shows the selection you should make, you can name the file anything you choose and save it anywhere. We’ll be importing it in a moment, just remember where you save it to:

Now we can import this into our Oracle Wallet. Open the wallet in OWM and right click on the Trusted Certificates – this will present you with a popup menu that has Import Trusted Certificate:

Appendix A

1216

You will select that option and in the next dialog that comes up, choose Select a file that contains the certificate

Use the standard ‘file open’ dialog that comes up to choose the certificate you just saved. Your screen should now look something like this:

Now, save the wallet using clicking on the SAVE WALLET (the yellow floppy disk) icon and let’s try our example again:

tkyte@TKYTE816> declare 2 l_output long; 3 4 l_url varchar2(255) default 5 ‘https://trading.etrade.com/cgi-bin/gx.cgi/AppLogic%2bHome’; 6 7 l_wallet_path varchar2(255) default 8 ‘file:C:\Documents and Settings\Thomas Kyte\ORACLE\WALLETS’; 9 10 11 begin 12 l_output := utl_http.request 13 ( url => l_url, 14 proxy => ‘www-proxy.us.oracle.com’, 15 wallet_path => l_wallet_path, 16 wallet_password => ‘oracle’ 17 ); 18 dbms_output.put_line(trim(substr(l_output,1,255))); 19 end; 20 /

UTL_HTTP

1217

<HTML><HEAD><META http-equiv=“Content-Type” content=“text/html; charset=ISO-8859-1”><TITLE>E*TRADE</TITLE><SCRIPT LANGUAGE=“Javascript” TYPE=“text/javascript”> <!--

function mac_comment(){ var agt=navigator.userAgent.toLowerCase(); var is_mac


This time we are successful. Now we know how to use and extend the Oracle wallet to do secure HTTPS.

Really Using UTL_HTTP Well, besides getting the content of a web page which is extremely useful – what else can we do with UTL_HTTP? Well, a common use for it is an easy way to make it so that PL/SQL can run a program – sort of like a HOST command. Since almost every web server can run cgi-bin programs and UTL_HTTPcan send URLs, we can in effect have PL/SQL execute host commands by configuring the commands we wanted to execute as cgi-bin programs under the webserver.

What I like to do in this case is setup a web server running on IP address 127.0.0.1 – which is the TCP/IP loop back. This IP address is only accessible if you are physically logged onto the machine the web server is running on. In that fashion, I can set up cgi-bin programs for my PL/SQL programs to run that no one else can – unless they break in to my server machine in which case I have much larger problems to deal with.

One use I’ve made of this facility in the past is to send e-mail from PL/SQL. Let’s say you have the Oracle 8i database without Java in it. Without Java you cannot use either UTL_TCP or UTL_SMTP – they both rely on the Java option in the database. So, without UTL_SMTP and/or UTL_TCP, how could we send an email? With UTL_HTTP and UTL_FILE – I can set up a cgi-bin program that receives a single input in the QUERY_STRING environment variable. The single input would be a file name. We will use /usr/lib/sendmail to send that file (on Windows I could use the public domain utility ‘blat’ to send mail, available from http://www.interlog.com/~tcharron/blat.html). Once I have that set up, I can simply run my host command via:

… results := utl_http.request( ‘http://127.0.0.1/cgi-bin/smail?filename.text’ ); …

The cgi-bin program I set up, smail, would return a ‘web page’ that indicated success or failure, I would look at the result’s variable to see if the mail was sent successfully or not. The full fledged implementation of this on Unix could be:

Appendix A

1218

[email protected]> create sequence sm_seq 2 / Sequence created.

[email protected]> create or replace procedure sm( p_to in varchar2, 2 p_from in varchar2, 3 p_subject in varchar2, 4 p_body in varchar2 ) 5 is 6 l_output utl_file.file_type; 7 l_filename varchar2(255); 8 l_request varchar2(2000); 9 begin 10 select ‘m’ || sm_seq.nextval || ‘.EMAIL.’ || p_to 11 into l_filename 12 from dual; 13 14 l_output := utl_file.fopen 15 ( ‘/tmp’, l_filename, ‘w’, 32000 ); 16 17 utl_file.put_line( l_output, ‘From: ‘ || p_from ); 18 utl_file.put_line( l_output, ‘Subject: ‘ || p_subject ); 19 utl_file.new_line( l_output ); 20 utl_file.put_line( l_output, p_body ); 21 utl_file.new_line( l_output ); 22 utl_file.put_line( l_output, ‘.’ ); 23 24 utl_file.fclose( l_output ); 25 26 l_request := utl_http.request 27 ( ‘http://127.0.0.1/cgi-bin/smail?’ || l_filename ); 28 29 dbms_output.put_line( l_request ); 30 end sm; 31 /

Procedure created.

You should refer to the section on UTL_SMTP to understand why I formatted the email as I did here with the From: and To: header records. In this routine we are using a sequence to generate a unique filename. We encode the recipient in the name of the file itself. Then, we write the email to an OS file. Lastly, we use UTL_HTTP to run our host command and pass it the name of the file. We simply print out the result of this in this test case – we would really be inspecting the value of l_result to ensure the e-mail was sent successfully.

The simple cgi-bin program could be:

#!/bin/sh

echo “Content-type: text/plain” echo ““

echo $QUERY_STRINGto=`echo $QUERY_STRING|sed ‘s/.*EMAIL\.//’` echo $to

(/usr/lib/sendmail $to < /tmp/$QUERY_STRING) 1>> /tmp/$$.log 2>&1 cat /tmp/$$.log rm /tmp/$$.log rm /tmp/$QUERY_STRING

UTL_HTTP

1219

The shell script starts by printing out the HTTP headers that we need in order to return a document; that is what the first two echoes are doing. Then, we are simply printing out the value of the QUERY_STRING (this is where the web server puts our inputs – the portion after the ? in the URL). We then extract the email address from the QUERY_STRING by using the sed (Stream EDitor) to remove everything in front of the word EMAIL/ in the filename. We then run sendmail in a subshell so as to be able to capture both its stdout and stderr output streams. I cat (type to stdout) the contents of the log we captured from sendmail. In this case I wanted both stdout and stderr to be returned as the contents of the web page, so the PL/SQL code could get any error messages and such. The easiest way to do that was to redirect both output streams to a temporary file and then type that file to stdout. We then clean up our temp files and return.

I can now test this via:

[email protected]> begin 2 sm( ‘[email protected]’, 3 ‘[email protected]’, 4 ‘testing’, 5 ‘hello world!’ ); 6 end; 7 / [email protected]@us.oracle.com

This shows us the QUERY_STRING and the To: environment variables, and since there is no other test (no error messages) we know the e-mail was sent.

So, this small example shows how UTL_HTTP can be used indirectly for something it was not really designed for; to allow PL/SQL to run HOST commands. You would set up a special purpose web server on IP Address 127.0.0.1, configure a cgi-bin type directory within it and place the commands you want PL/SQL developers to run in there. Then you have a secure way to allow PL/SQL to run the equivalent of SQL*PLUS’s host command.

A Better UTL_HTTP Given that we have the SocketType class developed in the UTL_TCP section (or just access to UTL_TCPalone) and knowledge of the HTTP protocol, we can make an improved UTL_HTTP package. We will call our implementation HTTP_PKG. It will support the “old fashioned” UTL_HTTP interface of REQUESTand REQUEST_PIECES but will also add support for:

❑ Getting the HTTP headers back with each request – These headers contain useful information such as the status of the request (e.g. 200 OK, 404 Not Found, and so on), the name of the server that executed the URL, the content type of the returned content, cookies and so on.

❑ Getting the content back as either a CLOB or a BLOB – This allows your PL/SQL to retrieve a large PDF file to be inserted into a table indexed by interMedia as well as retrieve plain text from another page or just to access any binary data returned by a web server.

❑ Perform a HEAD of a document – This is useful to check and see if the document you retrieved last week has been updated for example.

❑ URL-encoded strings – for example, if you have a space or a tilde (~) in a URL request, they must be ‘escaped’. This function escapes all characters that need to be.

Appendix A

1220

❑ Sending Cookies with requests – If you are using this HTTP_PKG to get access to a web site that uses a cookie based authentication scheme, this is crucial. You would have to GET the login page, sending the username and password in the GET request. You would then look at the HTTP headers returned by that page and extract their cookie. This cookie value is what you need to send on all subsequent requests to prove who you are.

❑ POST data instead of just GET – The GET protocol has limits that vary by web server on the size of the request. Typically, a URL should not exceed 1 to 2 KB in length. If it does, you should POST the data. POSTed data can be of unlimited size.

We can implement all of this in PL/SQL using our SocketType from the UTL_TCP section and we can do it with a fairly small amount of code. The specification for our new HTTP_PKG package follows. We’ll look at the specification and some examples that use it. What will not be in this book is the code that implements the body of the HTTP_PKG. This code is available and documented on the Apress website, (http://www.apress.com), it is about 15 printed pages long hence it is not included here. The package specification is as follows. The first two functions, REQUEST and REQUEST_PIECES, are functionally equivalent (minus SSL support) to functions found in the UTL_HTTP package in versions 7.3.x, 8.0.x, and 8.1.5, we even will raise the same sort of named exceptions they would:

tkyte@TKYTE816> create or replace package http_pkg 2 as 3 function request( url in varchar2, 4 proxy in varchar2 default NULL ) 5 return varchar2; 6 7 type html_pieces is table of varchar2(2000) 8 index by binary_integer; 9 10 function request_pieces(url in varchar2, 11 max_pieces natural default 32767, 12 proxy in varchar2 default NULL) 13 return html_pieces; 14 15 init_failed exception; 16 request_failed exception;

The next procedure is GET_URL. It invokes the standard HTTP GET command on a web server. The inputs are:

❑ p_url is the URL to retrieve.

❑ p_proxy is the name:<port> of the proxy server to use. Null indicates you need not use a proxy server. Examples: p_proxy => ‘www-proxy’ or p_proxy => ‘www-proxy:80’.

❑ p_status is returned to you. it will be the HTTP status code returned by the web server. 200indicates normal, successful completion. 401 indicates unauthorized, and so on.

❑ p_status_txt is returned to you. It contains the full text of the HTTP status record. For example it might contain: HTTP/1.0 200 OK.

UTL_HTTP

1221

�� p_httpHeaders may be set by you and upon return will contain the http headers from the requested URL. On input, any values you have set will be transmitted to the web server as part of the request. On output, the headers generated by the web server will be returned to you. You can use this to set and send cookies or basic authentication or any other http header record you wish.

�� p_content is a temporary CLOB or BLOB (depending on which overloaded procedure you call) that will be allocated for you in this package (you need not allocate it). It is a session temporary LOB. You may use dbms_lob.freetemporary to deallocate it whenever you want, or just let it disappear when you log out.

19 procedure get_url( p_url in varchar2, 20 p_proxy in varchar2 default NULL, 21 p_status out number, 22 p_status_txt out varchar2, 23 p_httpHeaders in out CorrelatedArray, 24 p_content in out clob ); 25 26 27 procedure get_url( p_url in varchar2, 28 p_proxy in varchar2 default NULL, 29 p_status out number, 30 p_status_txt out varchar2, 31 p_httpHeaders in out CorrelatedArray, 32 p_content in out blob );

The next procedure is HEAD_URL. Invokes the standard HTTP HEAD syntax on a web server. The inputs and outputs are identical to get_url above (except no content is retrieved). This function is useful to see if a document exists, what its mime-type is, or if it has been recently changed, without actually retrieving the document itself:

34 procedure head_url( p_url in varchar2, 35 p_proxy in varchar2 default NULL, 36 p_status out number, 37 p_status_txt out varchar2, 38 p_httpHeaders out CorrelatedArray );

The next function URL encode is used when building GET parameter lists or building POST CLOBs. It is used to escape special characters in URLs (for example, a URL may not contain a whitespace, a % sign, and so on). Given input such as Hello World, then urlencode will return Hello%20World’

40 function urlencode( p_str in varchar2 ) return varchar2;

Procedure Add_A_Cookie allows you to easily set a cookie value to be sent to a web server. You need only know the name and value of the cookie. The formatting of the HTTP header record is performed by this routine. The p_httpHeaders variable you send in/out of this routine would be sent in/out of the <Get|Head|Post>_url routines:

42 procedure add_a_cookie 43 ( p_name in varchar2, 44 p_value in varchar2, 45 p_httpHeaders in out CorrelatedArray );

Appendix A

1222

The next procedure Set_Basic_Auth allows you to enter a username/password to access a protected page. The formatting of the HTTP header record is performed by this routine. The p_httpHeadersvariable you send in/out of this routine would be sent in/out of the <Get|Head|Post>_url routines as well:

47 procedure set_basic_auth 48 ( p_username in varchar2, 49 p_password in varchar2, 50 p_httpHeaders in out CorrelatedArray );

The procedure set_post_parameter is used when retrieving a URL that needs a large (greater than 2,000 or so bytes) set of inputs. It is recommended that the POST method be used for large requests. This routine allows you to add parameter after parameter to a POST request. This post request is built into a CLOB which you supply:

52 procedure set_post_parameter 53 ( p_name in varchar2, 54 p_value in varchar2, 55 p_post_data in out clob, 56 p_urlencode in boolean default FALSE );

The next two routines are identical to GET_URL above with the addition of the p_post_data input p_post_data is a CLOB built by repeated calls to set_post_parameter above. The remaining inputs/outputs are defined the same as they were for GET_URL:

58 procedure post_url 59 (p_url in varchar2, 60 p_post_data in clob, 61 p_proxy in varchar2 default NULL, 62 p_status out number, 63 p_status_txt out varchar2, 64 p_httpHeaders in out CorrelatedArray, 65 p_content in out clob ); 66 67 procedure post_url 68 (p_url in varchar2, 69 p_post_data in clob, 70 p_proxy in varchar2 default NULL, 71 p_status out number, 72 p_status_txt out varchar2, 73 p_httpHeaders in out CorrelatedArray, 74 p_content in out blob ); 75 76 77 end; 78 /

Package created.

So, the specification of the package is done; the procedures defined within it are rather straightforward. I can do things like GET_URL to get a URL. This will use the HTTP GET syntax to retrieve the contents of a web page into a temporary BLOB or CLOB. I can HEAD_URL to get the headers for a URL. Using this I could look at the mime type for example to decide if I wanted to use a CLOB (text/html) to get the

UTL_HTTP

1223

URL or a BLOB (image/gif). I can even POST_URL to post large amounts of data to a URL. There are the other helper functions to set cookies in the header, to base 64 encode the username and password for basic authentication and so on.

Now, assuming you have downloaded the implementation of HTTP_PKG (the package body consists of about 500 lines of PL/SQL code) we are ready to try it out. We’ll introduce a pair of utility routines that will be useful to test with first. Print_clob below simply prints out the entire contents of a CLOB using DBMS_OUTPUT. Print_Headers does the same for the HTTP headers we retrieve above in our CorrelatedArray type (an object type that is part of the HTTP_PKG implementation). In the following, the procedure P is the P procedure I introduced in the DBMS_OUTPUT section to print long lines:

ops$tkyte@DEV816> create or replace procedure print_clob( p_clob in clob ) 2 as 3 l_offset number default 1; 4 begin 5 loop 6 exit when l_offset > dbms_lob.getlength(p_clob); 7 dbms_output.put_line( dbms_lob.substr( p_clob, 255, l_offset ) ); 8 l_offset := l_offset + 255; 9 end loop; 10 end; 11 /

ops$tkyte@DEV816> create or replace 2 procedure print_headers( p_httpHeaders correlatedArray ) 3 as 4 begin 5 for i in 1 .. p_httpHeaders.vals.count loop 6 p( initcap( p_httpHeaders.vals(i).name ) || ‘: ‘ || 7 p_httpHeaders.vals(i).value ); 8 end loop; 9 p( chr(9) ); 10 end; 11 /

Now onto the test:

ops$tkyte@DEV816> begin 2 p( http_pkg.request( ‘http://myserver/’ ) ); 3 end; 4 / <HTML><HEAD><TITLE>Oracle Service Industries</TITLE> </HEAD><FRAMESET COLS=“130,*” border=0><FRAME SRC=“navtest.html” NAME=“sidebar” frameborder=0> <FRAME SRC=“folder_home.html” NAME=“body” frameborder=“0” marginheight=“0” marginwidth=“0”> </FRAMESET>

</BODY></HTML>

Appendix A

1224

ops$tkyte@DEV816> declare 2 pieces http_pkg.html_pieces; 3 begin 4 pieces := 5 http_pkg.request_pieces( ‘http://www.oracle.com’, 6 proxy=>‘www-proxy1’); 7 8 for i in 1 .. pieces.count loop 9 p( pieces(i) ); 10 end loop; 11 end; 12 / <head><title>Oracle Corporation</title> <meta http-equiv=“Content-Type” content=“text/html; …

The above two routines simply show that the UTL_HTTP methods of REQUEST and REQUEST_PIECESfunction as expected with our new package. Their functionality is identical. Now we will invoke our URLENCODE function that translates ‘bad characters into escape sequences in URLs and POST data:

ops$tkyte@DEV816> select 2 http_pkg.urlencode( ‘A>C%{hello}\fadfasdfads~`[abc]:=$+’’”’ ) 3 from dual;

HTTP_PKG.URLENCODE(‘A>C%{HELLO}\FADFASDFADS~`[ABC]:=$+’’”’) ---------------------------------------------------------------A%3EC%25%7Bhello%7D%5Cfadfasdfads%7E%60%5Babc%5D%3A%3D%24%2B%27%22

That shows that characters like > and % are escaped into %3E and %25 respectively and other sequences such as the word hello are not escaped. This allows us to use any of these special characters in our HTTP requests safely.

Now we will see the first of the new HTTP URL procedures. This procedure call will return Yahoo’s home page via a proxy server, www-proxy1, (you’ll need to replace this with your own proxy server of course). Additionally, we get to see the HTTP status returned – 200 indicates success. We also see the HTTP headers Yahoo returned to us. The mime-type will always be in there and that tells us what type of content we can expect. Lastly, the content is returned and printed out:

ops$tkyte@DEV816> declare 2 l_httpHeaders correlatedArray; 3 l_status number; 4 l_status_txt varchar2(255); 5 l_content clob; 6 begin 7 http_pkg.get_url( ‘http://www.yahoo.com/’, 8 ‘www-proxy1’, 9 l_status, 10 l_status_txt, 11 l_httpHeaders, 12 l_content ); 13 14 p( ‘The status was ‘ || l_status );

UTL_HTTP

1225

15 p( ‘The status text was ‘ || l_status_txt ); 16 print_headers( l_httpHeaders ); 17 print_clob( l_content ); 18 end; 19 / The status was 200 The status text was HTTP/1.0 200 OK

Date: Fri, 02 Feb 2001 19:13:26 GMT Connection: close Content-Type: text/html

<html><head><title>Yahoo!</title><base href=http://www.yahoo.com/><meta http-equiv=“PICS-Label”

Next, we will try the HEAD request against the home page of a sample site, let’s call it, Sample.com, and see what we can discover:

ops$tkyte@DEV816> declare 2 l_httpHeaders correlatedArray; 3 l_status number; 4 l_status_txt varchar2(255); 5 begin 6 http_pkg.head_url( ‘http://www.sample.com/’, 7 ‘www-proxy1’, 8 l_status, 9 l_status_txt, 10 l_httpHeaders ); 11 12 p( ‘The status was ‘ || l_status ); 13 p( ‘The status text was ‘ || l_status_txt ); 14 print_headers( l_httpHeaders ); 15 end; 16 / The status was 200 The status text was HTTP/1.1 200 OK

Server: Microsoft-IIS/5.0 Date: Fri, 02 Feb 2001 19:13:26 GMT Connection: Keep-Alive Content-Length: 1270 Content-Type: text/html Set-Cookie: ASPSESSIONIDQQQGGNQU=PNMNCIBACGKFLHGKLLBPEPMD; path=/ Cache-Control: private

From the headers, it is obvious that Sample.com is running Windows with Microsoft IIS. Further, they are using ASP’s as indicated by the cookie they sent back to us. If we were to have retrieved that page, it would have had 1,270 bytes of content.

Now we would like to see how cookies might work. Here I am using a standard procedure that is used with OAS (the Oracle Application Server) and iAS (Oracle’s Internet Application Server); the cookiejar sample that shows how to use cookies in a PL/SQL web procedure. The routine cookiejar looks at the cookie value and if set, increments it by one and returns it to the client. We’ll see what how that would work using our package. We are going to send the value 55 to the server and we are expecting it to send us 56 back:

Appendix A

1226

ops$tkyte@DEV816> declare 2 l_httpHeaders correlatedArray; 3 l_status number; 4 l_status_txt varchar2(255); 5 l_content clob; 6 begin 7 http_pkg.add_a_cookie( ‘COUNT’, 55, l_httpHeaders ); 8 http_pkg.get_url 9 ( ‘http://myserver.acme.com/wa/webdemo/owa/cookiejar’,

10 null, 11 l_status, 12 l_status_txt, 13 l_httpHeaders, 14 l_content ); 15 16 p( ‘The status was ‘ || l_status ); 17 p( ‘The status text was ‘ || l_status_txt ); 18 print_headers( l_httpHeaders ); 19 print_clob( l_content ); 20 end;

21 /

The status was 200

The status text was HTTP/1.0 200 OK

Content-Type: text/html

Date: Fri, 02 Feb 2001 19:14:48 GMT

Allow: GET, HEAD Server: Oracle_Web_listener2.1/1.20in2 Set-Cookie: COUNT=56; expires=Saturday, 03-Feb-2001 22:14:48 GMT

<HTML>

<HEAD>

<TITLE>C is for Cookie</TITLE>

</HEAD>

<BODY>

<HR>

<IMG SRC=“/ows-img/ows.gif”>

<H1>C

is for Cookie</H1>

<HR>

You have visited this page <STRONG>56</STRONG> times in the last 24 hours.

…

As you can see, the cookie value of 55 was transmitted and the server incremented it to 56. It then sent us back the modified value along with an expiration date.

Next, we would like to see how to access a page that requires a username and password. This is done via the following:

UTL_HTTP

1227

ops$tkyte@DEV816> declare 2 l_httpHeaders correlatedArray; 3 l_status number; 4 l_status_txt varchar2(255); 5 l_content clob; 6 begin 7 http_pkg.set_basic_auth( ‘tkyte’, ‘tiger’, l_httpheaders ); 8 http_pkg.get_url 9 ( ‘http://myserver.acme.com:80/wa/intranets/owa/print_user’, 10 null, 11 l_status, 12 l_status_txt, 13 l_httpHeaders, 14 l_content ); 15 16 p( ‘The status was ‘ || l_status ); 17 p( ‘The status text was ‘ || l_status_txt ); 18 print_headers( l_httpHeaders ); 19 print_clob(l_content); 20 end; 21 / The status was 200 The status text was HTTP/1.0 200 OK

Content-Type: text/html Date: Fri, 02 Feb 2001 19:49:17 GMT Allow: GET, HEAD Server: Oracle_Web_listener2.1/1.20in2

remote user = tkyte

Here, I just set up a DAD (Database Access Descriptor) that did not store the username/password with the DAD. This means the web server is expecting the request to contain the username/password to use. Here I passed my credentials to a routine that simply printed out the REMOTE_USER cgi-environment variable in PL/SQL (the name of the remotely connected user).

Lastly, we would like to demonstrate the POST’ing of data. Here I am using a URL from Yahoo again. Yahoo makes it easy to get stock quotes in a spreadsheet format. Since the list of stock symbols you might be interested in could get quite large, I would suggest POSTing this data. Here is an example that gets a couple of stock quotes from Yahoo using HTTP. The data will be returned in CSV (Comma Separated Values) for easy parsing and loading into a table for example:

ops$tkyte@DEV816> declare 2 l_httpHeaders correlatedArray; 3 l_status number; 4 l_status_txt varchar2(255); 5 l_content clob; 6 l_post clob; 7 begin 8 http_pkg.set_post_parameter( ‘symbols’,’orcl ^IXID ^DJI ^SPC’, 9 l_post, TRUE ); 10 http_pkg.set_post_parameter( ‘format’, ‘sl1d1t1c1ohgv’, 11 l_post, TRUE ); 12 http_pkg.set_post_parameter( ‘ext’, ‘.csv’,

Appendix A

1228

13 l_post, TRUE ); 14 http_pkg.post_url( ‘http://quote.yahoo.com/download/quotes.csv’, 15 l_post, 16 ‘www-proxy’, 17 l_status, 18 l_status_txt, 19 l_httpHeaders, 20 l_content ); 21 22 p( ‘The status was ‘ || l_status ); 23 p( ‘The status text was ‘ || l_status_txt ); 24 print_headers( l_httpHeaders ); 25 print_clob( l_content ); 26 end; 27 / The status was 200 The status text was HTTP/1.0 200 OK

Date: Fri, 02 Feb 2001 19:49:18 GMT Cache-Control: private Connection: close Content-Type: application/octet-stream

“ORCL”,28.1875,”2/2/2001”,”2:34PM”,-1.875,29.9375,30.0625,28.0625,26479100“^IXID”,1620.60,”2/2/2001”,”2:49PM”,-45.21,1664.55,1667.46,1620.40,N/A“^DJI”,10899.33,”2/2/2001”,”2:49PM”,-84.30,10982.71,11022.78,10888.01,N/A“^SPC”,1355.17,”2/2/2001”,”2:49PM”,-18.30,1373.53,1376.16,1354.21,N/A

Summary In this section, we have seen how to use the built-in UTL_HTTP package. We have seen how through a little creative thinking, we can use UTL_HTTP not only to grab data from the web, but to enable PL/SQL to have the equivalent of a HOST command. With just a couple of lines of code, we made it possible that any release of Oracle from 7.3 on up can easily send mail using UTL_FILE and UTL_HTTP.

We then investigated how to use UTL_HTTP over SSL – a concept not well documented in the Oracle Supplied Packages Guide (or any other document for that matter). We learned how to make any SSL enabled website available to our PL/SQL routines using the Oracle Wallet Manager.

Additionally, we’ve seen how we can take a good idea and make it better. In comparison to the section on DBMS_OBFUSCATION_TOOLKIT where we ‘wrapped’ the functionality of that package to make it easier and more flexible to use, here we totally re-implemented a package giving it additionalfunctionality it never had. This came at a certain price, we do not support SSL in our implementation, but it is useful in many cases nonetheless.

We could easily take this a step further and user either Java or a C based External Procedure to add full support for SSL as well. There are also various third party and public domain classes and libraries out there to do just that.

UTL_RAW

1229

UTL_RAW

UTL_RAW is a package supplied with Oracle since version 7.1.6. It is a utility package developed by the Procedural Gateway development team initially for accessing and converting mainframe data into ASCII, and then later by the replication development team. It contains four functions I use frequently, and you've seen these scattered throughout the book. I am going to cover these four functions only, as they are the ones I find to be of most use. There are other functions contained in the package (thirteen more to be exact), but I won't be covering them here. Check the Supplied PL/SQL Packages Reference for more details on these.

The four functions I will cover are:

❑ CAST_TO_VARCHAR2 – Converts a RAW to a VARCHAR2

❑ CAST_TO_RAW – Converts a VARCHAR2 to a RAW

❑ LENGTH – Returns the length of a RAW variable

❑ SUBSTR – Returns a substring of a RAW variable

We use these functions heavily when dealing with binary data. This can be seen in the CRYPT_PKG we use in the sections on DBMS_OBFUSCATION_TOOLKIT, DBMS_LOB, and UTL_TCP .

We'll start with the CAST_ functions. These simply change the type field of a RAW variable to be VARCHAR2, and vice-versa. They do this without any translation of the data contained in the variable whatsoever. Normally, if I assign a RAW to a VARCHAR2, the VARCHAR2 would be twice as long as the RAW was, and would contain hexadecimal digits. Each byte of the RAW would be converted to hexadecimal (we took advantage of this translation in the DBMS_OBFUSCATION_TOOLKIT routine for example, to display encrypted data in hexadecimal on screen). In the cases where we do not desire this translation to take place, the CAST_TO_VARCHAR2 function comes in handy. To see what it does, we can use the DUMP SQL function as follows:

Appendix A

1230

tkyte@TKYTE816> create table t ( r raw(10) ); Table created.

tkyte@TKYTE816> insert into t values ( utl_raw.cast_to_raw('helloWorld' ) ); 1 row created.

tkyte@TKYTE816> select dump(r) r1, dump(utl_raw.cast_to_varchar2(r)) r1 2 from t;

R1 R1 ------------------------------ ------------------------------ Typ=23 Len=10: Typ=1 Len=10: 104,101,108,108,111,87,111,114 104,101,108,108,111,87,111,114 ,108,100 ,108,100

As you can see from the DUMP, the only thing that changed about the data was the TYP of it. It changed from 23 to 1. If you go to the Oracle Call Interface Programmer's Guide and look at the Internal Datatypeschart, you will find that type 23 is a RAW up to 2000 bytes in length, and type 1 is a VARCHAR2 up to 4000 bytes. The only thing CAST_TO_VARCHAR2 does is to change the data type flag in the variable – it does not touch the data at all. This is exactly what we need in the case where we use DBMS_LOB.SUBSTR on a BFILE, and this BFILE happens to contain 'clear text'. We need to convert this RAW into a VARCHAR2 without it being converted into hexadecimal – we only need change the data type.

UTL_RAW.CAST_TO_RAW goes the other way. If you have a VARCHAR2 that you need treated as a RAW,this will convert it by changing the type, and nothing else. We use this in our SIMPLE_TCP_CLIENTimplementation, in the section on the UTL_SMTP supplied package. Externally, the PL/SQL client is sending us VARCHAR2 data, but the Java layer needs byte arrays (RAWs). PL/SQL does this conversion easily for us.

The last two functions of note are UTL_RAW.LENGTH and UTL_RAW.SUBSTR. When we have a RAW and send it to the built-in routines LENGTH and SUBSTR, the RAW will be implicitly converted to a VARCHAR2(into hexadecimal) first. Unfortunately, the built-in functions are not overloaded to accept and receive RAW types, but rather they convert them into VARCHAR2. This means the return from LENGTH would always be twice the size, SUBSTR would always return a hexadecimal string, and we would have to fix the offset and length parameters as well. The UTL_RAW functions supply this missing functionality for us. They are equivalent to the following SQL:

tkyte@TKYTE816> select utl_raw.length(r), length(r)/2 from t;

UTL_RAW.LENGTH(R) LENGTH(R)/2 ----------------- ----------- 10 10

tkyte@TKYTE816> select utl_raw.substr(r,2,3) r1, 2 hextoraw(substr(r,3,6)) r2 3 from t 4 /

R1 R2 ---------- ---------- 656C6C 656C6C

Using the UTL_RAW functions is not mandatory, but it certainly makes life easier. Figuring out the byte offsets for a SUBSTR would be more complex, and remembering to divide by two is just something we don't need to do.

UTL_SMTP and Sending Mail

1231

UTL_SMTP and Sending Mail UTL_SMTP, introduced for the first time in Oracle 8.1.6, is an interface to the Simple Mail Transfer Protocol. It requires that you have an SMTP server in your network somewhere – most sites I have been to have at least one SMTP server running as it is the most popular method for sending mail.

The UTL_SMTP package is best suited for sending small, text only e-mails from the database. While its API supports the sending of attachments and everything else – it is left to you to actually encode the multi-part document – for example turning binary attachments into mime-encoded documents.

In this section we'll visit the example introduced in the DBMS_JOB section, which used UTL_SMTP, build upon it by adding additional functionality. We will also look at an alternative to UTL_SMTP that provides somewhat much more functionality – including the ability to easily send attachments with the e-mail. Since SMTP is a very low level protocol, we'll reuse existing public domain code to get an SMTP interface at much higher level – and we'll get it with very little code.

UTL_SMTP – a larger example In the DBMS_JOB section, we explore how to send an e-mail using UTL_SMTP and 'apparently' making it execute faster by doing it asynchronously. We also made e-mail transactional in nature in that section; if you rollback, the e-mail does not get sent, if you commit – out it goes. I highly recommend the use of DBMS_JOB as a layer on your e-mail's routines for these reasons. In that section, the example UTL_SMTP routine we used was:

Appendix A

1232

tkyte@TKYTE816> create or replace 2 PROCEDURE send_mail (p_sender IN VARCHAR2, 3 p_recipient IN VARCHAR2, 4 p_message IN VARCHAR2) 5 as 6 l_mailhost VARCHAR2(255) := 'yourserver.acme.com'; 7 l_mail_conn utl_smtp.connection; 8 BEGIN 9 l_mail_conn := utl_smtp.open_connection(l_mailhost, 25); 10 utl_smtp.helo(l_mail_conn, l_mailhost); 11 utl_smtp.mail(l_mail_conn, p_sender); 12 utl_smtp.rcpt(l_mail_conn, p_recipient); 13 utl_smtp.open_data(l_mail_conn ); 14 utl_smtp.write_data(l_mail_conn, p_message); 15 utl_smtp.close_data(l_mail_conn ); 16 utl_smtp.quit(l_mail_conn); 17 end; 18 / Procedure created.

tkyte@TKYTE816> begin 2 send_mail( '[email protected]', 3 '[email protected]', 4 'Hello Tom' ); 5 end; 6 /


This works OK but is very limited in nature. It sends e-mail to exactly one recipient, you cannot CC (Carbon Copy) or BCC (Blind Carbon Copy) anyone, you cannot setup a subject; the e-mail always arrives with a 'blank' subject line. We would like to support more options with this package.

A full discussion of all of the possibilities with UTL_SMTP would require in depth knowledge of the SMTP protocol itself – something that is outside the scope of this book. Readers interested in all of the opportunities available with SMTP should review RFC812 – which is the description of SMTP. This is available online at http://www.faqs.org/rfcs/rfc821.html. Below, I will simply present how to send an e-mail using UTL_SMTPthat supports:

❑ Multiple 'To' recipients.

❑ Multiple 'CC' recipients.

❑ Multiple 'BCC' recipients.

❑ A single body of up to 32 KB in size.

❑ A subject line.

❑ A descriptive 'from' line (instead of showing just the e-mail address as the 'from' in the e-mail client).

A specification for a PL/SQL package that supports this might look like the following. Here, we define an array type to allow a caller to easily send a list of recipients as well as provide the external specification of the PL/SQL routine we will be implementing:


1233

tkyte@TKYTE816> create or replace package mail_pkg 2 as 3 type array is table of varchar2(255); 4 5 procedure send( p_sender_e-mail in varchar2, 6 p_from in varchar2, 7 p_to in array default array(), 8 p_cc in array default array(), 9 p_bcc in array default array(), 10 p_subject in varchar2, 11 p_body in long ); 12 end; 13 / Package created.

The package body for this implementation is relatively straightforward – if understand just enough of the SMTP protocol and what an e-mail looks like (how e-mail clients get the From, To, CC and so on). Before we look at the code, we'll look at what an e-mail might actually look like. Consider the following ASCII text:

From: Oracle Database Account <[email protected]> Subject: This is a subject To: [email protected], [email protected] Cc: [email protected]

Hello Tom, this is the mail you need

That is what you would transmit as the body of the e-mail using UTL_SMTP to have the e-mail client set the From, Subject, and so on. There are no SMTP commands to do this piece of 'magic', rather, this header information is placed right in the body of the e-mail itself; separated from the text of the e-mail by a blank line. Once we understand this, sending an e-mail with all of the options that we need is pretty easy. The only thing we need to understand beyond that is that in order to send the e-mail to more then one recipient, we simply call UTL_SMTP.RCPT more then once – with different names. That's all of the information we need to know then to send an e-mail.

So, here is the package body. We start with a couple of constants and global variables. You will of course need to change the g_mailhost to be the name of a server you have access to, in this code I have given it a generic name; yourserver.acme.com:

tkyte@TKYTE816> create or replace package body mail_pkg 2 as 3 4 g_crlf char(2) default chr(13)||chr(10); 5 g_mail_conn utl_smtp.connection; 6 g_mailhost varchar2(255) := 'yourserver.acme.com'; 7

Next, we have an internal (unpublished) function to send an e-mail to many recipients – it in effect addresses the e-mail. At the same time, it builds the To: or CC: lines that we will eventually send as part of the e-mail itself and returns that formatted string. It was implemented as a separate function since we need to do this separately for the To, CC, and BCC lists:

Appendix A

1234

8 function address_email( p_string in varchar2, 9 p_recipients in array ) return varchar2 10 is 11 l_recipients long; 12 begin 13 for i in 1 .. p_recipients.count 14 loop 15 utl_smtp.rcpt(g_mail_conn, p_recipients(i) ); 16 if ( l_recipients is null ) 17 then 18 l_recipients := p_string || p_recipients(i) ; 19 else 20 l_recipients := l_recipients || ', ' || p_recipients(i); 21 end if; 22 end loop; 23 return l_recipients; 24 end; 25 26

Now we have the implementation of our published function, the one that people will actually call to send mail. It starts with an internal procedure writeData that is used to simplify the sending of the e-mail headers (the To:, From:, Subject: records). If the header record is not Null, this routine will use the appropriate UTL_SMTP call to send it – along with the necessary end of line marker (the carriage return/line feed):

27 procedure send( p_sender_email in varchar2, 28 p_from in varchar2 default NULL, 29 p_to in array default array(), 30 p_cc in array default array(), 31 p_bcc in array default array(), 32 p_subject in varchar2 default NULL, 33 p_body in long default NULL ) 34 is 35 l_to_list long; 36 l_cc_list long; 37 l_bcc_list long; 38 l_date varchar2(255) default 39 to_char( SYSDATE, 'dd Mon yy hh24:mi:ss' ); 40 41 procedure writeData( p_text in varchar2 ) 42 as 43 begin 44 if ( p_text is not null ) 45 then 46 utl_smtp.write_data( g_mail_conn, p_text || g_crlf ); 47 end if; 48 end;

Now we are ready to actually send the mail. This part is not very different from the very simple routine we started with. It begins in exactly the same fashion, by connecting to the SMTP server and starting a session:


1235

49 begin 50 g_mail_conn := utl_smtp.open_connection(g_mailhost, 25); 51 52 utl_smtp.helo(g_mail_conn, g_mailhost); 53 utl_smtp.mail(g_mail_conn, p_sender_email); 54

Here is where it differs, instead of calling UTL_SMTP.RCPT once; it uses the address_email function to call it (potentially) many times, building the To: and CC: list for us as well. It builds the BCC: list but we won't actually send that (we don't want the recipients to see that list!)

55 l_to_list := address_email( 'To: ', p_to ); 56 l_cc_list := address_email( 'Cc: ', p_cc ); 57 l_bcc_list := address_email( 'Bcc: ', p_bcc ); 58

Now, we use the OPEN_DATA call to start sending the body of the e-mail. The code on lines 61 through to 68 generates the header section of data. Line 69 sends the body of the e-mail (the contents of the e-mail) and line 70 terminates the e-mail for us.

59 utl_smtp.open_data(g_mail_conn ); 60 61 writeData( 'Date: ' || l_date ); 62 writeData( 'From: ' || nvl( p_from, p_sender_email ) ); 63 writeData( 'Subject: ' || nvl( p_subject, '(no subject)' ) ); 64 65 writeData( l_to_list ); 66 writeData( l_cc_list ); 67 68 utl_smtp.write_data( g_mail_conn, '' || g_crlf ); 69 utl_smtp.write_data(g_mail_conn, p_body ); 70 utl_smtp.close_data(g_mail_conn ); 71 utl_smtp.quit(g_mail_conn); 72 end; 73 74 75 end; 76 / Package body created.

Now I can test this API like this:

tkyte@TKYTE816> begin 2 mail_pkg.send 3 ( p_sender_email => '[email protected]', 4 p_from => 'Oracle Database Account <[email protected]>', 5 p_to => mail_pkg.array( '[email protected]',' [email protected] ' ), 6 p_cc => mail_pkg.array( ' [email protected] ' ), 7 p_bcc => mail_pkg.array( '[email protected]' ), 8 p_subject => 'This is a subject', 9 p_body => 'Hello Tom, this is the mail you need' ); 10 end; 11 /


Appendix A

1236

And that call is exactly what generated the ASCII text:

Date: 13 May 01 12:33:22 From: Oracle Database Account <[email protected] Subject: This is a subject To: [email protected], [email protected] Cc: [email protected]

Hello Tom, this is the mail you need

We saw above, this is what gets sent to all of these recipients, including [email protected], although we cannot see that recipient since it was on the BCC: line.

This covers most of the typical uses of the UTL_SMTP supplied package. Earlier I did say it is capable of sending e-mail with attachments and such but this would require an inordinate amount of effort on our part. We would have to:

❑ Learn how to format a multi-part mime encoded document, no small feat!

❑ Encode binary data using Base-64 (or use some equivalent encoding technique such as uuencoding, binhex, and so on).

That would be (conservatively) a couple of hundred, if not thousands of lines of PL/SQL code. Rather then do this; I will suggest that you use the already written and very robust JavaMail API as described below.

Loading and using the JavaMail API In order to use the UTL_SMTP package, you must already have a Java enabled database in Oracle 8i. This is because UTL_SMTP relies on UTL_TCP and UTL_TCP which in turn are built on Java functions. (Remember, if you don't have a Java enabled database you can use UTL_HTTP (see that section) to send simple e-mails). So, if you are able to use UTL_SMTP, we can go to the Sun website and download their JavaMail API. This will give us the ability to send much more complicated e-mails from the database; including attachments. The following is based on work performed by a co-worker of mine, Mark Piermarini who helps me out with lots of my Java issues.

If you go to http://java.sun.com/products/javamail/index.html, you'll be able to download their JavaMailAPI. The download you get will consist of a couple of hundred files; only one of which we are interested in. After you download the JavaMail API – make sure also to get their the JavaBeansTM ActivationFramework extension or JAF (javax.activation). This is needed to run the JavaMail API package.

After you have downloaded these two sets of files – you will need to extract mail.jar from the JavaMailAPIdownload and activation.jar from the JAF download. This is all you will need from this – feel free to read through the documentation, there is a lot of functionality in there we are not using, we are just using the 'send an e-mail' part of the API. The API includes functions for receiving mail as well from IMAP, POP, and other sources.

We will need to load the mail.jar and activation.jar into the database using loadjava but before we can do that we must repackage them. These jar files are compressed in a format that is not understood by the database byte code interpreter. You need to 'unjar' and 'rejar' them without compression or use a tool such as WinZip to 'rejar' them into a zip file. What I did on Windows 2000 was:


1237

1. Used WinZip to extract the contents of mail.jar into my c:\temp\mail directory

2. Used WinZip to create a new archive c:\temp\mail8i.zip

3. Put the contents of c:\temp\mail\*.* including subdirectories into this new archive

I did the same thing for activation.jar – only replacing mail with activation in the above steps. Now we are ready to load these zip (or jar files, whatever you named them) into the database. These files need to be loaded into the database using the SYS user since they have 'protected' Java packages that regular users cannot upload. We will use the loadjava commands:

loadjava -u sys/manager -o -r -v -f -noverify -synonym -g public activation8i.zip loadjava -u sys/manager -o -r -v -f -noverify -synonym -g public mail8i.zip

Where:

❑ -u sys/manager – is the user ID and password for your SYS account. Some of the packages are protected and must be loaded as SYS.

❑ -o – is shorthand for –oci8, I am using the oci8 driver. You could use the thin driver as well but you'll need to modify the command to do so

❑ -r – is short for –resolve. This will resolve all external references in the loaded classes helping to verify that the loaded java classes will be able to function after we load them

❑ -v – is short for –verbose. This gives us something to do while loadjava is running. We can see it work through each step of its process.

❑ -f – is short for –force. This isn't necessary on the first load but is OK to use. If you try a loadjava and hit an error, you can correct it, and reload – then you would either need to use the dropjava command to drop the jar file from the database or use –force. Using –forcejust makes it easier for us.

❑ -noverify – does not attempt to verify the bytecode. You must be granted oracle.aurora.security.JServerPermission(Verifier) to execute this option. In addition, this option must be used in conjunction with -r. SYS has this privilege. This is needed because the bytecode verifier will flag some issues with the mail.jar file and this works around that issue.

❑ -synonym – creates public synonyms for these classes. Since we will not install the mail java code we write as SYS, this allows us to 'see' the SYS loaded java classes.

❑ -g public – grants execute on these loaded classes to PUBLIC. If this is not desirable, change the –g to be just the user you want to create the 'send mail' routines in, for example -gUTILITY_ACCT.

You can find out more about loadjava and the above options in the Oracle8i Java Developers Guide.

After these packages are loaded, we are ready to create a Java stored procedure to actually send the mail. This procedure will act as a thin layer on top of the JavaMail API and will let us ultimately write a PL/SQL binding layer with the following spec:

Appendix A

1238

tkyte@TKYTE816> desc send FUNCTION send RETURNS NUMBER Argument Name Type In/Out Default? ------------------------------ ----------------------- ------ -------- P_FROM VARCHAR2 IN P_TO VARCHAR2 IN P_CC VARCHAR2 IN P_BCC VARCHAR2 IN P_SUBJECT VARCHAR2 IN P_BODY VARCHAR2 IN P_SMTP_HOST VARCHAR2 IN P_ATTACHMENT_DATA BLOB IN P_ATTACHMENT_TYPE VARCHAR2 IN P_ATTACHMENT_FILE_NAME VARCHAR2 IN

This function will give us the ability to use CC's and BCC's and send an attachment. It is left as an exercise for the reader to implement passing arrays of BLOBs or overloading this to support CLOB or BFILE types for attachments as well.

The Java stored procedure we will create follows. It uses the basic functionality of the JavaMail API class and is relatively straightforward. Again, we are not going into all of the uses of the JavaMail API (that could be a book in itself), just the basics here. The mail class below has a single method; send. This is the method we will use to send a message. As it is implemented, it returns the number 1 if it is successful in sending the mail and a 0 otherwise. This implementation is very basic – it could be much more sophisticated, providing support for many attachment types (CLOBs, BFILEs, LONGs, and so on). It could also be modified to report back to the caller the exact error received from SMTP such as invalid recipient, no transport.

tkyte@TKYTE816> create or replace and compile 2 java source named "mail" 3 as 4 import java.io.*; 5 import java.sql.*; 6 import java.util.Properties; 7 import java.util.Date; 8 import javax.activation.*; 9 import javax.mail.*; 10 import javax.mail.internet.*; 11 import oracle.jdbc.driver.*; 12 import oracle.sql.*; 13 14 public class mail 15 { 16 static String dftMime = "application/octet-stream"; 17 static String dftName = "filename.dat"; 18 19 public static oracle.sql.NUMBER 20 send(String from, 21 String to, 22 String cc, 23 String bcc, 24 String subject, 25 String body, 26 String SMTPHost, 27 oracle.sql.BLOB attachmentData, 28 String attachmentType, 29 String attachmentFileName)


1239

The above argument list matches up with the SQL call specification we outlined above – the arguments are mostly self-explanatory. The two that need some clarification are the attachmentType and the attachmentFileName. The attachmentType should be a MIME (Multi-purpose Internet Mail Extensions) type – as you may be familiar with, from HTML documents. The MIME type of a GIF image for example is image/gif, the mime type of a plain text document would be text/plain, and a HTML attachment would be text/html. The attachmentFileName in this example is not the name of an existing OS file that would be attached but rather the filename of the attachment in the e-mail itself – what the recipient of this e-mail will see the name of the attachment as. The actual attachment is the oracle.sql.BLOB that is sent to this routine. Now, onto the body of the code. We begin by setting the session property mail.smtp.host to the name of the SMTP host the caller sent to us – the JavaMail API reads this value when deciding what SMTP server to connect to:

30 { 31 int rc = 0; 32 33 try 34 { 35 Properties props = System.getProperties(); 36 props.put("mail.smtp.host", SMTPHost); 37 Message msg = 38 new MimeMessage(Session.getDefaultInstance(props, null)); 39

Next, we set up the e-mail headers. This part tells the JavaMail API who the message is from, who to send it to, who to send a carbon copy (CC) or blind carbon copy (BCC), what the subject of the e-mail is and what date should be associated with the e-mail:

40 msg.setFrom(new InternetAddress(from)); 41 42 if (to != null && to.length() > 0) 43 msg.setRecipients(Message.RecipientType.TO, 44 InternetAddress.parse(to, false)); 45 46 if (cc != null && cc.length() > 0) 47 msg.setRecipients(Message.RecipientType.CC, 48 InternetAddress.parse(cc, false)); 49 50 if (bcc != null && bcc.length() > 0) 51 msg.setRecipients(Message.RecipientType.BCC, 52 InternetAddress.parse(bcc, false)); 53 54 if ( subject != null && subject.length() > 0 ) 55 msg.setSubject(subject); 56 else msg.setSubject("(no subject)"); 57 58 msg.setSentDate(new Date()); 59

Next, we use one of two methods to send an e-mail. If the attachmentData argument is not Null, then we will MIME encode the e-mail – a standard that supports the sending of attachments and other multi-part documents. We do this by setting up multiple MIME body parts – in this case two of them, one for the body of the e-mail (the text) and the other for the attachment itself. Lines 76 through 78 need a little additional explanation. They are how we can send an e-mail via a BLOB. The JavaMail API doesn't understand the oracle.sql.BLOB type natively (it is after all a generic API). In order to send the BLOB attachment, we must provide a method for the JavaMail API to get at the BLOB data. We accomplish that by creating our own DataHandler – a class with an interface that the JavaMail API understands how to call in order to get data to populate the attachment. This class (BLOBDataHandler) is implemented by us as a nested class below.

Appendix A

1240

60 if (attachmentData != null) 61 { 62 MimeBodyPart mbp1 = new MimeBodyPart(); 63 mbp1.setText((body != null ? body : "")); 64 mbp1.setDisposition(Part.INLINE); 65 66 MimeBodyPart mbp2 = new MimeBodyPart(); 67 String type = 68 (attachmentType != null ? attachmentType : dftMime); 69 70 String fileName = (attachmentFileName != null ? 71 attachmentFileName : dftName); 72 73 mbp2.setDisposition(Part.ATTACHMENT); 74 mbp2.setFileName(fileName); 75 76 mbp2.setDataHandler(new 77 DataHandler(new BLOBDataSource(attachmentData, type)) 78 ); 79 80 MimeMultipart mp = new MimeMultipart(); 81 mp.addBodyPart(mbp1); 82 mp.addBodyPart(mbp2); 83 msg.setContent(mp); 84 }

If the e-mail does not have an attachment – setting the body of the e-mail is accomplished very simply via the single call to setText:

85 else 86 { 87 msg.setText((body != null ? body : "")); 88 } 89 Transport.send(msg); 90 rc = 1; 91 } catch (Exception e) 92 { 93 e.printStackTrace(); 94 rc = 0; 95 } finally 96 { 97 return new oracle.sql.NUMBER(rc); 98 } 99 } 100

Now for our nested class BLOBDataSource. It simply provides a generic interface for the JavaMail API to access our oracle.sql.BLOB type. It is very straightforward in its implementation:

101 // Nested class that implements a DataSource. 102 static class BLOBDataSource implements DataSource 103 { 104 private BLOB data; 105 private String type; 106107 BLOBDataSource(BLOB data, String type)


1241

108 { 109 this.type = type; 110 this.data = data; 111 } 112113 public InputStream getInputStream() throws IOException 114 { 115 try 116 { 117 if(data == null) 118 throw new IOException("No data."); 119120 return data.getBinaryStream(); 121 } catch(SQLException e) 122 { 123 throw new 124 IOException("Cannot get binary input stream from BLOB."); 125 } 126 } 127128 public OutputStream getOutputStream() throws IOException 129 { 130 throw new IOException("Cannot do this."); 131 } 132133 public String getContentType() 134 { 135 return type; 136 } 137138 public String getName() 139 { 140 return "BLOBDataSource"; 141 } 142 } 143 } 144 /

Java created.

Now that we have the Java class created for PL/SQL to bind to, we need to create that binding routine to map the PL/SQL types to their Java types and to bind the PL/SQL routine to this Java class. This is simply done by the following:

tkyte@TKYTE816> create or replace function send( 2 p_from in varchar2, 3 p_to in varchar2, 4 p_cc in varchar2, 5 p_bcc in varchar2, 6 p_subject in varchar2, 7 p_body in varchar2, 8 p_smtp_host in varchar2, 9 p_attachment_data in blob, 10 p_attachment_type in varchar2,

Appendix A

1242

11 p_attachment_file_name in varchar2) return number 12 as 13 language java name 'mail.send( java.lang.String, 14 java.lang.String, 15 java.lang.String, 16 java.lang.String, 17 java.lang.String, 18 java.lang.String, 19 java.lang.String, 20 oracle.sql.BLOB, 21 java.lang.String, 22 java.lang.String 23 ) return oracle.sql.NUMBER'; 24 /

Function created.

Now, the very last thing we must do before using this is to ensure our user (the owner of the above mail class and send stored procedure) has sufficient privileges to execute the routine. These would be the following:

sys@TKYTE816> begin 2 dbms_java.grant_permission( 3 grantee => 'USER', 4 permission_type => 'java.util.PropertyPermission', 5 permission_name => '*', 6 permission_action => 'read,write' 7 ); 8 dbms_java.grant_permission( 9 grantee => 'USER', 10 permission_type => 'java.net.SocketPermission', 11 permission_name => '*', 12 permission_action => 'connect,resolve'

13 ); 14 end; 15 /


Note that in the grant on java.net.SocketPermission, I used a wildcard in the permission_name.This allows USER to connect to and resolve any host. Technically, we could put in there just the name of the SMTP server we will be using. This would be the minimal grant we need. It is needed in order to resolve the hostname of our SMTP host and then connect to it. The other permission, java.util.PropertyPermission, is needed in order to set the mail.smtp.host in our sessions properties.

Now we are ready to test. I reused some code from the DBMS_LOB section where we had a routine load_a_file. I modified that and the DEMO table to have a BLOB column instead of a CLOB and loaded the mail8i.zip file we loaded in as a class into this demo table. Now I can use the following PL/SQL block to send it to myself as an attachment in an e-mail from the database:


1243

tkyte@TKYTE816> set serveroutput on size 1000000 tkyte@TKYTE816> exec dbms_java.set_output( 1000000 )

tkyte@TKYTE816> declare 2 ret_code number; 3 begin 4 for i in (select theBlob from demo ) 5 loop 6 ret_code := send( 7 p_from => '[email protected]', 8 p_to => '[email protected]', 9 p_cc => NULL, 10 p_bcc => NULL, 11 p_subject => 'Use the attached Zip file', 12 p_body => 'to send email with attachments....', 13 p_smtp_host => 'yourserver.acme.com', 14 p_attachment_data => i.theBlob, 15 p_attachment_type => 'application/winzip', 16 p_attachment_file_name => 'mail8i.zip'); 17 if ret_code = 1 then 18 dbms_output.put_line ('Successfully sent message...'); 19 else 20 dbms_output.put_line ('Failed to send message...'); 21 end if; 22 end loop; 23 end; 24 / Successfully sent message...


You definitely want to set serverouput on and call the DBMS_JAVA.SET_OUTPUT routine when testing this. This is because the exception is being printed by the Java stored procedure to System.out and by default that will go into a trace file on the server. If you want to see any errors in your SQL*PLUS session, you need to make these two settings. It will be very useful for debugging purposes!

Summary In this section, we briefly reviewed the existing UTL_SMTP package. Here we have seen how to send e-mails to multiple recipients with a custom From: and Subject: header. This should satisfy most people's needs for sending e-mail from the database. UTL_SMTP is good for sending simple text only e-mails but sending attachments or complex e-mails is beyond its capabilities (unless you want to encode the entire e-mail yourself). In the cases where you need this additional sophistication, we looked at how to use the JavaMail API. Since Sun has graciously supplied us with all of the logic we would need to do this – we'll just reuse their code. This section has demonstrated not only how to send mail but a powerful side effect of having Java as an alternative stored procedure language. Now you can use the entire set of public domain code and class libraries that are available. We can enable the database to do many things that were not previously possible. In fact, the PL/SQL developers at Oracle used the same technique themselves. UTL_TCP is built on Java itself in Oracle 8i.

Appendix A

1244

UTL_TCP

Oracle 8.1.6 introduced for the first time, the UTL_TCP package. This package allows PL/SQL to open a network socket connection over TCP/IP to any server accepting connections. Assuming you know the protocol of a server, you can now 'talk' to it from PL/SQL. For example, given that I know HTTP (Hyper Text Transfer Protocol), I can code in UTL_TCP the following:

test_jsock@DEV816> DECLARE 2 c utl_tcp.connection; -- TCP/IP connection to the web server 3 n number; 4 buffer varchar2(255); 5 BEGIN 6 c := utl_tcp.open_connection('proxy-server', 80); 7 n := utl_tcp.write_line(c, 'GET http://www.apress.com/ HTTP/1.0'); 8 n := utl_tcp.write_line(c); 9 BEGIN 10 LOOP 11 n:=utl_tcp.read_text( c, buffer, 255 ); 12 dbms_output.put_line( buffer ); 13 END LOOP; 14 EXCEPTION 15 WHEN utl_tcp.end_of_input THEN 16 NULL; -- end of input 17 end; 18 utl_tcp.close_connection(c); 19 END; 20 / HTTP/1.1 200 OK Date: Tue, 30 Jan 2001 11:33:50 GMT Server: Apache/1.3.9 (Unix) mod_perl/1.21 ApacheJServ/1.1Content-Type: text/html

<head><title>OracleCorporation</title>

UTL_TCP

1245

This lets me open a connection to a server, in this case a proxy server named proxy-server. It lets me get through our firewall to the outside Internet. This happens on line 6. I then request a web page on lines 7 and 8. On lines 10 through to 13, we receive the contents of the web page, including the all-important HTTP headers (something UTL_HTTP, another supplied package, won't share with us) and print it. When UTL_TCP throws the UTL_TCP.END_OF_INPUT exception, we are done, and we break out of the loop. We then close our connection and that's it.

This simple example demonstrates a majority of the functionality found in the UTL_TCP package. We didn't see functions such as AVAILABLE, which tells us if any data is ready to be received. We skipped FLUSH, which causes any buffered output to be transmitted (we didn't use buffering, hence did not need this call). Likewise, we did not use every variation of READ, WRITE, and GET to put and get data on the socket, but the example above shows how to use UTL_TCP fairly completely.

The one thing I don't necessarily like about the above is the speed at which it runs. It is in my experience that UTL_TCP, while functional, is not as high performing as it could be in this release (Oracle 8i). In Oracle 8.1.7.1, this performance issue is fixed (bug #1570972 corrects this issue).

So, how slow is slow? The above code, to retrieve a 16 KB document, takes anywhere from four to ten seconds, depending on platform. This is especially bad considering the native UTL_HTTP function can do the same operation with a sub-second response time. Unfortunately, UTL_HTTP doesn't permit access to cookies, HTTP headers, binary data, basic authentication, and the like, so using an alternative is many times useful. I think we can do better. To this end, we will implement our own UTL_TCP package. However, we will use the Object Type metaphor we discussed in Chapter 20 on Using Object Relational Features. What we will do is to implement a SocketType in PL/SQL with some of the underlying 'guts' in Java. In the UTL_HTTP section, we put this SocketType we created to use in building a better UTL_HTTP package for ourselves as well. Since our functionality will be modeled after the functionality available in the UTL_TCP package, when Oracle9i is released with native and faster UTL_TCP support, we can easily re-implement our type body using the real UTL_TCP package, and stop using our Java-supported one.

The SocketType Our SocketType Object Type will use the following specification:

tkyte@TKYTE816> create or replace type SocketType 2 as object 3 ( 4 -- 'Private data', rather than you 5 -- passing a context to each procedure, like you 6 -- do with UTL_FILE. 7 g_sock number, 8 9 -- A function to return a CRLF. Just a convenience. 10 static function crlf return varchar2, 11 12 -- Procedures to send data over a socket. 13 member procedure send( p_data in varchar2 ), 14 member procedure send( p_data in clob ), 15 16 member procedure send_raw( p_data in raw ), 17 member procedure send_raw( p_data in blob ),

Appendix A

1246

18 19 -- Functions to receive data from a socket. These return 20 -- Null on eof. They will block waiting for data. If 21 -- this is not desirable, use PEEK below to see if there 22 -- is any data to read. 23 member function recv return varchar2, 24 member function recv_raw return raw, 25 26 -- Convienence function. Reads data until a CRLF is found. 27 -- Can strip the CRLF if you like (or not, by default). 28 member function getline( p_remove_crlf in boolean default FALSE ) 29 return varchar2, 30 31 -- Procedures to connect to a host and disconnect from a host. 32 -- It is important to disconnect, else you will leak resources 33 -- and eventually will not be able to connect. 34 member procedure initiate_connection( p_hostname in varchar2, 35 p_portno in number ), 36 member procedure close_connection, 37 38 -- Function to tell you how many bytes (at least) might be 39 -- ready to be read. 40 member function peek return number 41 ); 42 /

Type created.

This set of functionality is pretty much modeled after the UTL_TCP package, and provides much of the same interface. In fact, it could be implemented on top of that package if you wanted. We are going to implement it on top of a different package however, one which I call the SIMPLE_TCP_CLIENT. This is a ordinary PL/SQL package that the SocketType will be built on. This is really our specification of a UTL_TCP package:

tkyte@TKYTE816> CREATE OR REPLACE PACKAGE simple_tcp_client 2 as 3 -- A function to connect to a host. Returns a 'socket', 4 -- which is really just a number. 5 function connect_to( p_hostname in varchar2, 6 p_portno in number ) return number; 7 8 -- Send data. We only know how to send RAW data here. Callers 9 -- must cast VARCHAR2 data to RAW. At the lowest level, all 10 -- data on a socket is really just 'bytes'. 11 12 procedure send( p_sock in number, 13 p_data in raw ); 14 15 -- recv will receive data. 16 -- If maxlength is -1, we try for 4k of data. If maxlength 17 -- is set to anything OTHER than -1, we attempt to 18 -- read up to the length of p_data bytes. In other words, 19 -- I restrict the receives to 4k unless otherwise told not to. 20 procedure recv( p_sock in number, 21 p_data out raw,

UTL_TCP

1247

22 p_maxlength in number default -1 ); 23 24 -- Gets a line of data from the input socket. That is, data 25 -- up to a \n. 26 procedure getline( p_sock in number, 27 p_data out raw ); 28 29 30 -- Disconnects from a server you have connected to. 31 procedure disconnect( p_sock in number ); 32 33 -- Gets the server time in GMT in the format yyyyMMdd HHmmss z 34 procedure get_gmt( p_gmt out varchar2 ); 35 36 -- Gets the server's timezone. Useful for some Internet protocols. 37 procedure get_timezone( p_timezone out varchar2 ); 38 39 -- Gets the hostname of the server you are running on. Again, 40 -- useful for some Internet protocols. 41 procedure get_hostname( p_hostname out varchar2 ); 42 43 -- Returns the number of bytes available to be read. 44 function peek( p_sock in number ) return number; 45 46 -- base64 encodes a RAW. Useful for sending e-mail 47 -- attachments or doing HTTP which needs the user/password 48 -- to be obscured using base64 encoding. 49 procedure b64encode( p_data in raw, p_result out varchar2 ); 50 end; 51 /

Package created.

Now, as none of these functions can actually be written in PL/SQL, we will implement them in Java instead. The Java for doing this is surprisingly small. The entire script is only 94 lines long. We are using the native Socket class for Java, and will maintain a small array of them, allowing PL/SQL to have up to ten connections open simultaneously. If you would like more than ten, just make the socketUsed array larger in the code below. I've tried to keep this as simple, and as small as possible, preferring to do the bulk of any work in PL/SQL. I'll present the small class we need, and then comment on it:

tkyte@TKYTE816> set define off

tkyte@TKYTE816> CREATE or replace and compile JAVA SOURCE 2 NAMED "jsock" 3 AS 4 import java.net.*; 5 import java.io.*; 6 import java.util.*; 7 import java.text.*; 8 import sun.misc.*; 9 10 public class jsock 11 { 12 static int socketUsed[] = { 0,0,0,0,0,0,0,0,0,0 }; 13 static Socket sockets[] = new Socket[socketUsed.length]; 14 static DateFormat tzDateFormat = new SimpleDateFormat( "z" ); 15 static DateFormat gmtDateFormat = 16 new SimpleDateFormat( "yyyyMMdd HHmmss z" ); 17 static BASE64Encoder encoder = new BASE64Encoder(); 18

Appendix A

1248

This class has some static variables – the two arrays, socketUsed and sockets are the main ones. When returns are called from PL/SQL, we must return to it something it can send to us on subsequent calls, to identify the socket connection it wants to use. We cannot return the Java Socket class to PL/SQL, so I am using an array in which to store them, and will return to PL/SQL an index into that array. If you look at the java_connect_to method, it looks in the socketsUsed array for an empty slot, and allocates this to the connection. That index into socketsUsed is what PL/SQL will see. We use this in the remaining sockets routines to access the actual Java class that represents a socket.

The other static variables are there for reasons of performance. I needed some date format objects, and rather than NEW them each time you call java_get_gmt or java_get_timezone, I allocate them once, and just reuse them. Lastly is the base 64 encoder object. For the same reason I allocate the date formatter objects, I allocate the encoder.

Now for the routine that connects over TCP/IP, to a server. This logic loops over the socketUsedarray looking for an empty slot (where socketUsed[I] is not set to 1). If it finds one, it uses the Java Socket class to create a connection to the host/port combination that was passed in, and sets the socketUsed flag for the array slot to 1. It then returns a -1 on error (no empty slots), or a non-negative number upon success:

19 static public int java_connect_to( String p_hostname, int p_portno ) 20 throws java.io.IOException 21 { 22 int i; 23 24 for( i = 0; i < socketUsed.length && socketUsed[i] == 1; i++ ); 25 if ( i < socketUsed.length ) 26 { 27 sockets[i] = new Socket( p_hostname, p_portno ); 28 socketUsed[i] = 1; 29 } 30 return i<socketUsed.length?i:-1; 31 } 32 33

The next routines are the two most frequently called Java routines. They are responsible for sending and receiving data on a connected TCP/IP socket. The java_send_data routine is straightforward – it simply gets the output stream associated with the socket, and writes the data. The java_recv_data is slightly more complex. It uses OUT parameters, hence the use of int[] p_length for example, in order to return data. This routine inspects the length that was sent in by the caller, and if the length was -1, it will allocate a 4 KB buffer to read into, else it will allocate a buffer of the size specified. It will then try to read that much data from the socket. The actual amount of data read (which will be less than or equal to the amount requested) is placed in p_length as a return value:

34 static public void java_send_data( int p_sock, byte[] p_data ) 35 throws java.io.IOException 36 { 37 (sockets[p_sock].getOutputStream()).write( p_data ); 38 } 39 40 static public void java_recv_data( int p_sock,

UTL_TCP

1249

41 byte[][] p_data, int[] p_length) 42 throws java.io.IOException 43 { 44 p_data[0] = new byte[p_length[0] == -1 ? 4096:p_length[0] ]; 45 p_length[0] = (sockets[p_sock].getInputStream()).read( p_data[0] ); 46 } 47

java_getline is a convenience function. Many Internet protocols respond to operations 'a line at a time', and being able to get a simple line of text is very handy. For example, the headers sent back in the HTTP protocol are simply lines of ASCII text. This routine works by using the DataInputStream.readLine method, and if a line of text is read in, it will return it (putting the new line, which readLine strips off, back on). Otherwise, the data will be returned as Null:

48 static public void java_getline( int p_sock, String[] p_data ) 49 throws java.io.IOException 50 { 51 DataInputStream d = 52 new DataInputStream((sockets[p_sock].getInputStream())); 53 p_data[0] = d.readLine(); 54 if ( p_data[0] != null ) p_data[0] += "\n"; 55 } 56

java_disconnect is very straightforward as well. It simply sets the socketUsed array flag back to zero, indicating we can reuse this slot in the array, and closes the socket down for us:

57 static public void java_disconnect( int p_sock ) 58 throws java.io.IOException 59 { 60 socketUsed[p_sock] = 0; 61 (sockets[p_sock]).close(); 62 } 63

The java_peek_sock routine is used to see if data on a socket is available to be read. This is useful for times when the client does not want to block on a receive of data. If you look to see if anything is available, you can tell if a receive will block, or return right away:

64 static public int java_peek_sock( int p_sock ) 65 throws java.io.IOException 66 { 67 return (sockets[p_sock].getInputStream()).available(); 68 } 69

Now we have our two time functions. java_get_timezone is used to return the time zone of the database server. This is particularly useful if you need to convert an Oracle DATE from one time zone to another using the NEW_TIME built-in function, or if you just need know the time zone in which the server is operating. The second function, java_get_gmt, is useful for getting the server's current date and time in GMT (Greenwich Mean Time):

Appendix A

1250

70 static public void java_get_timezone( String[] p_timezone ) 71 { 72 tzDateFormat.setTimeZone( TimeZone.getDefault() ); 73 p_timezone[0] = tzDateFormat.format(new Date()); 74 } 75 76 77 static public void java_get_gmt( String[] p_gmt ) 78 { 79 gmtDateFormat.setTimeZone( TimeZone.getTimeZone("GMT") ); 80 p_gmt[0] = gmtDateFormat.format(new Date()); 81 } 82

The b64encode routine will base 64 encode a string of data. Base 64 encoding is an Internet-standard method of encoding arbitrary data into a 7bit ASCII format, suitable for transmission. We will use this function in particular when implementing our HTTP package, as it will support basic authentication (used by many web sites that require you to log in via a username and password).

83 static public void b64encode( byte[] p_data, String[] p_b64data ) 84 { 85 p_b64data[0] = encoder.encode( p_data ); 86 } 87

The last routine in this class simply returns the hostname of the database server. Some Internet protocols request that you transmit this information (for example, SMTP – simple mail transfer protocol):

88 static public void java_get_hostname( String[] p_hostname ) 89 throws java.net.UnknownHostException 90 { 91 p_hostname[0] = (InetAddress.getLocalHost()).getHostName(); 92 } 93 94 } 95 /

Java created.

The Java methods themselves are rather straightforward. If you recall from Chapter 19 on Java Stored Procedures, in order to get OUT parameters, we must send, what appears to be an array, to Java. Hence, most of the procedures above take the form of:

40 static public void java_recv_data( int p_sock, 41 byte[][] p_data, int[] p_length)

This allows me to return a value in p_data, and return a value in p_length. Now that we have our Java class, we are ready to build our package body for the SIMPLE_TCP_CLIENT package. It consists almost entirely of bindings to Java:

UTL_TCP

1251

tkyte@TKYTE816> CREATE OR REPLACE PACKAGE BODY simple_tcp_client 2 as 3 4 function connect_to( p_hostname in varchar2, 5 p_portno in number ) return number 6 as language java 7 name 'jsock.java_connect_to( java.lang.String, int ) return int'; 8 9 10 procedure send( p_sock in number, p_data in raw ) 11 as language java 12 name 'jsock.java_send_data( int, byte[] )'; 13 14 procedure recv_i ( p_sock in number, 15 p_data out raw, 16 p_maxlength in out number ) 17 as language java 18 name 'jsock.java_recv_data( int, byte[][], int[] )'; 19 20 procedure recv( p_sock in number, 21 p_data out raw, 22 p_maxlength in number default -1 ) 23 is 24 l_maxlength number default p_maxlength; 25 begin 26 recv_i( p_sock, p_data, l_maxlength ); 27 if ( l_maxlength <> -1 ) 28 then 29 p_data := utl_raw.substr( p_data, 1, l_maxlength ); 30 else 31 p_data := NULL; 32 end if; 33 end;

Here, I have a RECV_I and a RECV procedure. RECV_I is a private procedure (the _I stands for internal), not directly callable out of this package. It is called by RECV. RECV provides a 'friendly' internal on top of RECV_I – it checks to see if any data was read from the socket and if so, it sets the length correctly. If you recall from the Java code above, we allocated a fixed size buffer in the RECVroutine, and read up to that many bytes from the socket. We need to resize our buffer to be exactly that size here, and this is the purpose of the UTL_RAW.SUBSTR function. Otherwise, if no data was read, we simply return Null.

34 35 procedure getline_i( p_sock in number, 36 p_data out varchar2 ) 37 as language java 38 name 'jsock.java_getline( int, java.lang.String[] )'; 39 40 procedure getline( p_sock in number, 41 p_data out raw ) 42 as 43 l_data long; 44 begin 45 getline_i( p_sock, l_data ); 46 p_data := utl_raw.cast_to_raw( l_data ); 47 end getline;

Appendix A

1252

Again, much like RECV_I/RECV above, GETLINE_I is an internal function called only by GETLINE. The external PL/SQL interface exposes all data as the RAW type, and the GETLINE function here simply converts the VARCHAR2 data into a RAW for us.

48 49 procedure disconnect( p_sock in number ) 50 as language java 51 name 'jsock.java_disconnect( int )'; 52 53 procedure get_gmt( p_gmt out varchar2 ) 54 as language java 55 name 'jsock.java_get_gmt( java.lang.String[] )'; 56 57 procedure get_timezone( p_timezone out varchar2 ) 58 as language java 59 name 'jsock.java_get_timezone( java.lang.String[] )'; 60 61 procedure get_hostname( p_hostname out varchar2 ) 62 as language java 63 name 'jsock.java_get_hostname( java.lang.String[] )'; 64 65 function peek( p_sock in number ) return number 66 as language java 67 name 'jsock.java_peek_sock( int ) return int'; 68 69 procedure b64encode( p_data in raw, p_result out varchar2 ) 70 as language java 71 name 'jsock.b64encode( byte[], java.lang.String[] )'; 72 end; 73 /


We are now ready to test some of our functions to see that they are installed, and actually work:

tkyte@TKYTE816> declare 2 l_hostname varchar2(255); 3 l_gmt varchar2(255); 4 l_tz varchar2(255); 5 begin 6 simple_tcp_client.get_hostname( l_hostname ); 7 simple_tcp_client.get_gmt( l_gmt ); 8 simple_tcp_client.get_timezone( l_tz ); 9 10 dbms_output.put_line( 'hostname ' || l_hostname ); 11 dbms_output.put_line( 'gmt time ' || l_gmt ); 12 dbms_output.put_line( 'timezone ' || l_tz ); 13 end; 14 / hostname tkyte-dell gmt time 20010131 213415 GMT timezone EST


UTL_TCP

1253

An important point for running the TCP/IP components of this package is that we need special permission to use TCP/IP in the database. For more information on the DBMS_JAVA package and privileges associated with Java, please see the DBMS_JAVA section in this appendix. In this case, we specifically we need to execute:

sys@TKYTE816> begin 2 dbms_java.grant_permission( 3 grantee => 'TKYTE', 4 permission_type => 'java.net.SocketPermission', 5 permission_name => '*', 6 permission_action => 'connect,resolve' ); 7 end; 8 /


Refer to the section on DBMS_JAVA for more details on what, and how, this procedure works. In a nutshell, it allows the user TKYTE to create connections and resolve hostnames to IP addresses to any host (that's the '*' above). If you are using Oracle 8.1.5, you will not have the DBMS_JAVA package. Rather, in this version you would grant the JAVASYSPRIV to the owner of jsock. You should be aware that the JAVASYSPRIV is a very 'broad' privilege. Whereas DBMS_JAVA.GRANT_PERMISSION is very granular, JAVASYSPRIV is very broad, and conveys a lot of privileges at once. Now that I have this permission, we are ready to implement and test our SocketType, similar to the way we tested in which we tested UTL_TCP initially. Here is the body of SocketType. The type body contains very little actual code, and is mostly a layer on the SIMPLE_TCP_CLIENT package we just created. It hides the 'socket' from the caller:

tkyte@TKYTE816> create or replace type body SocketType 2 as 3 4 static function crlf return varchar2 5 is 6 begin 7 return chr(13)||chr(10); 8 end; 9 10 member function peek return number 11 is 12 begin 13 return simple_tcp_client.peek( g_sock ); 14 end; 15 16 17 member procedure send( p_data in varchar2 ) 18 is 19 begin 20 simple_tcp_client.send( g_sock, utl_raw.cast_to_raw(p_data) ); 21 end; 22 23 member procedure send_raw( p_data in raw ) 24 is 25 begin 26 simple_tcp_client.send( g_sock, p_data ); 27 end; 28 29 member procedure send( p_data in clob )

Appendix A

1254

30 is 31 l_offset number default 1; 32 l_length number default dbms_lob.getlength(p_data); 33 l_amt number default 4096; 34 begin 35 loop 36 exit when l_offset > l_length; 37 simple_tcp_client.send( g_sock, 38 utl_raw.cast_to_raw( 39 dbms_lob.substr(p_data,l_amt,l_offset) ) ); 40 l_offset := l_offset + l_amt; 41 end loop; 42 end;

The SEND routine is overloaded for various data types, and takes a CLOB of arbitrary length. It will break the CLOB into 4 KB chunks for transmission. The SEND_RAW routine below is similar, but performs the operation for a BLOB:

43 44 member procedure send_raw( p_data in blob ) 45 is 46 l_offset number default 1; 47 l_length number default dbms_lob.getlength(p_data); 48 l_amt number default 4096; 49 begin 50 loop 51 exit when l_offset > l_length; 52 simple_tcp_client.send( g_sock, 53 dbms_lob.substr(p_data,l_amt,l_offset) ); 54 l_offset := l_offset + l_amt; 55 end loop; 56 end; 57 58 member function recv return varchar2 59 is 60 l_raw_data raw(4096); 61 begin 62 simple_tcp_client.recv( g_sock, l_raw_data ); 63 return utl_raw.cast_to_varchar2(l_raw_data); 64 end; 65 66 67 member function recv_raw return raw 68 is 69 l_raw_data raw(4096); 70 begin 71 simple_tcp_client.recv( g_sock, l_raw_data ); 72 return l_raw_data; 73 end; 74 75 member function getline( p_remove_crlf in boolean default FALSE ) 76 return varchar2 77 is 78 l_raw_data raw(4096); 79 begin

UTL_TCP

1255

80 simple_tcp_client.getline( g_sock, l_raw_data ); 81 82 if ( p_remove_crlf ) then 83 return rtrim( 84 utl_raw.cast_to_varchar2(l_raw_data), SocketType.crlf ); 85 else 86 return utl_raw.cast_to_varchar2(l_raw_data); 87 end if; 88 end; 89 90 member procedure initiate_connection( p_hostname in varchar2, 91 p_portno in number ) 92 is 93 l_data varchar2(4069); 94 begin 95 -- we try to connect 10 times and if the tenth time 96 -- fails, we reraise the exception to the caller 97 for i in 1 .. 10 loop 98 begin 99 g_sock := simple_tcp_client.connect_to( p_hostname, p_portno ); 100 exit; 101 exception 102 when others then 103 if ( i = 10 ) then raise; end if; 104 end; 105 end loop; 106 end;

We try the connection ten times in order to avoid issues with 'server busy' type messages. It is not entirely necessary, but makes it so the caller doesn't get errors as often as it otherwise might on a busy web server, or some other service.

107108 member procedure close_connection 109 is 110 begin 111 simple_tcp_client.disconnect( g_sock ); 112 g_sock := NULL; 113 end; 114115 end; 116 /

Type body created.

As you can see, these are mostly convenience routines layered on top of SIMPLE_TCP_CLIENT to make this package easier to use. It also serves as a nice way to encapsulate the functionality of the SIMPLE_TCP_CLIENT in an object type. Using SocketType instead of UTL_TCP, our simple 'get a web page via a proxy' routine looks like this:

Appendix A

1256

tkyte@TKYTE816> declare 2 s SocketType := SocketType(null); 3 buffer varchar2(4096); 4 BEGIN 5 s.initiate_connection( 'proxy-server', 80 ); 6 s.send( 'GET http://www.oracle.com/ HTTP/1.0'||SocketType.CRLF); 7 s.send( SocketType.CRLF); 8 9 loop 10 buffer := s.recv; 11 exit when buffer is null; 12 dbms_output.put_line( substr( buffer,1,255 ) ); 13 end loop; 14 s.close_connection; 15 END; 16 / HTTP/1.1 200 OK Date: Thu, 01 Feb 2001 00:16:05 GMT Server: Apache/1.3.9 (Unix) mod_perl/1.21 ApacheJServ/1.1yyyyyyyyyy: close Content-Type: text/html

<head><title>Oracle Corporation</title>

This code is not radically different from using UTL_TCP directly, but it does show how encapsulating your packages with an Object Type can add a nice feeling of object-oriented programming to your PL/SQL. If you are a Java or C++ programmer, you feel very comfortable with the above code, declaring a variable of type SocketType and then calling methods against that type. This is as opposed to declaring a variable of some record type that you pass down to each routine as UTL_TCP does. The above is more object-oriented than the procedural method first shown.

Summary In this section we looked at the new functionality provided by the UTL_TCP package. We also investigated an alternative implementation in Java. Additionally, we packaged this functionality into a new Object Type for PL/SQL, fully encapsulating the capabilities of the TCP/IP socket nicely. We saw how easy it is to integrate networking functionality into our PL/SQL applications using this facility, and in an earlier section on UTL_HTTP, we saw how we can make use of this to provide full access to the HTTP protocol.

Index

A Guide to the Index The index is arranged hierarchically, in alphabetical order, with symbols preceding the letter A. Most second-level entries and many third-level entries also occur as first-level entries. This is to ensure that users will find the information they require however they choose to search for it.

Symbols 1023 byte limit

overloaded FOPEN function gets around byte limit, 1205

UTL_FILE API, 1205 2PC protocol

see two-phase distributed commit protocol.

A ABOUT operator

interMedia Text, 756 accessing adjacent rows

analytic functions, 582 LAG function, 583 LEAD function, 583

ACID properties Atomicity, 135 Consistency, 135 Durability, 135 Isolation, 135 transactional mechanisms, 135

ADD_POLICY routine DBMS_RLS package, 921 parameters, 921

ADDRESS = (PROTOCOL = IPC) (KEY = EXTPROC1) LISTENER.ORA file, 775 TNSNAMES.ORA file, 776

address space isolation dedicated server, 86

administration burden reducing using partitioning, 629

advanced Oracle features, importance of, 44 Advanced Queues

EMNn, 96 Oracle database, 26, 44 QMNn, 96

AFTER trigger compared to BEFORE trigger, 169

aggregate compatibility materialized views, 604

ALL_OUTLINE_HINTS table optimizer plan stability, 517

ALL_OUTLINES table optimizer plan stability, 516

ALTER ANY OUTLINE privilege stored outlines, creating, 519

ALTER INDEX statement synchronizing text index, 754

ALTER OUTLINE command DDL, 525

ALTER SESSION command ALTER SESSION SET CREATE_STORED_OUTLINE

command, 520 ALTER SESSION SET USE_STORED_OUTLINE

command, 513 auto binding, 520 implementing tuning, 510 optimizer plan stability, problems with, 533 stored outlines, creating, 520 syntax, 520

ALTER SESSION SET CREATE_STORED_OUTLINE command stored outlines, creating, 520

ALTER SESSION SET USE_STORED_OUTLINE command development tool, 515 implementing tuning, 513 optimizer plan stability, 513, 515

ALTER TYPE command object methods, 875

ALTER USER command GRANT CONNECT THROUGH clause

n-Tier authentication, 965, 973 security privileges, 975 syntax, 975

REVOKE CONNECT THROUGH clause, 975 analytic functions, 545

advantages over standard SQL, 548, 571, 585 avoiding fragile code, 573

examples, 566 accessing adjacent rows, 582 pivot queries, 576

generic pivot queries, 578 TOP-N queries, 566

function types AVG function, 562 CORR function, 563 COUNT function, 563 COVAR_POP function, 563 COVAR_SAMP function, 563 CUME_DIST function, 563 DENSE_RANK function, 563 FIRST_VALUE function, 563 LAG function, 550, 564 LAST_VALUE function, 564

analytic functions (Continued)

1266

analytic functions (Continued) function types (Continued)

LEAD function, 550, 564 MAX function, 564 MIN function, 564 NTILE function, 564 PERCENT_RANK function, 564 RANK function, 564 ranking functions, 550 RATIO_TO_REPORT function, 565 REGR_ functions, 565 reporting functions, 550 ROW_NUMBER function, 565 statistical functions, 551 STDDEV function, 565 STDDEV_POP function, 565 STDDEV_SAMP function, 565 SUM function, 565 VAR_POP function, 565 VAR_SAMP function, 565 VARIANCE function, 565 windowing functions, 550

performance testing, 571, 583 SQL_TRACE, 571 TIMED_STATISTICS, 571 TKPROF, 571

problems with, 586 NULLS, 588 PL/SQL, using with, 586 tuning, 590 Where clause, 588

syntax, 549 function clause, 550 ORDER BY clause, 551 partition clause, 551 windowing clause, 553

BETWEEN, 561 CURRENT ROW, 560 Numeric Expression FOLLOWING, 560 Numeric Expression PRECEDING, 560 range windows, 555 row windows, 558 UNBOUNDED PRECEDING, 560

ANALYZE_DATABASE function DBMS_UTILITY package, 1179 should not be used, not ever, 1179

ANALYZE_SCHEMA function DBMS_UTILITY package, 1176 limitations, 1178 using with changing schema, 1178

ANSI compliant databases differences between, 38 Oracle database, 38 SQL Server, 38 Sybase, 38

API-based database programming dynamic SQL, 700

application context bind variables, 716, 919 creating application context, 929 Fine Grained Access Control, 913, 918 global variables, 931 implementing security policy with FGAC, 923

testing, 934 namespace variables, 919 native dynamic SQL, 716 security policy, 930 setting application context, 927

ON LOGON database trigger, 927 SYS_CONTEXT function, 716 testing, 930 trustworthiness of application context, 929

application development, simplifying Fine Grained Access Control, reasons for using, 916

application domain indexes, 271, 297 interMedia Text indexes, 297

application server application server session, 970 n-Tier authentication, 968

Application Server Providers see ASPs.

application server session client session, 970 n-Tier authentication, 970

APPNAME record fields, meaning of, 465 trace files, using and interpreting, 465

AQsee Advanced Queues.

Archive Process see ARCn.

archived redo log files, 69, 158 ARCHIVELOG mode, 69

compared to NOARCHIVELOG mode, 69 ARCn

focused background processes, 94 Array interface, getArray() method

manipulating arrays, 854 array processing

DBMS_SQL, 726 native dynamic SQL, 728

ARRAY type, Java passed from Oracle DATEARRAY type

Java stored procedures, 850 passed from Oracle NUMARRAY type

Java stored procedures, 850 passed from Oracle STRARRAY type

Java stored procedures, 850 printing metadata about array types, 854

arrays manipulating arrays, 854 passing OUT parameter using arrays, 852

ASPs hosting applications as ASP, 917

atomic instructions latches, 122

Atomicity Oracle transactions, 136 statement level atomicity, 136

Oracle database, 137 PL/SQL blocks, 138 triggers, 138

transactional mechanisms, 135 AUDIT command

auditing attempts to modify secure information, 662 BY <proxy> clause, 976 n-Tier authentication, 976 ON BEHALF OF ALL clause, 976 ON BEHALF OF clause, 976

authentication n-Tier authentication, 963 Operating System authentication, 969 password authentication, 969 web-based authentication, 964

auto binding ALTER SESSION command, 520 cursor_sharing=force parameter, 533

automated query rewrite facility materialized views, brief history, 594

blocking

1267

autonomous transactions, 659 auditing attempts to modify secure information, 662 compared to DBMS_JOB package, 666 compared to non-autonomous transactions, 660 ending autonomous transactions, 686 errors

ORA-00060, 696 ORA-06519, 695 ORA-14450, 695

modular code, 678 mutating tables, avoiding, 665 Oracle database, 44 PRAGMA AUTONOMOUS_TRANSACTION directive, 660 problems with, 689

distributed transactions, 689 mutating tables, 693 require PL/SQL, 689 rollback, 690 temporary tables, 691

quick example, 660 reasons for using, 662 savepoints, 687 scope, 681

database changes, 682 locks, 685 packaged variables, 681 session state, 682

transactional control flow, 679 trasactional control flow

nesting autonomous transactions, 680 triggers, 663

performing DDL, 666 potential failure with, 668

trigger reads from table in query, 663 writing to database from SQL functions, 670

really strict auditing, 671 using only SELECT statements, 675

care in using, 676 auto-sectioning

interMedia Text, 761 AUTOTRACE

controlling report, 15 execution plan, 15 generating report, 14 setting up, 14

availability increasing using partitioning, 628

AVG function analytic functions, 562

BB*Tree Cluster indexes, 270 B*Tree indexes, 270, 271

branch blocks, 271 commonly used, 271 compared to bitmap indexes, 287 compressing index, 272 leaf nodes, 271 reasons for using, 277 related to reverse key indexes, 275

background processes database architecture, 55, 85, 90 focused background processes, 90 UNIX-based systems, 55 utility background processes, 95 Windows-based systems, 55

backup EXP/IMP tools

role in backup, 340 unsuitable for main backup, 340

Large pool, 83 BAD file

using with SQLLDR, 383, 385, 424 BEFORE trigger

compared to AFTER trigger, 169 increases Redo, 169

benchmarking benchmarking in isolation, 434 benchmarking to scale, 435 tuning, 434

BETWEENwindowing clause, 561

BFILE, 821 LOB_IO procedure, 824 reading file using, 393

BigDecimal type, Java passed from Oracle Number type

Java stored procedures, 849 Binary File

see BFILE. Binary Large Object type

see BLOB type. BIND record

fields, meaning of, 469 trace files, using and interpreting, 469

bind variables application context, 716, 919 collection types, using

querying from PL/SQL routine, 892 cursor_sharing=force parameter, 441 description, 27 determining if bind variables being used, 449 dynamic PL/SQL, 714 dynamic SQL, 29 latch free event, 439 Oracle applications, 27 over binding, 446 scalability, 27, 436

testing with multiple users, 436 Shared pool, 81 static SQL, 440 tuning, 436 using new data types as bind variables, 870

bitmap indexes, 270, 285 compared to B*Tree indexes, 287 reasons for using, 286

BLOB type, 821 BLOB to VARCHAR2 conversions, 1077 LOB_IO procedure, 824 storing VARRAYS as, 880 writing BLOB to disk, 1088

block buffer cache KEEP pool, 80 RECYCLE pool, 80 SGA, 78

block server process see BSP.

blocking DELETE statement, 107 INSERT statement, 107 locking policy, 107 SELECT FOR UPDATE statement, 107 UPDATE statement, 107

blocks

1268

blocks block buffer cache, 78 cleanout of, 177

Commit clean out, 177 design issues, 179 ORA-01555, 185, 191

necessary conditions for error, 192 rarity of error, 194 solutions to error, 195

extents, 62 branch blocks

B*Tree indexes, 271 breakable parse locks, 119

DBA_DDL_LOCKS view, 120 stored procedures and, 121 using, 121

BSP focused background processes, 95

BSTAT/ESTAT replaced by StatsPack, 477

BY <proxy> clause AUDIT command, 976

byte type, Java passed from Oracle RAW type

Java stored procedures, 851

CC code

C-based stored procedures, first example, 794 error codes, 801 functions that return values, 813 mapping Oracle data types to C data types, 802

OCICollAppend API call, 809 OCICollGetElem API call, 809 OCICollSize API call, 809 OCINumberFromReal API call, 802

n-Tier authentication, 966 OCI programs, 967 portability macro, 802 Pro*C code, 823 strtok function, 967 template for external procedures, 794

debugf function, 795 debugf macro, 796 global context, 794 init function, 798 lastOCiError function, 797 raise_application_error function, 797 term function, 801

C compiler, 15 C data types

mapping Oracle data types to C data types, 787, 802 table mapping external types to C types, 793

CASCADE option DROP USER command, 533

cascade update deferring constraint checking, 141

case sensitivity optimizer plan stability, problems with, 531

case statement, SQL, 382 CATEGORY column

DBA_OUTLINES table, 516 C-based stored procedures, 771

compared to Java stored procedures, 848, 857

configuring server, 775 LISTENER.ORA file, 775 ORA-28575 error, 776 TNSNAMES.ORA file, 775, 776

errors, 833 ORA-06520, 837 ORA-06521, 837 ORA-06523, 838 ORA-06525, 839 ORA-06526, 839 ORA-06527, 840 ORA-28575, 833 ORA-28576, 834 ORA-28577, 834 ORA-28578, 835 ORA-28579, 835 ORA-28580, 836 ORA-28582, 836

first example, 783 C code, 794

error codes, 801 functions that return values, 813 mapping Oracle data types to C data types, 802 portability macro, 802 template for external procedures, 794

installing and running procedure, 819 makefiles, 817

building extproc.dll, 817 building extproc.so, 819 porting extproc to UNIX, 818

PL/SQL prototype, 784 creating library object, 786 creating package body, 786 data type functions, 784 mapping Oracle data types to C data types, 787

implementing, 773 EXTPROC OS process, 773

LOB_IO procedure installing and running procedure, 829 LOBS, manipulating, 821 makefiles, 827 Pro*C code, 823

LOBS, manipulating, 821 LOB_IO procedure, 821

makefiles, 816 reasons for using, 772 template for external procedures, 783, 794

debugf function, 795 debugf macro, 796 error handling, 784 global context, 794 init function, 798 lastOCiError function, 797 parameter setting, 784 raise_application_error function, 797 state management, 783 term function, 801 tracing mechanisms, 783

testing installation, 779 DEMOLIB library, creating, 781 demonstration program, 779 extproc.c code, compiling, 780 installing and running PL/SQL code, 782 SCOTT/TIGER account, setting up, 780

UTL_TCP package, 772 CBO

FULL TABLE SCAN, 282 function based indexes, 288 materialized views, 595 temporary tables and, 255

chained rows LogMiner packages, limitations, 1116

CREATE CLUSTER statement

1269

Character Large Object type see CLOB type.

Checkpoint Process see CKPT.

checkpointing online redo log files, 67

CKPT focused background processes, 93

client application statements trace files, using and interpreting, 471

client session application server session, 970 n-Tier authentication, 969

CLOB type, 821 Java stored procedures, 850 LOB_IO procedure, 824 using Java input/output stream types with, 853 writing CLOB to disk, 1088

CLOSE function FILETYPE object type, 885

cluster index index clustered tables, 226

clustered tables hash clustered tables, 198, 231 index clustered tables, 197, 224

CLUSTERING_FACTOR column USER_INDEXES view, 283

coding conventions, 16 collection types, using

array fetch into tables, 892, 895 inserting records, 892, 896 object relational features, 892 querying from PL/SQL routine, 892

bind variables, 892 comma separated values

see CSV format. COMMA_TO_TABLE function

DBMS_UTILITY package, 1191 command line options

TKPROF, 462 Commit clean out

block cleanout, 177 forcing cleanout, 177

COMMIT statement cannot COMMIT over database link, 149 compared to rollback, 163 description, 158 flat response time operation, 160 increases Redo, 168 LGWR, 160 ORA-01555, 185, 190 SCN, generating, 160 synchronous calls to LGWR from COMMIT, 160 transaction control statements, 136

COMPATIBLE parameter materialized views, setting up, 602

COMPILE_SCHEMA function DBMS_UTILITY package, 1172

complex data types Java stored procedures, 849 JPublisher, 849

composite partitioning, 633, 636 COMPRESS N option

compared to NOCOMPRESS option, 220 CREATE TABLE statement

index organized tables, 217 Indexes, 217

concurrency control definition, 123 importance of, 30 Informix, 102 locking policy, 30, 102 multi-versioning, 33 Oracle database, 30, 103 Sybase, 102 transaction isolation levels, 124

dirty read, 124 non-repeatable read, 124 phantom read, 124 read committed isolation level, 124, 126 read only isolation level, 125, 132 read uncommitted isolation level, 124, 125 repeatable read isolation level, 124, 128 serializable isolation level, 124, 130

CONNECT_DATA = (SID = PLSExtProc) TNSNAMES.ORA file, 776

Consistency transactional mechanisms, 135

consistent answers, obtaining repeatable read isolation level, 128

constraints care in using, 609 integrity constraints, 140 materialized views, 605 metadata, 607

CONTEXT index type interMedia Text, 749

ConText Option interMedia Text, brief history, 740

control file, SQLLDR, 371 command line overrides control file, 426 simple example, 371

control files database architecture, file types, 60, 66

cooked file systems compared to RAW devices, 181

CORR function analytic functions, 563

corruption, detecting EXP tool, reasons for using, 319

Cost-Based Optimizer see CBO.

COUNT function analytic functions, 563 generic pivot queries, 578 TOP-N queries, 568

COVAR_POP function analytic functions, 563

COVAR_SAMP function analytic functions, 563

covert channel referential integrity, 941

CREATE ANY CONTEXT privilege DBMS_RLS package, 918

CREATE ANY OUTLINE privilege optimizer plan stability, 508 stored outlines, creating, 519

CREATE CLUSTER statement compared to CREATE TABLE statement, 225 hash clustered tables, 232

HASHKEYS option, 232 single table hash cluster, 238 size parameter, 232

index clustered tables, 225 size parameter, 226

CREATE INDEX statement

1270

CREATE INDEX statement function based indexes, 289 substr function, 293 synchronizing text index, 754

CREATE LIBRARY privilege SCOTT/TIGER account, setting up, 781

CREATE LIBRARY statement DEMOLIB library, creating, 781

CREATE OR REPLACE OUTLINE command CREATE ANY OUTLINE privilege, 508 optimizer plan stability, 508

CREATE TABLE statement compared to CREATE CLUSTER statement, 225 heap organized tables, 210

FREELIST, 211 INITRANS parameter, 212 PCTFREE parameter, 212 PCTUSED parameter, 212

IMP tool, problems with, 360 increasing complexity of, 360 index clustered tables, 226 index organized tables, 216

COMPRESS N option, 217 INCLUDING option, 220, 222 NOCOMPRESS option, 217 OVERFLOW option, 220, 222 PCTTHRESHOLD parameter, 217, 222

multi-tablespace objects, 361 nested tables, 241 object tables, 258, 261

CREATE TYPE statement data types, adding, 869

CSV format delimited data, 374

CTXCAT catalog index type interMedia Text, 749 interMedia Text, problems with, 765

ctxsrv program synchronizing text index, 753

CUME_DIST function analytic functions, 563

CURRENT ROW windowing clause, 560

cursor caching description, 949 Fine Grained Access Control, problems with, 945

CURSOR function relational tables and, 908

cursor record fields, meaning of, 466 trace files, using and interpreting, 466

cursor variables generic pivot queries, 582 native dynamic SQL, 711 OPEN FOR clause, 711

cursor_sharing=force parameter auto binding, 533 bind variables, 441

over binding, 446 improving performance with, 48 limitations, 441 optimizer plan stability, problems with, 533 problems with, 48, 442

optimizer related issues, 443 query output issues, 447

tuning, 441 customized access control

compared to Fine Grained Access Control, 991 invoker rights, reasons for using, 991

DData Definition Language

see DDL. data dictionary applications

index clustered tables, 229 invoker rights, reasons for using, 988 problems with, 989

data files database architecture, file types, 60, 62 SYSTEM data, 62 tablespaces, 64 USER data, 62

data loading, 367 load dates, 379 loading into LOBS, 409

DBMS_LOB package, 409 Directory object, 410 LOBFILEs, 414 PL/SQL, 409 SQLLDR, 412

loading data into object columns, 416 loading inline data, 413 loading out of line data, 414

OCI programs, 368 SQLLDR, 367

loading data with embedded newlines, 391 loading delimited data, 374 loading fixed format data, 377 loading into LOBS, 412 loading into LONG field, 390 loading into LONG RAW field, 389 loading nested tables, 418 loading report-style data, 387 loading tab-delimited data, 375 loading VARRAYS, 418 using functions, 380 using sequences, 380

Data Manipulation Language see DML.

data source management interMedia Text

problems with, 764 using, 743, 745

data sufficiency materialized views, 603

data type functions C-based stored procedures, first example, 784 creating collection types, 784 LOB_IO procedure, 821 mapping Oracle data types to C data types, 787 passing parameters, 784 returning SCALAR types, 785

data type names table of codes for, 722

data types, adding adding member function to type, 872 CREATE TYPE statement, 869 object relational features, using, 869 using new type, 870

using as bind variables, 870 data unloading

unloading in SQLLDR friendly format, 399 unloader PL/SQL utility, 399

data warehousing global index partition, 647

global indexes should be avoided, 651

DBMS_LOB package

1271

data with embedded newlines loading with SQLLDR, 391

converting other character into newline, 391 using FIX attribute, 392 using STR attribute, 398 using VAR attribute, 397

data, copying between platforms EXP tool, reasons for using, 320 IMP tool, reasons for using, 320

database compared to instance, 54 definition, 54

database architecture, 53 background processes, 55, 85, 90

focused background processes, 90 UNIX-based systems, 55 utility background processes, 95 Windows-based systems, 55

dedicated server, 56 file types, 60

control files, 60, 66 data files, 60, 62 parameter files, 60 password files, 60 redo log files, 60, 66 temp files, 60, 65

memory structures, 70 PGA, 70 SGA, 55, 70, 74 UGA, 70

MTS, 57 OPS, 54 Oracle database, 25 server processes, 85

dedicated server, 85 MTS, 85, 86

slave processes, 85, 97 I/O slaves, 97 parallel query slaves, 97

Database Block Writer see DBWn.

database buffer cache online redo log files, 67

database changes scope in autonomous transactions, 682

database dependence not necessarily a problem, 46 vendor-specific features, making use of, 41

database environment, verifying resolving ORA-28575 error, 777 TNSNAMES.ORA file, 778

database independence infeasibility of total independence, 41 Oracle applications, 37 problems with, 37

database link cannot COMMIT over database link, 149 cannot do DDL over database link, 149 cannot issue SAVEPOINT over database link, 150 distributed transactions, 148

database logon triggers see ON LOGON database trigger.

database standards portability and, 41 SQL92 standard, 40

database tables see tables.

datastore object DETAIL_DATASTORE datastore object, 744 DIRECT_DATASTORE datastore object, 743

FILE_DATASTORE datastore object, 745 interMedia Text, 743 URL_DATASTORE datastore object, 746

date mask SQLLDR, load dates, 379

Date type, Oracle manipulating using Timestamp class, 852 passing to Java Timestamp type

Java stored procedures, 850 SQLLDR, load dates, 379

DATEARRAY type, Oracle passing to Java ARRAY type

Java stored procedures, 850 DB_VERSION function

DBMS_UTILITY package, 1193 DBA_DDL_LOCKS view

DDL locks, 120 DBA_OUTLINE_HINTS table

HINT column, 517 JOIN_POS column, 517 NAME column, 517 NODE column, 517 optimizer plan stability, 517 OWNER column, 517 STAGE column, 517

DBA_OUTLINES table CATEGORY column, 516 NAME column, 516 optimizer plan stability, 516 OWNER column, 516 SQL_TEXT column, 517 TIMESTAMP column, 517 USED column, 516 VERSION column, 517

DBMS_ALERT package concurrent signals, 1034 many calls to signal, 1037 necesssary supplied packages, 1031, 1032 repeated calls, 1036

DBMS_APPLICATION_INFO package necesssary supplied packages, 1042 SET_CLIENT_INFO call, 1043 V$SESSION_LONGOPS view, 1045

DBMS_JAVA package compiler options, 1051 dropjava function, 1055 loadjava function, 1055 LONGNAME routine, 1050 necesssary supplied packages, 1050 permission procedures, 1056 SET_OUTPUT procedure, 1055 SHORTNAME routine, 1050

DBMS_JOB package compared to autonomous transactions, 666 custom scheduling, 1068 error messages, 1071 monitoring jobs, 1070 necesssary supplied packages, 1059 one-off jobs, running, 1063 performing DDL in triggers, 666 recurring jobs, scheduling, 1066

DBMS_LOB package conversions, 1077

BLOB to VARCHAR2 conversions, 1077 conversion on the fly, 1086 LONG RAW to LOB conversions, 1081 LONG to LOB conversions, 1081 mass conversion example, 1083

DBMS_LOB package (Continued)

1272

DBMS_LOB package (Continued) GETLENGTH method, 411 LOADFROMFILE function, 409 LOBS

displaying LOBS on Web, 1089 loading LOBS, 1074 loading data into LOBS, 409 manipulating LOBS, 821, 1073

necesssary supplied packages, 1073 READ method, 412 SELECT FOR UPDATE statement, 1075 substr function, 1074 writing BLOB to disk, 1088 writing CLOB to disk, 1088

DBMS_LOCK package necesssary supplied packages, 1092 user-defined locks, 123

DBMS_LOGMNR package see LogMiner packages.

DBMS_LOGMNR_D package see LogMiner packages.

DBMS_OBFUSCATION_TOOLKIT package encryption, 1123 key management, 1140 necesssary supplied packages, 1123 problems with, 1139 wrapper, 1125

DBMS_OLAP package ESTIMATE_SUMMARY_SIZE routine, 618 materialized views, 618

editing views, 618, 622 estimating size of view, 618 evaluating utilization of view, 618 validating dimension objects, 618, 620

RECOMMEND_MV routine, 622 RECOMMEND_MV_W routine, 622 VALIDATE_DIMENSION routine, 621

DBMS_OUTPUT package creating DBMS_OUTPUT functionality, 1154 description, 1145 enhancing with wrapper function, 1153 limitations, 1153 making programming environments DBMS_OUTPUT-

aware, 1149 necesssary supplied packages, 1144 using UTL_FILE instead, 1154

DBMS_PIPE package compared to external procedures, 1040 necesssary supplied packages, 1031, 1038 online example, 1041

DBMS_PROFILER necesssary supplied packages, 1161 problems with, 1171 tuning with, 475 using, 475

DBMS_RLS package ADD_POLICY routine, 921 CREATE ANY CONTEXT privilege, 918 EXECUTE ON DBMS_RLS privilege, 924 EXECUTE_CATALOG_ROLE privilege, 918 Fine Grained Access Control, 918

DBMS_SHARED_POOL package Shared pool, 81

DBMS_SPACE package hash clustered tables, measuring space used by, 232 setting value of PCTFREE parameter, 205 setting value of PCTUSED parameter, 205

DBMS_SQL array processing, 726 compared to native dynamic SQL, 702, 713

bind variables, 714 repeated statements, 725 unknown number of outputs, 718

DESCRIBE_COLUMNS API, 720 DML statements, using with, 707 dynamic PL/SQL, 702 performance testing, 726, 727 PL/SQL blocks, using with, 707 pseudo-code for, 704 structure of processes, 704 using, 703

DBMS_STATS package temporary tables and, 254

DBMS_UTILITY package ANALYZE_DATABASE function, 1179

should not be used, not ever, 1179 ANALYZE_SCHEMA function, 1176

limitations, 1178 using with changing schema, 1178

COMMA_TO_TABLE function, 1191 COMPILE_SCHEMA function, 1172 DB_VERSION function, 1193 FORMAT_CALL_STACK function, 1181 FORMAT_ERROR_STACK function, 1179 GET_HASH_VALUE function, 1193 GET_PARAMETER_VALUE function, 1185 GET_TIME function, 1184 NAME_RESOLVE function, 1186 NAME_TOKENIZE function, 1188 necesssary supplied packages, 1172 PORT_STRING function, 1193 TABLE_TO_COMMA function, 1191

DBWnfocused background processes, 93 online redo log files, 68

DDLALTER OUTLINE command, 525 always commits in Oracle, 119 cannot do DDL over database link, 149 DROP OUTLINE command, 527 extracting

EXP tool INDEXFILE option, 334 reasons for using, 319 SHOW = Y option, 334

scripts, 337 performing in triggers, 666

autonomous transactions, 666 potential failure with, 668

DBMS_JOB package, 666 stored outlines, creating, 519 stored outlines, managing, 525

DDL locks, 112, 119 breakable parse locks, 119

DBA_DDL_LOCKS view, 120 exclusive DDL locks, 119 share DDL locks, 119

deadlocks causes of, 108

foreign keys, 108 locking policy, 108 Oracle Forms and, 109 rarity in Oracle, 108 trace files for, 108

DEBUG package tuning with, 476

documented parameters

1273

debug.f package Fine Grained Access Control, debugging, 955

debugf function template for external procedures, 795

debugf macro template for external procedures, 796

dedicated server address space isolation, 86 advantages, 87 compared to MTS, 57, 87 creating, 58 database architecture, 56 listener, 58 Net8 listener, 86 remote execution, 86 server processes, 85 SQL*PLUS, 86 usually preferred option, 89

DEFERRABLE constraint INITIALLY IMMEDIATE constraint, 142 integrity constraints, 142

definer definition, 981

definer rights compared to invoker right, 983, 994 description, 995 directly granted security privileges, 996

documentation concerning, 996 errors, 1025 Java stored procedures and, 1021 reasons for using

scalability, 994 security logic in database, 995

roles not enabled, 996, 999 static behavior, 984 stored procedures and, 982

compiling stored procedure, 998 setting up dependencies, 998 verifying accessed objects, 998

DELETE statement blocking, 107 generating Undo, 185 nested tables, 245

delimited data CSV format, 374 loading

SQLLDR, 374 tab-delimited data, 375

DEMOLIB library, creating C-based stored procedures, testing, 781 CREATE LIBRARY statement, 781

demonstration program C-based stored procedures, testing, 779

DENSE_RANK function analytic functions, 563 TOP-N queries, 567

dependency chain, breaking dynamic SQL, problems with, 734

descending indexes, 270, 276 DESCRIBE_COLUMNS API

DBMS_SQL, 720 using, 720

DETAIL_DATASTORE datastore object, 744 deterministic keyword

function based indexes, 292

development tool ALTER SESSION SET USE_STORED_OUTLINE command, 515 ON LOGON database trigger, 514 optimizer plan stability, 514

dictionary cache StatsPack report, interpreting, 492

dictionary-managed tablespace, 64 recursive SQL, 64

dimensions creating, 613 DBMS_OLAP package

validating dimension objects, 618, 620 description, 609 hierarchies and, 616

direct path exports EXP tool, 345

DIRECT_DATASTORE datastore object, 743 directly granted security privileges

definer rights, 996 invoker rights, 1006

directory listing, obtaining Java stored procedures, 858 security privileges, 859 SQLJ, 859

Directory object loading data into LOBS, 410

dirty read not supported by Oracle, 125 transaction isolation levels, 124

dirty read isolation level concurrency control, 124

dispatcher process connecting to, 59 MTS, 57

Distributed Database Recovery see RECO.

distributed locks, 112 distributed transactions, 148

autonomous transactions, problems with, 689 database link, 148

cannot COMMIT over database link, 149 cannot do DDL over database link, 149 cannot issue SAVEPOINT over database link, 150

limitations, 149 two-phase distributed commit protocol, 149

DML locks, 112, 113 TM locks, 118 TX locks, 113

DML performance enhancing using partitioning, 630

parallel DML, 630 DML statements

DBMS_SQL, using with, 707 DMP file

see dump file. document filters

interMedia Text, 743 document management

interMedia Text problems with, 762 using, 743

document services interMedia Text, problems with, 764

documented parameters parameter files, 61

DROP ANY OUTLINE privilege

1274

DROP ANY OUTLINE privilege stored outlines, creating, 519

DROP OUTLINE command DDL, 527

DROP USER command CASCADE option, 533 optimizer plan stability, problems with, 533

DROP_BY_CAT procedure OUTLN_PKG package, 529

DROP_UNUSED procedure OUTLN_PKG package, 529

dropjava function DBMS_JAVA package, 1055

dump file EXP tool, 319 IMP tool, 319

duplicate row IDs index clustered tables, 229

Durability transactional mechanisms, 135

dynamic SQL, 699 API-based database programming, 700 bind variables, 29 compared to static SQL, 700 generic pivot queries, 579 PL/SQL, 699

bind variables, 714 DBMS_SQL, 702

required if output number unknown, 718 repeated statements, 725 unknown number of outputs, 718

problems with, 734 dependency chain, breaking, 734 fragile code, 735 tuning, 735

reasons for using, 702 UTL_FILE API, 707, 723

Ee-mail, sending and receiving

Java stored procedures, 844 EMNn

Advanced Queues, 96 utility background processes, 96

encapsulation object methods, reasons for using, 873

encryption DBMS_OBFUSCATION_TOOLKIT package, 1123 key management, 1140

ENFORCED value QUERY_REWRITE_INTEGRITY parameter, 602

enqueues see internal locks.

environment, setting up, 12 error codes

C-based stored procedures, first example, 801 LOB_IO procedure, 822

error handling invoker rights, problems with, 1015 Java stored procedures, 845, 863 lastOCiError function, 797 ORA-29459 Java Session State cleared, 864 ORA-29531 no method X in class Y, 864 permissions errors, 864 raise_application_error function, 797 template for external procedures, 784

error messages DBMS_JOB package, 1071

ESTIMATE_SUMMARY_SIZE routine DBMS_OLAP package, 618

Event Monitor Processes EMNn.

evolutionary application development Fine Grained Access Control, reasons for using, 916

exception handling UTL_FILE API, 1202

exclusive DDL locks, 119 EXEC record


EXECUTE IMMEDIATE clause native dynamic SQL, 709 syntax, 709

EXECUTE ON DBMS_RLS privilege DBMS_RLS package, 924

EXECUTE ON OUTLN_PKG privilege stored outlines, creating, 519

EXECUTE phase TKPROF, analyzing query, 458

EXECUTE_CATALOG_ROLE privilege DBMS_RLS package, 918

EXP tool, 317 DDL, extracting, 334

INDEXFILE option, 334 SHOW = Y option, 334

direct path exports, 345 dump file, 319 Fine Grained Access Control, problems with, 951, 952 HELP = Y option, 321 large exports, 325

exporting smaller pieces, 326 exporting to OS pipe, 327 exporting to tape device, 328 FILESIZE parameter, 325 problems with, 325

name-value pairs, 321 parameters, 321

table of parameters, 322 problems with, 345

cloning schemas, 346 disappearance of indexes, 353 National Language Support, 359 using across different Oracle versions, 353

quick example, 318 reasons for using, 319

cloning schemas, 319 copying data between platforms, 320 detecting corruption, 319 extracting DDL, 319 rebuilding instances, 320 transporting tablespaces, 320

role in backup, 340 subsetting data, 328

QUERY= parameter, 328 transporting data, 329

limitations, 329 read only data, 333 tablespaces, 330, 331

unsuitable as reorganization tool, 340 unsuitable for main backup, 340

explain plan facility problems with, 462 TKPROF, analyzing query, 462

full exact text match

1275

extensible indexing see application domain indexes.

extents blocks, 62 segments, 62

external data types table mapping external types to C types, 793 table mapping SQL types to external types, 792

external procedure errors interMedia Text, 767

external procedures C-based stored procedures, 771 compared to DBMS_PIPE package, 1040

EXTPROC OS process C-based stored procedures, implementing, 773 Net8 listener, 773 verifying program

resolving ORA-28575 error, 777 extproc.c code, compiling

C-based stored procedures, testing, 780 compiling in UNIX, 780 compiling in Windows, 780

EXTPROC_CONNECTION_DATA TNSNAMES.ORA file, 776

FFAST FULL SCAN

compared to INDEX RANGE SCAN, 279 Fear Uncertainty and Doubt

about Oracle database, 21 FETCH phase

TKPROF, analyzing query, 458 FGAC

see Fine Grained Access Control. field section

interMedia Text, 758 file system-based solution

compared to interMedia Text, 746 FILE_DATASTORE datastore object, 745 FILESIZE parameter

EXP tool, large exports, 325 FILETYPE object type

CLOSE function, 885 demonstrating functionality, 890 encapsulating UTL_FILE in new PL/SQL data type,

884GET_LINE function, 886 isOpen method, 886 OPEN function, 884, 885 supported by PL/SQL package, 885 WRITE_IO procedure, 884, 887

Fine Grained Access Control, 913 application context, 913, 918

bind variables, 919 implementing security policy with FGAC, 923

testing, 934 namespace variables, 919 security policy, 931

compared to customized access control, 991 DBMS_RLS package, 918 debugging, 955

debug.f package, 955 errors, 955

ORA-28106, 959 ORA-281110, 955 ORA-28112, 957 ORA-28113, 958

ON LOGON database trigger, 918 Oracle database, 47 problems with, 940

cursor caching, 945 EXP tool, 951, 952 IMP tool, 951, 954 referential integrity, 941

covert channel, 941 ON DELETE CASCADE clause, 942 ON DELETE SET NULL clause, 943

reasons for using, 914 application development, simplifying, 916 evolutionary application development, 916 hosting applications as ASP, 917 maintainability, 915 security logic in database, 915 shared accounts, avoiding, 916 single account environment, supporting, 917

security policy, 918 application context, 931 implementing policy with FGAC, 919

simple example, 914 SYS_CONTEXT function, 918

FIRST_VALUE function analytic functions, 563

FIX attribute INFILE clause, 392 loading data with embedded newlines, 392

reading file using BFILE, 393 fixed format data, loading

POSITION keyword, 377 SQLLDR, 377

fixed SGA, 77 focused background processes, database architecture, 90

ARCn, 94 BSP, 95 CKPT, 93 DBWn, 93 LCKN, 95 LGWR, 94 LMD, 95 LMON, 95 PMON, 91 RECO, 93 SMON, 92

FOR UPDATE clause care in using, 23 Oracle database, locking policy, 32

foreign keys deadlocks, causes of, 108 foreign keys and indexes, 302

FORMAT_CALL_STACK function DBMS_UTILITY package, 1181

FORMAT_ERROR_STACK function DBMS_UTILITY package, 1179

fragile code analytic functions, avoided by, 573 dynamic SQL, problems with, 735

FREELIST CREATE TABLE statement

heap organized tables, 211 tables, syntax, 200

FROMUSER option IMP tool, 319

full exact text match materialized views, 603

FULL TABLE SCAN

1276

FULL TABLE SCAN CBO, 282 compared to TABLE ACCESS BY INDEX ROWID, 279

function based indexes, 270, 288 CBO, 288 CREATE INDEX statement, 289 deterministic keyword, 292 example, 289 GLOBAL QUERY REWRITE privilege, 288 implementing, 288 performance testing, 295 problems with, 296 QUERY REWRITE privilege, 288 substr function, 288

hiding substr call, 295 function clause

analytic functions, syntax, 550

Ggeneric object types

invoker rights, reasons for using, 991 generic pivot queries

analytic functions, 578 COUNT function, 578 cursor variables, 582 dynamic SQL, 579 MAX function, 578 ROW_NUMBER function, 578

generic utilities, developing invoker rights, reasons for using, 985 problems with, 985 security policy, 985

GET_HASH_VALUE function DBMS_UTILITY package, 1193

GET_LINE function FILETYPE object type, 886

GET_PARAMETER_VALUE function DBMS_UTILITY package, 1185

GET_TIME function DBMS_UTILITY package, 1184

getallcode.sql script DDL, extracting, 337

getallviews.sql script DDL, extracting, 338

getArray() method, Array interface manipulating arrays, 854

getaview.sql script DDL, extracting, 338

getcode.sql script DDL, extracting, 337

GETLENGTH method DBMS_LOB package, 411

gettrig.sql script DDL, extracting, 339

global context template for external procedures, 794

global index partition, 637, 645 compared to local index partition, 638, 645 data warehousing, 647

global indexes should be avoided, 651 OLTP systems, 651 sliding window implementation, 647

should be avoided, 651 global namespace of outlines

optimizer plan stability, problems with, 539

GLOBAL QUERY REWRITE privilege function based indexes, 288

global variables application context, 931

GRANT CONNECT THROUGH clause ALTER USER command

n-Tier authentication, 965, 973 security privileges, 975 syntax, 975

GRANT CREATE MATERIALIZED VIEW privilege materialized views, 595

GRANT CREATE SESSION privilege materialized views, 595

GRANT CREATE TABLE privilege materialized views, 595

GRANT QUERY REWRITE privilege materialized views, 595

grouping compatibility materialized views, 604

Hhard parsing

queries, 27 hash clustered tables

compared to index clustered tables, 231 CREATE CLUSTER statement, 232

HASHKEYS option, 232 size parameter, 232

description, 231 performance testing, 234

SQL_TRACE, 234 TKPROF, 234

single table hash cluster, 238 space used by, measuring

DBMS_SPACE package, 232 table types, 198, 231

hash partitioning, 633, 635 HASHKEYS option, CREATE CLUSTER statement

hash clustered tables, 232 heap organized tables

compared to index organized tables, 212 CREATE TABLE statement, 210

FREELIST, 211 INITRANS parameter, 212 PCTFREE parameter, 212 PCTUSED parameter, 212

description, 209 table types, 197, 209

HELP = Y option EXP tool, 321 IMP tool, 321

hidden columns invoker rights, problems with, 1018

hierarchies dimensions and, 616

high water mark tables, syntax, 199

HINT column DBA_OUTLINE_HINTS table, 517

hinted views stored outlines, optimizing, 524

hinting mechanism optimizer plan stability, 506

hosting applications as ASP Fine Grained Access Control, reasons for using, 917

Indexes

1277

HTML section searching interMedia Text, 757

HTTP_PKG package enhancing UTL_HTTP package, 1219 SocketType object type, 1220

II/O slaves

database architecture, 97 I/Os logical and physical

StatsPack report, interpreting, 485 IFILE directive

init.ora file, 62 IMP tool, 317

DDL, extracting, 335 dump file, 319 errors

ORA-00959, 364 Fine Grained Access Control, problems with, 951, 954 FROMUSER option, 319 HELP = Y option, 321 importing into different structures, 341

structure with additional column, 341 structure with changed data type, 341

INSTEAD OF trigger, 343 views, 343

structure with missing column, 341 INSTEAD OF trigger, 343 views, 343

name-value pairs, 321 parameters, 323


cloning schemas, 346 CREATE TABLE statement, 361 disappearance of indexes, 353 National Language Support, 359 using across different Oracle versions, 353

quick example, 318 reasons for using, 319

cloning schemas, 319 copying data between platforms, 320 rebuilding instances, 320 transporting tablespaces, 320

role in backup, 340 TOUSER option, 319 transporting data, 329

limitations, 329 read only data, 333 tablespaces, 330, 332

unsuitable as reorganization tool, 340 unsuitable for main backup, 340

implementing tuning ALTER SESSION command, 510

ALTER SESSION SET USE_STORED_OUTLINE command, 513

ON LOGON database trigger, 512 optimizer plan stability, 509 SQL_TRACE, 510 TKPROF, 510

implicit transactions Oracle database, 143

IN parameter Java stored procedures, 849

INCLUDING option compared to PCTTHRESHOLD parameter, 223 CREATE TABLE statement

index organized tables, 220, 222 OVERFLOW option, 223

index clustered tables cluster index, 226 compared to hash clustered tables, 231 CREATE CLUSTER statement, 225

size parameter, 226 CREATE TABLE statement, 226 data dictionary applications, 229 description, 224 duplicate row IDs, 229 reasons not to use, 230 table types, 197, 224

index organized tables, 270 compared to heap organized tables, 212 CREATE TABLE statement, 216

COMPRESS N option, 217 INCLUDING option, 220, 222 NOCOMPRESS option, 217 OVERFLOW option, 220, 222 PCTTHRESHOLD parameter, 217, 222

description, 212 reasons for using, 212 secondary indexes, 224 storage of nested tables, 249 table types, 197, 212

index out of date errors interMedia Text, 767

index partitioning schemes, 637 global index partition, 637, 645

data warehousing, 647 OLTP systems, 651 sliding window implementation, 647

local index partition, 637, 638 local non-prefixed indexes, 638 local prefixed indexes, 638 uniqueness and, 643

sliding window implementation, 638 INDEX RANGE SCAN

compared to FAST FULL SCAN, 279 compared to TABLE ACCESS BY INDEX ROWID, 279

Indexes application domain indexes, 271, 297 B*Tree Cluster indexes, 270 B*Tree indexes, 270, 271

reasons for using, 277 bitmap indexes, 270, 285

reasons for using, 286 cluster index, 226 COMPRESS N option, 217 descending indexes, 270, 276 determining index usage, 308

stored outlines, using, 308 tablespaces, 308

frequently asked questions, 299 foreign keys and indexes, 302 views and indexes, 299

function based indexes, 270, 288 example, 289 implementing, 288 problems with, 296

index organized tables, 270 interMedia Text indexes, 271

Indexes (Continued)

1278

Indexes (Continued) introduction, 269 myths about indexes, 308

most discriminating elements should go first, 311 space never reused in index, 308

NOCOMPRESS option, 217 physical data layout, importance of, 281 reverse key indexes, 270, 275 secondary indexes, 224 unused indexes, reasons for, 303

indexes, disappearance of EXP tool, problems with, 353 IMP tool, problems with, 353

INDEXFILE option compared to SHOW = Y option, 335 DDL, extracting, 334

INDICATOR parameter, 791 INFILE clause

FIX attribute, 392 Informix

concurrency control, 102 locking policy, 102

init function, external procedures template, 798 OCIContextGetValue API call, 798 OCIContextSetValue API call, 799 OCIExtProcGetEnv API call, 798 OCIExtractFromFile API call, 800 OCIExtractInit API call, 800 OCIExtractSetKey API call, 800 OCIExtractSetNumKeys API call, 800 OCIExtractTerm API call, 800 OCIExtractTo API call, 800 OCIFileInit API call, 801 OCIMemoryAllocate API call, 799

init.ora file configuration settings, 61 documented parameters, 61 IFILE directive, 62 parameter files, 60 SGA, 77 undocumented parameters, 61 UTL_FILE_DIR parameter, 1200

INITIAL parameter considered obsolete, 208 tables, syntax, 208

INITIALLY IMMEDIATE constraint integrity constraints, 142

INITRANS parameter CREATE TABLE statement

heap organized tables, 212 tables, syntax, 209 TX locks, 117

INLINE VIEW compared to temporary tables, 254

INSERT statement blocking, 107 generating Undo, 185 nested tables, 245 SQLLDR, 380

instance compared to database, 54 definition, 54 rebuilding

EXP tool, reasons for using, 320 IMP tool, reasons for using, 320

INSTEAD OF trigger importing into structures with changed data type, 343 importing into structures with missing column, 343 object relational views, updating views, 901

INSTR function text search, 741

instrumentation DEBUG package, 476 tuning, 475

int type Java stored procedures, 851 problems with, 856

integrity constraints DEFERRABLE constraint, 142

INITIALLY IMMEDIATE constraint, 142 deferring constraint checking, 141

cascade update, 141 transactional mechanisms, 140 validated after execution, 141

interMedia, 739 interMedia Text, 739

interMedia Text, 739 ABOUT operator, 756 brief history, 739

ConText Option, 740 SQL*TextRetrieval, 739 TextServer3, 740

compared to file system-based solution, 746 CONTEXT index type, 749 creating index, 742 CTXCAT catalog index type, 749 datastore object, 743

DETAIL_DATASTORE datastore object, 744 DIRECT_DATASTORE datastore object, 743 FILE_DATASTORE datastore object, 745 URL_DATASTORE datastore object, 746

description, 749 document filters, 743 external procedure errors, 767 future prospects, 768 index out of date errors, 767 operator object, 750 problems with, 762

CTXCAT catalog index type, 765 data source management, 764 document management, 762 document services, 764 synchronizing index, 763

section searching, 756 auto-sectioning, 761 field section, 758 HTML section searching, 757 section group, 757 XML section searching, 760 zone section, 761

sections object, 749 tables created by, 751 text indexing, 753

stoplists, 755 synchronizing index, 753

using, 740 data source management, 743, 745 document management, 743 text search, 742 theme generation, 747 XML indexing, 749

interMedia Text indexes, 271 application domain indexes, 297

internal locks, 112, 122 see also latches. compared to latches, 122

invoker definition, 981

Large pool

1279

invoker rights compared to definer right, 983, 994 description, 1000 directly granted security privileges, 1006 dynamic behavior, 984 errors, 1025 example, 982 Java stored procedures

invoker rights used by default, 1020 when default behavior matters, 1024

Oracle database, 981 problems with, 1010

error handling, 1015 hidden columns, 1018 parsing performance time, 1013 scalability, 994 SELECT * statements, 1017 shared pool utilization, 1010

reasons for using, 984 customized access control, 991 data dictionary applications, 988 generic object types, 991 generic utilities, developing, 985

roles and, 1006 security privileges, conveying, 1001 stored procedures and, 982


environments for procedures, 1001 table of environments, 1001

PL/SQL pieces, 1000 SQL pieces, 1000

template objects, 1007, 1008 IOTs

see index organized tables. Isolation

transactional mechanisms, 135 isOpen method

FILETYPE object type, 886

JJava data types

mapping Java data types to SQL data types, 847 Java input/output stream types

using with CLOB type, 853 Java pool

MTS, 84 SGA, 76, 83

Java stored procedures, 843 CLOB type, 850 compared to C-based stored procedures, 848, 857 compared to PL/SQL stored procedures, 844 complex data types, 849

JPublisher, 849 Date type passed to Timestamp type, 850 DATEARRAY type passed to ARRAY type, 850 definer rights and, 1021 directory listing, obtaining, 844 e-mail, sending and receiving, 844 error handling, 845, 863

ORA-29459 Java Session State cleared, 864 ORA-29531 no method X in class Y, 864 permissions errors, 864

examples, 857 directory listing, obtaining, 858 getting time down to milliseconds, 863 Operating System, running command or program, 860

IN parameter, 849 int type, 851 invoker rights used by default, 1020

when default behavior matters, 1024 null concept supported, 852 NUMARRAY type passed to ARRAY type, 850 Number type passed to BigDecimal type, 849 Operating System, running command or program, 844 OUT parameter, 849

passing OUT parameter using arrays, 852 parameter setting, 845, 848 PL/SQL calls for, 849 RAW type passed to byte type, 851 reasons for using, 844 scalar types, 851 state management, 845 STRARRAY type passed to ARRAY type, 850 tracing mechanisms, 845 Varchar2 type passed to String type, 850

JavaMail API loading, 1236 using, 1237 UTL_SMTP package, 1236

JDBC API autocommit by default, 148

job queues see SNPn.

join compatibility materialized views, 603

JOIN_POS column DBA_OUTLINE_HINTS table, 517

JPublisher Java stored procedures with complex data types, 849

KKEEP pool

block buffer cache, 80 Shared pool, 82

key management DBMS_OBFUSCATION_TOOLKIT package, 1140 encryption, 1140 keys in client application, 1141 keys in database, 1141 keys in file system, 1142

LLAG function

accessing adjacent rows, 583 analytic functions, 550, 564 syntax, 585

Large pool backup, 83 MTS, 83 parallel execution of statements, 83 RECYCLE pool, 82 SGA, 76, 82

LAST_VALUE function

1280

LAST_VALUE function analytic functions, 564

lastOCiError function template for external procedures, 797

latch free event bind variables, 439

latches, 112, 122 see also internal locks. atomic instructions, 122 compared to internal locks, 122 Oracle database, 27 StatsPack report, interpreting, 491

LCKN focused background processes, 95

LEAD function accessing adjacent rows, 583 analytic functions, 550, 564 syntax, 585

leaf nodes B*Tree indexes, 271

LENGTH parameter, 790 LGWR

COMMIT statement, 160 focused background processes, 94 synchronous calls to LGWR from COMMIT, 160

library object, creating C-based stored procedures, first example, 786 C-based stored procedures, testing, 781 LOB_IO procedure, 821

LIKE operator text search, 741

linguistic analysis theme generation, 747

listener dedicated server, 58 MTS, 59 verifying

LISTENER.ORA file, 779 resolving ORA-28575 error, 779

LISTENER.ORA file ADDRESS = (PROTOCOL = IPC) (KEY = EXTPROC1), 775 C-based stored procedures, configuring server, 775 configuring, 775

Net8 Assistant, 775 plain text editor, 775

listener, verifying, 779 SID_DESC = (SID_NAME = PLSExtProc), 775

listing executed SQL ON LOGON database trigger, 515 optimizer plan stability, 515

listing used indexes ON LOGON database trigger, 515 optimizer plan stability, 515

LMDfocused background processes, 95

LMONfocused background processes, 95

LOADFROMFILE function DBMS_LOB package, 409

loadjava function DBMS_JAVA package, 1055

LOB_IO procedure BFILE, 824 BLOB type, 824 C-based stored procedures, 821 CLOB type, 824

data type functions, 821 error codes, 822 installing and running procedure, 829 library object, creating, 821 LOBS, manipulating, 821 lobToFile library, 822 makefiles, 827

building extproc.dll, 827 building extproc.so, 829

Pro*C code, 823 LOBFILEs

loading data into LOBS, 414 TERMINATED BY EOF statement, 415

LOBS displaying on Web, 1089 loading data into, 409

DBMS_LOB package, 409 Directory object, 410 LOBFILEs, 414 ORDYS.ORDIMAGE object, 416 PL/SQL, 409 SQLLDR, 412

loading data into object columns, 416 loading inline data, 413 loading out of line data, 414

LONG RAW to LOB conversions, 1081 LONG to LOB conversions, 1081 manipulating

C-based stored procedures, 821 DBMS_LOB package, 821, 1073 LOB_IO procedure, 821 NON-POLLING method, 826

lobToFile library LOB_IO procedure, 822 Pro*C code, 823

local index partition, 637, 638 compared to global index partition, 638, 645 local non-prefixed indexes, 638 local prefixed indexes, 638 uniqueness and, 643

local non-prefixed indexes compared to local prefixed indexes, 639

local prefixed indexes compared to local non-prefixed indexes, 639

locally-managed tablespace, 65 lock conversion

compared to lock escalation, 111 Oracle database, 112

lock escalation, 111 compared to lock conversion, 111 not supported by Oracle, 111

Lock Manager Daemon see LMD.

Lock Monitor Process see LMON.

Lock Process see LCKN.

lock promotion see lock conversion.

LOCK TABLE statement manual locking, 123

locking policy blocking, 107 concurrency control, 102 deadlocks, 108 FOR UPDATE clause, 32 Informix, 102 lost update prevention, 104 manual locking, 123

metadata

1281

Locking policy (Continued) non-blocking reads, 31 optimistic locking, 106 Oracle database, 30, 103 pessimistic locking, 105 Sybase, 102 user-defined locks, 123

locks DDL locks, 112, 119

breakable parse locks, 119 exclusive DDL locks, 119 share DDL locks, 119

definition, 102 distributed locks, 112 DML locks, 112, 113

TM locks, 118 TX locks, 113

internal locks, 112, 122 latches, 112, 122 PCM locks, 112 scope in autonomous transactions, 685 shared read locks, 128

log contention causes, 180 redo log files, 180

log file SQLLDR, 372

log switch online redo log files, 67

Log Writer see LGWR.

LogMiner packages analyzing redo, 184 creating Data Dictionary, 1099 discovering when error happened, 1110 limitations, 1113

chained rows, 1116 migrated rows, 1116 object types, 1113

necesssary supplied packages, 1097 options, 1107 PGA and, 1111 using, 1102 V$LOGMNR_CONTENTS table, 1119

LONG field loading into field using SQLLDR, 390 LONG to LOB conversions, 1081

LONG RAW field loading into field using SQLLDR, 389 LONG RAW to LOB conversions, 1081

LONGNAME routine DBMS_JAVA package, 1050

lost update prevention locking policy, 104 optimistic locking, 106 Oracle Forms, 104 pessimistic locking, 105 repeatable read isolation level, 129 serializable isolation level, 129

Mmaintainability

Fine Grained Access Control, reasons for using, 915

make command building extproc.so, 819

makefiles building extproc.dll, 817, 827 building extproc.so, 819, 829 C-based stored procedures, 816 porting extproc to UNIX, 818

manual locking LOCK TABLE statement, 123 locking policy, 123 SELECT...FOR UPDATE statement, 123

MAP methods compared to ORDER methods, 875 description, 874 object methods, 873

materialized views, 593 aggregate compatibility, 604 brief history, 594

automated query rewrite facility, 594 Snapshot, 594 summary table management, 594

CBO, 595 constraints, 605 data sufficiency, 603 DBMS_OLAP package, 618

editing views, 618, 622 estimating size of view, 618 evaluating utilization of view, 618 validating dimension objects, 618, 620

description, 601 dimensions

creating, 613 hierarchies and, 616

full exact text match, 603 grouping compatibility, 604 join compatibility, 603 metadata, 601 partial text match, 603 problems with, 624

OLTP systems, 624 QUERY_REWRITE_INTEGRITY parameter, 624

query rewrite methods, 603 quick example, 595

pre-calculating object count, 598 REFRESH ON COMMIT, 600

reasons for using, 601 decreased CPU consumption, 601 faster response times, 601 less reads and writes, 601

security privileges, 595 GRANT CREATE MATERIALIZED VIEW privilege, 595 GRANT CREATE SESSION privilege, 595 GRANT CREATE TABLE privilege, 595 GRANT QUERY REWRITE privilege, 595

setting up, 602 COMPATIBLE parameter, 602 QUERY_REWRITE_ENABLED parameter, 602 QUERY_REWRITE_INTEGRITY parameter, 602

MAX function analytic functions, 564 generic pivot queries, 578 pivot queries, 577

MAXEXTENTS parameter rollback segments, 185, 187 tables, syntax, 208

MAXLEN parameter, 788 MAXTRANS parameter

tables, syntax, 209 TX locks, 117

metadata constraints as, 607 materialized views, 601

migrated rows

1282

migrated rows avoiding

PCTFREE parameter, 202 definition, 202 LogMiner packages, limitations, 1116 tables, syntax, 202

MIN function analytic functions, 564

MINEXTENTS parameter tables, syntax, 208

modular code autonomous transactions, 678

most discriminating elements should go first disproving myth, 314 myths about indexes, 311

MTSadvantages, 88 compared to dedicated server, 57, 87 database architecture, 57 description, 25 dispatcher process, 57

connecting to, 59 Java pool, 84 Large pool, 83 listener, 59 Net8 listener, 86 Oracle database, using with, 25

avoiding long transactions, 25 server processes, 85, 86 when and how to use, 89

multiple users online redo log files, factors affecting, 68

multiple versions of SQL query StatsPack report, interpreting, 486

multi-tablespace objects CREATE TABLE statement, 361

multi-threaded server see MTS.

multi-versioning example, 33 non-blocking queries, 33 Oracle database, 33 read-consistent queries, 33, 125 repeatable read isolation level, 128

mutating tables autonomous transactions, problems with, 693 avoiding with autonomous transactions, 665

NNAME column

DBA_OUTLINE_HINTS table, 517 DBA_OUTLINES table, 516

NAME_RESOLVE function DBMS_UTILITY package, 1186

NAME_TOKENIZE function DBMS_UTILITY package, 1188

named pipes OS pipe, exporting to, 327

namespace variables application context, 919

National Language Support EXP tool, problems with, 359 IMP tool, problems with, 359

native dynamic SQL application context, 716 array processing, 728

compared to DBMS_SQL, 702, 713 bind variables, 714 repeated statements, 725 unknown number of outputs, 718

cursor variables, 711 EXECUTE IMMEDIATE clause, 709 OPEN FOR clause, 711 performance testing, 728 using, 709

example, 709 necesssary supplied packages

DBMS_ALERT package, 1031, 1032 DBMS_APPLICATION_INFO package, 1042 DBMS_JAVA package, 1050 DBMS_JOB package, 1059 DBMS_LOB package, 1073 DBMS_LOCK package, 1092 DBMS_OBFUSCATION_TOOLKIT package, 1123 DBMS_OUTPUT package, 1144 DBMS_PIPE package, 1031, 1038 DBMS_PROFILER, 1161 DBMS_UTILITY package, 1172 description, 1028 LogMiner packages, 1097 Oracle database, 1027 reasons for using, 1028 UTL_FILE API, 1199 UTL_HTTP package, 1208 UTL_RAW package, 1229 UTL_SMTP package, 1231 UTL_TCP package, 1244

nested tables compared to relational tables, 244 compared to VARRAYS, 876, 880 CREATE TABLE statement, 241 DELETE statement, 245 description, 240 extending PL/SQL with object relational features, 240 INSERT statement, 245 loading

SQLLDR, 418 object relational features, 240 referential integrity not supported by, 242 storage of nested tables, 248

index organized tables, 249 syntax, 240

exotic features of syntax, 242 table types, 198, 240 treating as real table, 247

NESTED_TABLE_GET_REFS hint, 247 undocumemted features, 245 UPDATE statement, 244 using as physical storage mechanism, 240

disadvantages, 245, 264 NESTED_TABLE_GET_REFS hint

treating nested tables as real tables, 247 nesting autonomous transactions, 680 Net8 Assistant

LISTENER.ORA file, configuring, 775 Net8 listener

dedicated server, 86 EXTPROC OS process, 773 MTS, 86 PMON, 91

network programming UTL_TCP package, 1244

networking parameter files and, 60

OCICollSize API call

1283

NEXT parameter considered obsolete, 208 tables, syntax, 208

NLSsee National Language Support.

nmake command building extproc.dll, 817

NOARCHIVELOG mode, 69 compared to ARCHIVELOG mode, 69

NOCOMPRESS option compared to COMPRESS N option, 220 CREATE TABLE statement

index organized tables, 217 Indexes, 217

NODE column DBA_OUTLINE_HINTS table, 517

NOLOGGING clause care in using, 175 limitations, 175 preventing redo log generation, 173 replaces UNRECOVERABLE clause, 174 tables, syntax, 208 using implicitly, 175

non-autonomous transactions compared to autonomous transactions, 660

non-blocking queries Oracle database, multi-versioning, 33

non-blocking reads Oracle database, locking policy, 31

non-default roles, 999 NON-POLLING method

LOBS, manipulating, 826 non-repeatable read

transaction isolation levels, 124 n-Tier authentication, 963

ALTER USER command, GRANT CONNECT THROUGH clause, 965, 973

application server, 968 application server session, 970

AUDIT command, 976 C code, 966 care in using, 977 client session, 969

application server session, 970 mechanics, 966 OCI programs, 966 Operating System authentication, 969 password authentication, 969 reasons for using, 964 security model, 966 web-based authentication, 965

NTILE function analytic functions, 564

NULL pool SGA, 76

NULLS analytic functions, problems with, 588 Java stored procedures, 852

NUMARRAY type, Oracle passing to Java ARRAY type

Java stored procedures, 850 Number type, Oracle

passing to Java BigDecimal type Java stored procedures, 849

Numeric Expression FOLLOWING windowing clause, 560

Numeric Expression PRECEDING windowing clause, 560

Oobject methods, 872

ALTER TYPE command, 875 difficulties in altering, 873 MAP methods, 873 ORDER methods, 873 reasons for using, 873

binding methods to data, 873 encapsulation, 873

object relational features, 867 adding data types, 869

adding member function to type, 872 CREATE TYPE statement, 869 using new type, 870

collection types, using, 892 array fetch into tables, 892, 895 inserting records, 892, 896 querying from PL/SQL routine, 892

nested tables, 240 extending PL/SQL, 240

object methods, 872 object tables, 260 reasons for using, 868

creating object tables, 868 enforcing standardization, 868 extending PL/SQL, 868, 882

creating new PL/SQL data type, 883 presenting object relational views, 868, 897

VARRAYS, 876 object relational views, 897

advantages of using, 910 data types, 898 relational tables and, 900, 906 synthesizing view, 898 updating views, 901

INSTEAD OF trigger, 901 using view, 899

object tables compared to relational tables, 264 CREATE TABLE statement, 258, 261 creating using object relational features, 868 description, 258 external face of table, 260 object relational features, 260 object types, 258 SYS_NC_OID$ column, 260

analyzing, 262 table types, 198, 258 true structure of table, 260 undocumented features, 262 using as physical storage mechanism

disadvantages, 264 object types

FILETYPE object type, 884 LogMiner packages, limitations, 1113 object tables, 258 schemas, cloning, 349 SocketType object type, 1245

OCI programs C code, 967 data loading, 368 n-Tier authentication, 966

OCICollAppend API call mapping Oracle data types to C data types, 809

OCICollGetElem API call mapping Oracle data types to C data types, 809

OCICollSize API call mapping Oracle data types to C data types, 809

OCIContextGetValue API call

1284

OCIContextGetValue API call init function, external procedures template, 798

OCIContextSetValue API call init function, external procedures template, 799

OCIExtProcGetEnv API call init function, external procedures template, 798

OCIExtractFromFile API call init function, external procedures template, 800

OCIExtractInit API call init function, external procedures template, 800

OCIExtractSetKey API call init function, external procedures template, 800

OCIExtractSetNumKeys API call init function, external procedures template, 800

OCIExtractTerm API call init function, external procedures template, 800

OCIExtractTo API call init function, external procedures template, 800

OCIFileInit API call init function, external procedures template, 801

OCIMemoryAllocate API call init function, external procedures template, 799

OCINumberFromReal API call mapping Oracle data types to C data types, 802

ODBC API autocommit by default, 148

OLTP systems global index partition, 651 materialized views, problems with, 624

ON BEHALF OF ALL clause AUDIT command, 976

ON BEHALF OF clause AUDIT command, 976

ON COMMIT DELETE ROWS clause transaction-based temporary tables, 252

ON COMMIT PRESERVE ROWS clause session-based temporary tables, 252

ON DELETE CASCADE clause referential integrity, 942

ON DELETE SET NULL clause referential integrity, 943

ON LOGON database trigger development tool, 514 Fine Grained Access Control, 918 implementing tuning, 512 listing executed SQL, 515 listing used indexes, 515 setting application context, 927

online redo log files, 67, 158 checkpointing, 67 database buffer cache, 67 DBWn, 68 factors affecting, 68

multiple users, 68 recovery time, 69 standby database, 68

log switch, 67 OPEN FOR clause

cursor variables, 711 native dynamic SQL, 711

OPEN function FILETYPE object type, 884, 885

Operating System Oracle database as virtual operating system, 20 running command or program

Java stored procedures, 844, 860 security privileges, 861

Operating System authentication n-Tier authentication, 969

Operating System Dependent code see OSD code.

operator object interMedia Text, 750

OPS database architecture, 54

optimistic locking compared to pessimistic locking, 106 SELECT FOR UPDATE NOWAIT statement, 106 strategies for avoiding lost updates, 106

optimizer plan stability, 505 ALTER SESSION SET USE_STORED_OUTLINE

command, 513, 515 CREATE OR REPLACE OUTLINE command, 508

CREATE ANY OUTLINE privilege, 508 description, 506, 516 errors

ORA-18001, 540 ORA-18002, 541 ORA-18003, 541 ORA-18004, 541 ORA-18005, 542 ORA-18006, 542 ORA-18007, 542

hinting mechanism, 506 OUTLINE_HINTS table, 517

ALL_OUTLINE_HINTS table, 517 DBA_OUTLINE_HINTS table, 517 USER_OUTLINE_HINTS table, 517

OUTLINES table, 516 ALL_OUTLINES table, 516 DBA_OUTLINES table, 516 USER_OUTLINES table, 516

OUTLN user, 521 performance testing, 536 problems with, 531

ALTER SESSION command, 533 case sensitivity, 531 cursor_sharing=force parameter, 533 DROP USER command, 533 global namespace of outlines, 539 OR-Expansion, 535 OUTLN user in SYSTEM tablespace by default, 535 quoted identifiers, 533 text matching, 534

quick example, 506 reasons for using, 506 stored outlines

creating, 518 managing, 525 moving between databases, 522 optimizing, 522

using, 509 development tool, 514 implementing tuning, 509 listing executed SQL, 515 listing used indexes, 515

optimizer related issues cursor_sharing=force parameter, 443

ORA-00060 autonomous transactions, errors, 696

ORA-00959 IMP tool, errors, 364

ORA-01555, 185 block cleanout, 185, 191

example, 192 necessary conditions for error, 192 rarity of error from, 194

Oracle database

1285

ORA-01555 (Continued) causes, 185 COMMIT statement, 185, 190

example, 190 error cannot always be avoided, 146 read consistency model, 186 read only isolation level, 132 rollback segment size, 185, 186

example, 188 serializable isolation level, 130 solutions to error, 186

analyzing related objects, 186, 195 avoiding unnecessary COMMITs, 195 increasing rollback segments, 186, 195 tuning query, 186, 195

ORA-01562 error avoidable by right sizing, 146


ORA-06520 C-based stored procedures, errors, 837







ORA-18001 optimizer plan stability, errors, 540







ORA-28106 Fine Grained Access Control, errors, 959




ORA-28575 C-based stored procedures

configuring error, 776 errors, 833

resolving error, 777 database environment, verifying, 777 EXTPROC OS process, verifying, 777 listener, verifying, 779







ORA-29459 Java Session State cleared Java stored procedures, 864

ORA-29531 no method X in class Y Java stored procedures, error handling, 864

Oracle applications, 19 bind variables, 27 database independence, 37 developing, 19 project failures, investigating, 24 project failures, reasons for, 21 security logic in database, 995 solving problems simply, 44 tuning, 23, 47, 431

Oracle Call Interface see OCI programs.

Oracle data types BFILE, 821 BLOB type, 821 CLOB type, 821 mapping Oracle data types to C data types, 787, 802

Oracle database Advanced Queues, 26, 44 ANSI compliant databases, 38 ARCHIVELOG mode, 69 autonomous transactions, 44, 661

writing to database from SQL functions, 670 background processes, 55 C compiler, 15 compared to SQL Server, 39 concurrency control, 30, 103 data loading, 367 database architecture, 25, 53 DDL always commits, 119 deadlocks

causes of, 108 rarity of, 108

developing, 21 distributed transactions, 148 does not support dirty read, 125 does not support lock escalation, 111 EXP tool, 317 explain plan facility, 462 Fear Uncertainty and Doubt about, 21 features and functions, 43

advanced features importance of, 44 Fine Grained Access Control, 47, 913 IMP tool, 317 implicit transactions, 143 importance of familiarity with, 21 Indexes, 270 interMedia, 739 invoker rights, 981 latches, 27 lock conversion, 112 locking policy, 30, 103

FOR UPDATE clause, 32 non-blocking reads, 31

MTS, using with, 25 avoiding long transactions, 25

Oracle database (Continued)

1286

Oracle database (Continued) multi-versioning, 33

example, 33 non-blocking queries, 33 read-consistent queries, 33, 125 repeatable read isolation level, 128

necesssary supplied packages, 1027-1256 NOARCHIVELOG mode, 69 object relational features, 867 optimizer plan stability, 505 Oracle applications, 19 OSD code, 42 partitioning, 627 redo log files, 157 scalability, 25 segments, 62 serializable transactions, 130 SQL92 standard, 40 statement level atomicity, 137 tables, 197 tablespaces, 63 transactional mechanisms, 35, 135

Atomic transactions, 136 serializable transactions, 23

treating as black box, disadvantages of, 21 virtual operating system, Oracle as, 20

Oracle Forms deadlocks and, 109 strategies for avoiding lost updates, 104

Oracle Parallel Server see OPS.

ORDER BY clause analytic functions, syntax, 551

ORDER methods compared to MAP methods, 875 description, 874 object methods, 873

ORDYS.ORDIMAGE object loading data into LOBS, 416 SETPROPERTIES method, 418 structure, 416

OR-Expansion optimizer plan stability, problems with, 535

OS pipe, exporting to EXP tool, large exports, 327 named pipes, 327 requires UNIX system, 327

OSD code, 42 OUT parameter

Java stored procedures, 849 passing OUT parameter using arrays, 852

OUTLINE_HINTS table ALL_OUTLINE_HINTS table, 517 DBA_OUTLINE_HINTS table, 517 optimizer plan stability, 517 USER_OUTLINE_HINTS table, 517

OUTLINES table ALL_OUTLINES table, 516 DBA_OUTLINES table, 516 optimizer plan stability, 516 USER_OUTLINES table, 516

OUTLN user in SYSTEM tablespace by default, 521

optimizer plan stability, problems with, 535 moving from SYSTEM to TOOLS, 521 optimizer plan stability, 521

OUTLN_PKG package DROP_BY_CAT procedure, 529 DROP_UNUSED procedure, 529 functions undocumented, 528 stored outlines, managing, 528 UPDATE_BY_CAT procedure, 529

over binding cursor_sharing=force parameter, 446

OVERFLOW option CREATE TABLE statement

index organized tables, 220, 222 INCLUDING option, 223 PCTTHRESHOLD parameter, 222

OWNER column DBA_OUTLINE_HINTS table, 517 DBA_OUTLINES table, 516

Ppackaged variables

scope in autonomous transactions, 681 Parallel Cache Management

see PCM. parallel DML, 630

enhancing DML performancr, 630 limitations, 630 requires partitioning, 630

parallel execution of statements Large pool, 83

parallel operations enhancing query performance, 631 limitations, 631

parallel query slaves database architecture, 97

parameter files database architecture, file types, 60 init.ora file, 60

configuration settings, 61 documented parameters, 61 IFILE directive, 62 undocumented parameters, 61

networking and, 60 SID, 60

parameter setting Java stored procedures, 845, 848 list of non-default parameters

StatsPack report, interpreting, 494 template for external procedures, 784

parameters, ADD_POLICY routine, 921 parameters, SQLLDR, 369

table of parameters, 369 PARSE errors


PARSE phase TKPROF, analyzing query, 458

parse-related statistics ratio of soft versus hard parses, 481 StatsPack report, interpreting, 480

parsing hard parsing, 27 performance time

invoker rights, problems with, 1013 measuring with TKPROF, 1013

ratio of soft versus hard parses, 481 soft parsing, 27

avoiding soft parse, 441

PL/SQL stored procedures

1287

partial text match materialized views, 603

partition clause analytic functions, syntax, 551

partition elimination enhancing query performance, 631 limitations, 631

partitioning, 627 enhancing DML performance, 630 enhancing query performance, 631

parallel operations, 631 partition elimination, 631

increasing availability, 628 parallel DML, 630

index partitioning schemes, 637 global index partition, 637, 645

data warehousing, 647 OLTP systems, 651

local index partition, 637, 638 local non-prefixed indexes, 638 local prefixed indexes, 638 uniqueness and, 643

sliding window implementation, 638 reasons for using, 627 reducing administration burden, 629 table partitioning schemes, 632

composite partitioning, 633, 636 hash partitioning, 633, 635 range partitioning, 632, 633

password authentication n-Tier authentication, 969

password files database architecture, file types, 60

password protected roles, 999 PCM locks, 112 PCTFREE parameter

CREATE TABLE statement heap organized tables, 212

row migration, avoiding, 202 setting value of parameter, 204

DBMS_SPACE package, 205 tables, syntax, 202

PCTINCREASE parameter considered obsolete, 208 tables, syntax, 208

PCTTHRESHOLD parameter compared to INCLUDING option, 223 CREATE TABLE statement

index organized tables, 217, 222 OVERFLOW option, 222

PCTUSED parameter CREATE TABLE statement

heap organized tables, 212 setting value of parameter, 204

DBMS_SPACE package, 205 tables, syntax, 202

PDML see parallel DML.

PERCENT_RANK function analytic functions, 564

performance metrics, identifying, 434 PERFSTAT user

StatsPack, setting up, 478 permissions errors

Java stored procedures, 864 pessimistic locking

compared to optimistic locking, 106 strategies for avoiding lost updates, 105

PGA database architecture, 70 LogMiner packages and, 1111 testing usage, 71

phantom read transaction isolation levels, 124

physical data layout importance of, 281

pinging, avoiding reverse key indexes, 275

pivot queries analytic functions, 576 generic pivot queries, 578

COUNT function, 578 cursor variables, 582 dynamic SQL, 579 MAX function, 578 ROW_NUMBER function, 578

MAX function, 577 ROW_NUMBER function, 577

PL/SQL analytic functions, problems with, 586 blocks

DBMS_SQL, using with, 707 statement level atomicity, 138

collection types, using, 892 compared to SQLJ, 844 dynamic SQL, 699

bind variables, 714 DBMS_SQL, 702

required if output number unknown, 718 repeated statements, 725 unknown number of outputs, 718

extending with object relational features, 868, 882

creating new PL/SQL data type, 883 nested tables, 240

loading data into LOBS, 409 required for autonomous transactions, 689 running programs using UTL_HTTP package, 1217 unloader PL/SQL utility, 399 writing mini-SQLLDR, 422

PL/SQL data type creating new PL/SQL data type, 883

encapsulating UTL_FILE functionality, 883 PL/SQL package

encapsulating UTL_FILE in new PL/SQL data type, 884 supporting FILETYPE object type, 885

PL/SQL pieces invoker rights and stored procedures, 1000

PL/SQL prototype C-based stored procedures, first example, 784 creating library object, 786 creating package body, 786 data type functions, 784

creating collection types, 784 passing parameters, 784 returning SCALAR types, 785

INDICATOR parameter, 791 LENGTH parameter, 790 mapping Oracle data types to C data types, 787 MAXLEN parameter, 788 RETURN keyword, 790

PL/SQL stored procedures see also stored procedures compared to Java stored procedures, 844

plain text editor

1288

plain text editor LISTENER.ORA file, configuring, 775

PMON focused background processes, 91 Net8 listener, 91

PORT_STRING function DBMS_UTILITY package, 1193

portability macro mapping Oracle data types to C data types, 802

POSITION keyword loading fixed format data, 377

PRAGMA AUTONOMOUS_TRANSACTION directive autonomous transactions, 660

pre-calculating object count materialized views, 598

pre-compiler based database programming static SQL, 700

Pro*C code LOB_IO procedure, 823 lobToFile library, 823 NON-POLLING method, 826

Process Global Area see PGA.

Process Monitor see PMON.

proxy authentication see n-Tier authentication.

QQMNn

Advanced Queues, 96 utility background processes, 96

queries hard parsing, 27 multiple versions of SQL query, 486 non-blocking queries, 33 pivot queries, 576

generic pivot queries, 578 read-consistent queries, 33 soft parsing, 27 TOP-N queries, 566

query output issues cursor_sharing=force parameter, 447

query performance enhancing using partitioning, 631

parallel operations, 631 partition elimination, 631

query plan TKPROF, analyzing query, 460

query rewrite methods materialized views, 603

QUERY REWRITE privilege function based indexes, 288

QUERY_REWRITE_ENABLED parameter materialized views, setting up, 602

QUERY_REWRITE_INTEGRITY parameter ENFORCED value, 602 materialized views

problems with, 624 setting up, 602

STALE_TOLERATED value, 602 TRUSTED value, 602

QUERY= parameter EXP tool, subsetting data, 328

Queue Monitor Processes see QMNn.

quoted identifiers optimizer plan stability, problems with, 533

Rraise_application_error function

template for external procedures, 797 range partitioning, 632, 633 range windows

analytic functions, syntax, 555 RANK function

analytic functions, 564 ranking functions

analytic functions, 550 RATIO_TO_REPORT function

analytic functions, 565 RAW devices

compared to cooked file systems, 181 redo log files, 181

RAW type, Oracle passing to Java byte type

Java stored procedures, 851 storing VARRAYS as, 880

read committed isolation level behavior varies between databases, 127 concurrency control, 124, 126

read consistency model ORA-01555, 186

read consistent queries example, 34 Oracle database, 125

multi-versioning, 33 READ method

DBMS_LOB package, 412 read only data

transporting data EXP tool, 333 IMP tool, 333

read only isolation level compared to serializable isolation level, 132 concurrency control, 125, 132 ORA-01555, 132

read uncommitted isolation level concurrency control, 124, 125

really strict auditing autonomous transactions, 671

RECOfocused background processes, 93

RECOMMEND_MV routine DBMS_OLAP package, 622

RECOMMEND_MV_W routine DBMS_OLAP package, 622

recovery and backup see backup.

recovery time online redo log files, factors affecting, 69

recursive SQL dictionary-managed tablespace, 64

RECYCLE pool block buffer cache, 80 Large pool, 82

Redoanalyzing redo, 184

LogMiner packages, 184 BEFORE trigger, increase Redo, 169 block cleanout, 177 COMMIT statement, increases Redo, 168

Run-time errors

1289

Redo (Continued) estimating amount of Redo, 173 introduction, 157 measuring amount of Redo, 164

redo size statistic, 164 table of results, 167 V$MYSTAT view, 164 V$STATNAME view, 164

preventing redo log generation UNRECOVERABLE clause, 174

system crashes and, 151 temporary tables, 182

measuring redo/rollback, 182 transactional mechanisms, 150

redo log buffer rollback segments, 151

Undo, 151 SGA, 77 system crashes and, 153

redo log files analyzing redo, 184

LogMiner packages, 184 archived redo log files, 69, 158 cannot allocate new log message, dealing with, 176

adding redo log files, 177 increasing size of redo log files, 177 more frequent checkpointing, 177tuning DBWR, 176

database architecture, file types, 60, 66 importance of, 66 log contention, 180

causes, 180 online redo log files, 67, 158 Oracle database, 157 preventing redo log generation, 173

NOLOGGING clause, 173 RAW devices, 181

redo size statistic measuring amount of Redo, 164

REF CURSOR see cursor variables.

referential integrity covert channel, 941 Fine Grained Access Control, problems with, 941 not supported by nested tables, 242 ON DELETE CASCADE clause, 942 ON DELETE SET NULL clause, 943 schemas, cloning, 347

REFRESH ON COMMIT materialized views, 600

REGR_ functions analytic functions, 565

relational tables advantages of using, 906 compared to nested tables, 244 compared to object tables, 264 CURSOR function and, 908 object relational views and, 900, 906

remote execution dedicated server, 86

reorganization EXP/IMP tools, unsuitable for, 340

repeatable read isolation level concurrency control, 124, 128 consistent answers, obtaining, 128 lost update prevention, 129 Oracle database, multi-versioning, 128 shared read locks, 128

repeated statements dynamic PL/SQL, 725

performance testing, 725 reporting functions

analytic functions, 550 report-style data

loading using SQLLDR, 387 stored procedures, 388

RETURN keyword, 790 reverse key indexes, 270, 275

pinging, avoiding, 275 related to B*Tree indexes, 275

REVOKE CONNECT THROUGH clause ALTER USER command, 975

roles invoker rights and, 1006 non-default roles, 999 not enabled for definer rights, 996, 999 password protected roles, 999

rollback see also Undo. autonomous transactions, problems with, 690 compared to COMMIT statement, 163 description, 163 introduction, 184 ORA-01555, 185 temporary tables, 182

measuring redo/rollback, 182 ROLLBACK TO <SAVEPOINT> statement

transaction control statements, 136 rollback segments

importance of adequate sizing, 146 issues with rollback segments, 184 MAXEXTENTS parameter, 185, 187 problems with, 144 redo log buffer, 151 SET TRANSACTION statement, 185 size of

ORA-01555, 185, 186 SQLLDR, problems with, 426 transactional mechanisms, 144 Undo, 151

ROLLBACK statement transaction control statements, 136

row IDs duplicate row IDs, 229

Row Level Security see Fine Grained Access Control.

row migration see migrated rows.

row windows analytic functions, syntax, 558

ROW_NUMBER function analytic functions, 565 generic pivot queries, 578 obtaining slices of data, 570 pivot queries, 577 TOP-N queries, 570

rows inserting with SQLLDR, 385 updating with SQLLDR, 385

RUN function unloader PL/SQL utility, 403

Run-time errors fields, meaning of, 474 trace files, using and interpreting, 472

SAVEPOINT statement

1290

SSAVEPOINT statement

cannot issue SAVEPOINT over database link, 150 transaction control statements, 136

savepoints autonomous transactions, 687

scalability bind variables, 27, 436

testing with multiple users, 436 definer rights, reasons for using, 994 invoker rights, problems with, 994 Oracle database, 25

scalar types Java stored procedures, 851

schemas, cloning EXP tool

problems with, 346 reasons for using, 319

IMP tool problems with, 346 reasons for using, 319

object types, 349 referential integrity, 347

SCN, generating COMMIT statement, 160

scope autonomous transactions, 681 database changes, 682 locks, 685 packaged variables, 681 session state, 682

SCOTT/TIGER account, setting up, 11 C-based stored procedures, testing, 780 CREATE LIBRARY privilege, 781

scripts DDL, extracting, 337

getallcode.sql script, 337 getallviews.sql script, 338 getaview.sql script, 338 getcode.sql script, 337 gettrig.sql script, 339

secondary indexes index organized tables, 224

section group interMedia Text, 757

section searching auto-sectioning, 761 field section, 758 HTML section searching, 757 interMedia Text, 756 section group, 757 XML section searching, 760 zone section, 761

sections object interMedia Text, 749

secure information auditing attempts to modify secure information

AUDIT command, 662 autonomous transactions, 662

Secure Sockets Layer see SSL.

security logic in database definer rights, reasons for using, 995 Fine Grained Access Control, reasons for using, 915 Oracle applications, 995

security model n-Tier authentication, 966

security policy application context, 930 Fine Grained Access Control, 918 generic utilities, developing, 985 implementing policy with FGAC, 919

security privileges ALTER ANY OUTLINE privilege, 519 ALTER USER command, GRANT CONNECT THROUGH clause, 975 CREATE ANY CONTEXT privilege, 918 CREATE ANY OUTLINE privilege, 508, 519 CREATE LIBRARY privilege, 781 directly granted security privileges

definer rights, 996 invoker rights, 1006

directory listing, obtaining, 859 DROP ANY OUTLINE privilege, 519 EXECUTE ON DBMS_RLS privilege, 924 EXECUTE ON OUTLN_PKG privilege, 519 EXECUTE_CATALOG_ROLE privilege, 918 GLOBAL QUERY REWRITE privilege, 288 GRANT CREATE MATERIALIZED VIEW privilege, 595 GRANT CREATE SESSION privilege, 595 GRANT CREATE TABLE privilege, 595 GRANT QUERY REWRITE privilege, 595 invoker rights, conveying privileges for, 1001 materialized views, 595 Operating System, running command or program, 861 QUERY REWRITE privilege, 288 stored outlines, creating, 519

segments, 62 extents, 62

blocks, 62 tablespaces, 63

SELECT * statements invoker rights, problems with, 1017

SELECT FOR UPDATE NOWAIT statement optimistic locking, 106

SELECT FOR UPDATE statement blocking, 107 DBMS_LOB package, 1075

SELECT statements writing to database from SELECT statements

autonomous transactions, 675 care in using, 676

SELECT...FOR UPDATE statement manual locking, 123

serializable isolation level compared to read only isolation level, 132 concurrency control, 124, 130 lost update prevention, 129 ORA-01555, 130 serializable transactions, 130

serializable transactions Oracle database, 23, 130 serializable isolation level, 130

server processes database architecture, 85 dedicated server, 85 MTS, 85, 86

session state scope in autonomous transactions, 682

session-based temporary tables, 251 ON COMMIT PRESERVE ROWS clause, 252

SQL_TEXT column

1291

SET EVENTS approach, SQL_TRACE not supported or documented by Oracle, 453

SET TRANSACTION statement rollback segments, 185 transaction control statements, 136

SET_CLIENT_INFO call DBMS_APPLICATION_INFO package, 1043

SET_OUTPUT procedure DBMS_JAVA package, 1055

SETPROPERTIES method ORDYS.ORDIMAGE object, 418

SGAblock buffer cache, 78

KEEP pool, 80 RECYCLE pool, 80

database architecture, 55, 70, 74 fixed SGA, 77 init.ora file, 77 Java pool, 76, 83 Large pool, 76, 82

RECYCLE pool, 82 NULL pool, 76 redo log buffer, 77 Shared pool, 76, 80

bind variables, 81 DBMS_SHARED_POOL package, 81 KEEP pool, 82 problems with, 81

UNIX-based systems, 75 Windows-based systems, 75

share DDL locks, 119 shared accounts, avoiding

Fine Grained Access Control, reasons for using, 916

Shared pool bind variables, 81 DBMS_SHARED_POOL package, 81 KEEP pool, 82 problems with, 81 SGA, 76, 80

shared pool utilization invoker rights, problems with, 1010 StatsPack report, interpreting, 482, 493

shared read locks problems with, 129 repeatable read isolation level, 128

SHORTNAME routine DBMS_JAVA package, 1050

SHOW = Y option compared to INDEXFILE option, 335 DDL, extracting, 334

SIDparameter files, 60

SID_DESC = (SID_NAME = PLSExtProc) LISTENER.ORA file, 775

Simple Mail Transfer Protocol see SMTP.

single account environment, supporting Fine Grained Access Control, reasons for using, 917

single table hash cluster, 238 CREATE CLUSTER statement

SINGLE TABLE keywords, 238 SINGLE TABLE keywords

CREATE CLUSTER statement single table hash cluster, 238

Site Identifier see SID.

size parameter CREATE CLUSTER statement

hash clustered tables, 232 index clustered tables, 226

slave processes database architecture, 85, 97 I/O slaves, 97 parallel query slaves, 97

sliding window implementation global index partition, 647

should be avoided, 651 index partitioning schemes, 638

SMONclean up functions, 92 coalescing free space, 92 focused background processes, 92 recovery functions, 92 rollback functions, 92

SMTP UTL_SMTP package, 1231

Snapshot materialized views, brief history, 594

Snapshot Processes see SNPn.

SNPn utility background processes, 96

SocketType object type HTTP_PKG package, 1220 UTL_TCP package, 1245

soft parsing queries, 27

space never reused in index disproving myth, 311 myths about indexes, 308

SQLanalytic functions, 545 case statement, 382 listing executed SQL, 515 recursive SQL, 64 static SQL, 699 writing to database from SQL functions

autonomous transactions, 670 SQL data types

mapping Java data types to SQL data types, 847 mapping Oracle data types to C data types, 787, 802 table mapping SQL types to external types, 792

SQL pieces invoker rights and stored procedures, 1000

SQL Server ANSI compliant databases, 38 compared to Oracle database, 39

SQL*LOADER tool see SQLLDR.

SQL*PLUS AUTOTRACE

controlling report, 15 execution plan, 15 generating report, 14 setting up, 14

dedicated server, 86 environment, setting up, 12

SQL*TextRetrieval interMedia Text, brief history, 739

SQL_TEXT column DBA_OUTLINES table, 517

SQL_TRACE

1292

SQL_TRACE analytic functions, performance testing, 571 guidelines for using, 454 hash clustered tables, performance testing, 234 implementing tuning, 510 selectively enabling, 452

enabling in current session, 453 packaged procedure, using, 453 SET EVENTS approach, 453

setting up, 452 tuning with, 451

SQL92 standard entry-level, 40 full implementation, 40 intermediate-level, 40 portability and, 41 transaction isolation levels, 124 transitional-level, 40

SQLJcompared to PL/SQL, 844 directory listing, obtaining, 859

SQLLDR BAD file, using with, 383, 385, 424 calling from stored procedure, 421

problems with, 421 writing mini-SQLLDR, 422

control file, 371 command line overrides control file, 426 simple example, 371

conventional path, 368 data loading, 367

loading data into LOBS, 412 loading data into object columns, 416 loading inline data, 413 loading out of line data, 414

loading data with embedded newlines, 391 loading delimited data, 374 loading fixed format data, 377 loading into LONG field, 390 loading into LONG RAW field, 389 loading nested tables, 418 loading report-style data, 387 loading tab-delimited data, 375 loading VARRAYS, 418 using functions, 380 using sequences, 380

direct path, 368 INSERT statement, 380 introduction, 368 load dates, 379

DATE data type, 379 date mask, 379

log file, 372 parameters, 369


command line overrides control file, 426 default length of input fields, 426 rollback segments, 426 TRUNCATE option, 426

rows inserting new rows, 385 updating rows, 385

unloading data in SQLLDR friendly format, 399 unloader PL/SQL utility, 399

SSLUTL_HTTP package, 1210

STAGE column DBA_OUTLINE_HINTS table, 517

STALE_TOLERATED value QUERY_REWRITE_INTEGRITY parameter, 602

standardization enforcing with object relational features, 868

standby database online redo log files, factors affecting, 68

STAT record fields, meaning of, 472 trace files, using and interpreting, 472

state management Java stored procedures, 845 template for external procedures, 783

statement level atomicity, 136 Oracle database, 137 PL/SQL blocks, 138 triggers, 138

static SQL bind variables, 440 compared to dynamic SQL, 700 pre-compiler based database programming, 700

statistical functions analytic functions, 551

StatsPack replaces BSTAT/ESTAT, 477 report, interpreting, 479

detailed numerical information, 488 dictionary cache, 492 I/Os logical and physical, 485 latches, 491 list of non-default parameters, 494 multiple versions of SQL query, 486 parse-related statistics, 480 ratio of soft versus hard parses, 481 shared pool utilization, 482, 493 Wait events, 482

setting up, 477 PERFSTAT user, 478

tuning with, 477 STDDEV function

analytic functions, 565 STDDEV_POP function

analytic functions, 565 STDDEV_SAMP function

analytic functions, 565 stoplists

text indexing, 755 stored outlines, creating

ALTER ANY OUTLINE privilege, 519 ALTER SESSION command, 520

ALTER SESSION SET CREATE_STORED_OUTLINE command, 520

CREATE ANY OUTLINE privilege, 519 DROP ANY OUTLINE privilege, 519 EXECUTE ON OUTLN_PKG privilege, 519 optimizer plan stability, 518 security privileges, 519 using DDL, 519

stored outlines, managing optimizer plan stability, 525 OUTLN_PKG package, 528 using DDL, 525

stored outlines, moving between databases optimizer plan stability, 522

stored outlines, optimizing hinted views, 524 optimizer plan stability, 522

stored outlines, using determining index usage, 308

template for external procedures

1293

stored procedures breakable parse locks and, 121 calling SQLLDR from, 421

problems with, 421 writing mini-SQLLDR, 422

C-based stored procedures, 771 customized access control, 991 data dictionary applications, 989 definer rights and, 982


generic utilities, developing, 985 invoker rights and, 982


environments for procedures, 1001 table of environments, 1001

PL/SQL pieces, 1000 SQL pieces, 1000

Java stored procedures, 843 loading report-style data, 388

STR attribute hexadecimal specification, 398 loading data with embedded newlines, 398

STRARRAY type, Oracle passing to Java ARRAY type

Java stored procedures, 850 String type, Java

ease of manipulation, 853 passed from Oracle Varchar2 type

Java stored procedures, 850 strtok function

C code, 967 substr function

CREATE INDEX statement, 293 DBMS_LOB package, 1074 function based indexes, 288 hiding substr call, 295

SUM function analytic functions, 565

summary table management materialized views, brief history, 594

supplied packages see necesssary supplied packages.

Sybase ANSI compliant databases, 38 concurrency control, 102 locking policy, 102

synchronizing index interMedia Text, 753

problems with, 763 SYS_CONTEXT function

application context, 716 changing value during application, 950 Fine Grained Access Control, 918

SYS_NC_OID$ column analyzing, 262 object tables, 260

System Change Number see SCN.

system crashes Redo and, 151 redo log buffer and, 153 Undo and, 151

SYSTEM data data files, 62

System Global Area see SGA.

System Monitor see SMON.

Ttab-delimited data, 375

loading with SQLLDR, 375 TERMINATED BY clause, 375

TABLE ACCESS BY INDEX ROWID compared to FULL TABLE SCAN, 279 compared to INDEX RANGE SCAN, 279

table partitioning schemes, 632 composite partitioning, 633, 636 hash partitioning, 633, 635 range partitioning, 632, 633

TABLE_TO_COMMA function DBMS_UTILITY package, 1191

tables, 197 general information about tables, 198 introduction, 197 syntax, 199

FREELIST, 200 high water mark, 199 INITIAL parameter, 208 INITRANS parameter, 209 MAXEXTENTS parameter, 208 MAXTRANS parameter, 209 migrated rows, 202 MINEXTENTS parameter, 208 NEXT parameter, 208 NOLOGGING clause, 208 PCTFREE parameter, 202 PCTINCREASE parameter, 208 PCTUSED parameter, 202

table types, 197 hash clustered tables, 198, 231 heap organized tables, 197, 209 index clustered tables, 197, 224 index organized tables, 197, 212 nested tables, 198, 240 object tables, 198, 258 temporary tables, 198, 251

tablespaces, 63 data files, 64 determining index usage, 308 dictionary-managed tablespace, 64 EXP tool, transporting data, 330, 331 IMP tool, transporting data, 330, 332 locally-managed tablespace, 65 multi-tablespace objects, 361 segments, 63 transporting

EXP tool, reasons for using, 320 IMP tool, reasons for using, 320

tape device, exporting to EXP tool, large exports, 328 requires UNIX system, 328

temp files cannot be restored, 65 database architecture, file types, 60, 65

template for external procedures C code, 794 C-based stored procedures, 783 debugf function, 795 debugf macro, 796 error handling, 784 global context, 794 init function, 798 lastOCiError function, 797

template for external procedures (Continued)

1294

template for external procedures (Continued) parameter setting, 784 raise_application_error function, 797 state management, 783 term function, 801 tracing mechanisms, 783

template objects invoker rights, 1007, 1008

temporary tables autonomous transactions, problems with, 691 CBO and, 255 compared to INLINE VIEW, 254 DBMS_STATS package and, 254 description, 251 differences from permanent tables, 253 disadvantages, 253 measuring redo/rollback, 182 reasons for using, 254 Redo, 182 rollback, 182 session-based temporary tables, 251 table types, 198, 251 transaction-based temporary tables, 251

term function template for external procedures, 801

TERMINATED BY clause loading tab-delimited data, 375 TERMINATED BY WHITESPACE, 375 TERMINATED BY X’09’, 376

TERMINATED BY EOF statement LOBFILEs, 415

text indexing interMedia Text, 753 stoplists, 755 synchronizing index, 753

ALTER INDEX statement, 754 CREATE INDEX statement, 754 ctxsrv program, 753

text matching full exact text match, 603 optimizer plan stability, problems with, 534 partial text match, 603

text search INSTR function, 741 interMedia Text, using, 742 LIKE operator, 741

TextServer3 interMedia Text, brief history, 740

theme generation interMedia Text, using, 747 linguistic analysis, 747

time, getting time down to milliseconds Java stored procedures, 863

TIMED_STATISTICS analytic functions, performance testing, 571 tuning with, 451

Timestamp class manipulating Oracle date types, 852

TIMESTAMP column DBA_OUTLINES table, 517

Timestamp type, Java passed from Oracle Date type

Java stored procedures, 850 TKPROF

analytic functions, performance testing, 571

analyzing query, 455 EXECUTE phase, 458 explain plan facility, 462 FETCH phase, 458 PARSE phase, 458 query plan, 460

command line options, 462 hash clustered tables, performance testing, 234 implementing tuning, 510 interpreting output, 458

column headings, meanings of, 458 important points, 459

parsing performance time, measuring, 1013 trace files, using and interpreting, 464 tuning with, 451 using, 454

TM locks configuring number of allowed locks, 119 DML locks, 118

TNSLISTENER see Net8 listener.

TNSNAMES.ORA file ADDRESS = (PROTOCOL = IPC) (KEY = EXTPROC1), 776 C-based stored procedures, configuring server, 775, 776 CONNECT_DATA = (SID = PLSExtProc), 776 database environment, verifying, 778 EXTPROC_CONNECTION_DATA, 776

TOP-N queries ambiguity, 566 analytic functions, 566 COUNT function, 568 DENSE_RANK function, 567 ROW_NUMBER function, 570

TOUSER option IMP tool, 319

trace files deadlocks, trace files for, 108 using and interpreting, 464

APPNAME record, 465 BIND record, 469 client application statements, 471 cursor record, 466 EXEC record, 466 PARSE errors, 472 Run-time errors, 472 STAT record, 472 TKPROF, 464 wait record, 467 XCTEND record, 472

tracing mechanisms Java stored procedures, 845 template for external procedures, 783

transaction control statements, 136 COMMIT statement, 136 ROLLBACK TO <SAVEPOINT> statement, 136 ROLLBACK statement, 136 SAVEPOINT statement, 136 SET TRANSACTION statement, 136

transaction isolation levels concurrency control, 124 dirty read, 124 non-repeatable read, 124 phantom read, 124 read committed isolation level, 124, 126

unused indexes, reasons for

1295

transaction isolation levels (Continued) read only isolation level, 125, 132 read uncommitted isolation level, 124, 125 repeatable read isolation level, 124, 128 serializable isolation level, 124, 130 SQL92 standard, 124

transactional mechanisms ACID properties, 135

Atomicity, 135 Consistency, 135 Durability, 135 Isolation, 135

advantages of large transactions, 158 autonomous transactions, 659 bad habits, 143

autocommit by default, 148 failure to ensure enough rollback space, 144 using loops for updates, 143

COMMIT statement, 158 distributed transactions, 148 implicit transactions, 143 integrity constraints, 140 introduction, 135 Oracle database, 35, 135 Redo, 150 rollback segments, 144 serializable transactions, 23 transaction control statements, 136 TX locks, 113 Undo, 150

transaction-based temporary tables, 251 ON COMMIT DELETE ROWS clause, 252

trigger reads from table in query autonomous transactions, 663 mutating tables, avoiding, 665

triggers autonomous transactions, 663 performing DDL, 666

potential failure with, 668 statement level atomicity, 138 trigger reads from table in query, 663

mutating tables, avoiding, 665 TRUNCATE option

SQLLDR, problems with, 426 TRUSTED value

QUERY_REWRITE_INTEGRITY parameter, 602 tuning, 429

analytic functions, problems with, 590 benchmarking, 434

benchmarking in isolation, 434 benchmarking to scale, 435

bind variables, 436 cursor_sharing=force parameter, 441 latch free event, 439 static SQL, 440

continuing tuning, 432 DBMS_PROFILER, 475 defensive programming, 434 dynamic SQL, problems with, 735 experimental approach, 433 implementing with optimizer plan stability, 509 instrumentation, 475

DEBUG package, 476 Oracle applications, 23, 47, 431 performance metrics, identifying, 434 problems with, 430 soft parse of query, avoiding, 441 SQL_TRACE, 451 StatsPack, 477 TIMED_STATISTICS, 451

TKPROF, 451 V$ tables, 494

V$EVENT_NAME table, 494 V$FILESTAT table, 495 V$LOCK table, 495 V$MYSTAT table, 495 V$OPEN_CURSOR table, 496 V$PARAMETER table, 498 V$SESS_IO table, 501 V$SESSION table, 498 V$SESSION_EVENT table, 500 V$SESSION_LONGOPS table, 500 V$SESSION_WAIT table, 500 V$SESSTAT table, 500 V$SQL table, 501 V$SQLAREA table, 501 V$STATNAME table, 501 V$SYSSTAT table, 501 V$SYSTEM_EVENT table, 501 V$TEMPSTAT table, 495

two-phase distributed commit protocol distributed transactions, 149

TX locks DML locks, 113 INITRANS parameter, 117 MAXTRANS parameter, 117 simple example, 114 transactional mechanisms, 113 using, 113

UUGA

database architecture, 70 testing usage, 71

UNBOUNDED PRECEDING windowing clause, 560

Undosee also rollback. generating Undo, 185

DELETE statement, 185 INSERT statement, 185 UPDATE statement, 185

redo log buffer, 151 rollback segments, 151 system crashes and, 151 transactional mechanisms, 150

undocumented parameters care in using, 61 parameter files, 61

uniqueness local index partition and, 643

unknown number of outputs dynamic PL/SQL, 718

requires DBMS_SQL, 718 unloader PL/SQL utility

control file, 400 data file, 400 RUN function, 403 unloading data in SQLLDR friendly format, 399 using, 407

UNRECOVERABLE clause preventing redo log generation, 174replaced by NOLOGGING clause, 174

unused indexes, reasons for, 303 functions used implicitly on columns, 304 functions used on columns, 303 index on nullable columns, 303 index used to be slower, 307

unused indexes, reasons for (Continued)

1296

unused indexes, reasons for (Continued) index would be slower, 305 leading edge of index not used in query, 303

UPDATE statement blocking, 107 generating Undo, 185 nested tables, 244

UPDATE_BY_CAT procedure OUTLN_PKG package, 529

URL_DATASTORE datastore object, 746 USED column

DBA_OUTLINES table, 516 USER data

data files, 62 User Global Area

see UGA. USER_INDEXES view

CLUSTERING_FACTOR column, 283 USER_OUTLINE_HINTS table

optimizer plan stability, 517 USER_OUTLINES table

optimizer plan stability, 516 user-defined locks

creating, 123 DBMS_LOCK package, 123 locking policy, 123

utility background processes database architecture, 95 EMNn, 96 QMNn, 96 SNPn, 96

UTL_FILE API 1023 byte limit, 1205

overloaded FOPEN function gets around byte limit, 1205

accessing mapped Windows drives, 1201 dynamic SQL, 707, 723 encapsulating functionality in new PL/SQL data type, 883 exception handling, 1202 limitations, 1199 necesssary supplied packages, 1199 reading directory, 1205 using instead of DBMS_OUTPUT, 1154 UTL_FILE_DIR parameter, 1200 web pages, dumping to disk, 1203

UTL_FILE_DIR parameter init.ora file, 1200 UTL_FILE API, 1200

UTL_HTTP package enhancing

HTTP_PKG package, 1219 necesssary supplied packages, 1208 running PL/SQL programs, 1217 SSL, 1210 using, 1209

UTL_RAW package necesssary supplied packages, 1229

UTL_SMTP package example, 1231 JavaMail API, 1236 necesssary supplied packages, 1231

UTL_TCP package C-based stored procedures, 772 necesssary supplied packages, 1244 network programming, 1244 SocketType object type, 1245

VV$ tables, tuning with, 494

V$EVENT_NAME table, 494 V$FILESTAT table, 495 V$LOCK table, 495 V$MYSTAT table, 495 V$OPEN_CURSOR table, 496 V$PARAMETER table, 498 V$SESS_IO table, 501 V$SESSION table, 498 V$SESSION_EVENT table, 500 V$SESSION_LONGOPS table, 500 V$SESSION_WAIT table, 500 V$SESSTAT table, 500 V$SQL table, 501 V$SQLAREA table, 501 V$STATNAME table, 501 V$SYSSTAT table, 501 V$SYSTEM_EVENT table, 501 V$TEMPSTAT table, 495

V$EVENT_NAME table, tuning with, 494 V$FILESTAT table, tuning with, 495 V$LOCK table, tuning with, 495 V$LOGMNR_CONTENTS table

LogMiner packages, 1119 V$MYSTAT table

measuring amount of Redo, 164 tuning with, 495

V$OPEN_CURSOR table, tuning with, 496 V$PARAMETER table, tuning with, 498 V$SESS_IO table, tuning with, 501 V$SESSION table, tuning with, 498 V$SESSION_EVENT table, tuning, 500 V$SESSION_LONGOPS table

DBMS_APPLICATION_INFO package, 1045 tuning with, 500

V$SESSION_WAIT table, tuning with, 500 V$SESSTAT table, tuning with, 500 V$SQL table, tuning with, 501 V$SQLAREA table, tuning with, 501 V$STATNAME table

measuring amount of Redo, 164 tuning with, 501

V$SYSSTAT table, tuning with, 501 V$SYSTEM_EVENT table, tuning with, 501 V$TEMPSTAT table, tuning with, 495 VALIDATE_DIMENSION routine

DBMS_OLAP package, 621 VAR attribute

loading data with embedded newlines, 397 VAR_POP function

analytic functions, 565 VAR_SAMP function

analytic functions, 565 Varchar2 type, Oracle

BLOB to VARCHAR2 conversions, 1077 passing to Java String type

Java stored procedures, 850 VARIANCE function

analytic functions, 565 VARRAYS

compared to nested tables, 876, 880 loading

SQLLDR, 418 object relational features, 876

zone section

1297

VARRAYS (Continued) storing, 880

BLOB type, 880 RAW type, 880

using, 877 vendor-specific database features

making use of, 41 VERSION column

DBA_OUTLINES table, 517 views

importing into structures with changed data type, 343 importing into structures with missing column, 343 presenting object relational views, 868 views and indexes, 299

virtual operating system Oracle database as virtual operating system, 20

Virtual Private Database (VPD) see Fine Grained Access Control.

VPD (Virtual Private Database) see Fine Grained Access Control.

WWait events

StatsPack report, interpreting, 482 wait record


web pages dumping to disk with UTL_FILE API, 1203

web-based authentication, 964 n-Tier authentication, 965 problems with, 965

WHEN OTHERS exception handler avoiding using, 473

Where clause cannot contain analytic functions, 588

windowing clause analytic functions, syntax, 553 BETWEEN, 561 CURRENT ROW, 560 Numeric Expression FOLLOWING, 560 Numeric Expression PRECEDING, 560 range windows, 555 row windows, 558 UNBOUNDED PRECEDING, 560

windowing functions analytic functions, 550

WRITE_IO procedure FILETYPE object type, 884, 887

XXCTEND record


XML indexing interMedia Text, using, 749

XML section searching interMedia Text, 760

Zzone section

interMedia Text, 761

License Agreement (Single-User Products)THIS IS A LEGAL AGREEMENT BETWEEN YOU, THE END USER, AND APRESS. BY OPENING THE SEALED DISK PACKAGE, YOU ARE AGREEING TO BE BOUND BY THE TERMS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO THE TERMS OF THIS AGREEMENT, PROMPTLY RETURN THE UNOPENED DISK PACKAGE AND THE ACCOMPANYING ITEMS (INCLUDING WRITTEN MATERIALS AND BINDERS AND OTHER CONTAINERS) TO THE PLACE YOU OBTAINED THEM FOR A FULL REFUND.

APRESS SOFTWARE LICENSE1. GRANT OF LICENSE. Apress grants you the right to use one copy of this enclosed Apress

software program (the “SOFTWARE”) on a single terminal connected to a single computer (e.g., with a single CPU). You may not network the SOFTWARE or otherwise use it on more than one computer or computer terminal at the same time.

2. COPYRIGHT. The SOFTWARE copyright is owned by Apress and is protected by United States copyright laws and international treaty provisions. Therefore, you must treat the SOFTWARE like any other copyrighted material (e.g., a book or musical recording) except that you may either (a) make one copy of the SOFTWARE solely for backup or archival purposes, or (b) transfer the SOFTWARE to a single hard disk, provided you keep the original solely for backup or archival purposes. You may not copy the written material accompanying the SOFTWARE.

3. OTHER RESTRICTIONS. You may not rent or lease the SOFTWARE, but you may transfer the SOFTWARE and accompanying written materials on a permanent basis provided you retain no copies and the recipient agrees to the terms of this Agreement. You may not reverse engineer, decompile, or disassemble the SOFTWARE. If SOFTWARE is an update, any transfer must include the update and all prior versions.

4. By breaking the seal on the disc package, you agree to the terms and conditions printed in the Apress License Agreement. If you do not agree with the terms, simply return this book with the still-sealed CD package to the place of purchase for a refund.

DISCLAIMER OF WARRANTY NO WARRANTIES. Apress disclaims all warranties, either express or implied, including, but not limited to, implied warranties of merchantability and fitness for a particular purpose, with respect to the SOFTWARE and the accompanying written materials. The software and any related documentation is provided “as is.” You may have other rights, which vary from state to state.

NO LIABILITIES FOR CONSEQUENTIAL DAMAGES. In no event shall be liable for any damages whatsoever (including, without limitation, damages from loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use or inability to use this Apress product, even if Apress has been advised of the possibility of such damages. Because some states do not allow the exclusion or limitation of liability for consequential or incidental damages, the above limitation may not apply to you.

U.S. GOVERNMENT RESTRICTED RIGHTSThe SOFTWARE and documentation are provided with RESTRICTED RIGHTS. Use, duplication, or disclosure by the Government is subject to restriction as set forth in subparagraph (c) (1) (ii) of The Rights in Technical Data and Computer Software clause at 52.227-7013. Contractor/manufacturer is Apress, 2560 Ninth Street, Suite 219, Berkeley, California, 94710.

This Agreement is governed by the laws of the State of California.

Should you have any questions concerning this Agreement, or if you wish to contact Apress for any reason, please write to Apress, 2560 Ninth Street, Suite 219, Berkeley, California, 94710.

Necessary Supplied Packages - Springer

Documents

Transcript of Necessary Supplied Packages - Springer