trace Daemon

Purpose

Records selected system events.

Syntax

trace [ -a [ -g ] ] [ -f | -l ] [ -b | -B] [ -c] [ -C [ CPUList | all ]] [ -d ] [ -e string-cmd ] [ -h ] [ -j EventList ] [ -k EventgroupList ] [ -J EventgroupList ] [ -K EventgroupList ] [ -m Message ] [ -M ] [ -n ] [ -o Name ] [ -o- ] [ -p ] [ -r reglist ] [ -s ] [ -A ProcessIDList ] [ -t ThreadIDList ] [ -x program-specification | -X program-specification ] [ -I ] [ -P trace-propagation ][ -L Size ] [ -T Size ] [ -W ] [ -@ WparList ]

Description

The trace daemon configures a trace session and starts the collection of system events. The data collected by the trace function is recorded in the trace log. A report from the trace log can be generated with the trcrpt command.

When invoked with the -a, -x, or -X flags, the trace daemon is run asynchronously (for example, as a background task). Otherwise, it is run interactively and prompts you for subcommands.

To put the WPAR's configured ID (CID) in the trace hooks, use the -W flag.

To trace specific WPARs, use the -@ flag with a list of WPAR names that you want to trace.

You can use the System Management Interface Tool (SMIT) to run the trace daemon. To use SMIT, enter:

smit trace

The following are modes of trace data collection:

Item	Description
Alternate (the default)	All trace events are captured in the trace log file.
Circular ( -l)	The trace events wrap within the in-memory buffers and are not captured in the trace log file until the trace data collection is stopped.
Single ( -f)	The collection of trace events stops when the in-memory trace buffer fills up and the contents of the buffer are captured in the trace log file.
Buffer Allocation	Trace buffers are allocated from either the kernel heap, or are put into separate segments. By default, buffers are allocated from the kernel heap unless the buffer size requested is too large for buffers to fit in the kernel heap, in which case they are allocated in separate segments. Allocating buffers from separate segments hinders trace performance somewhat. However, buffers in separate segments will not take up paging space, just pinned memory. The type of buffer allocation can be specified with the optional -b or -B flags.

Beginning with AIX® 5.3, you can elect to trace only selected processes or threads. You can also trace a single program. You can specify whether the trace is to be propagated or extended to newly created processes or threads. You can optionally include interrupt events in such traces. This is only valid for trace channel 0.

Note:

Unless the trace is started before the process that is being traced, the process startup events are not captured. If the trace is started before the process that is being traced, some events from processes other than the process being traced will be captured as well.
When trace uses memory from the kernel heap which is the case for the -B option (32-bit kernel only), this memory remains part of kernel memory until the next reboot of the system. Thus, care should be taken when using large buffers. Furthermore, it should be noted that, prior to AIX 5.2 with the 5200-05 Maintenance Level, when using the multi-processor trace for all processors, the -C all option, memory is allocated for all possible processors, not just those running at the time the trace was started. This is fixed in AIX 5.2 with the 5200-05 Maintenance Level and AIX 5.3.

Flags

Item	Description
-@ WparList	Traces the workload partitions that you specify in the WparList parameter. Multiple WPAR names can either be separated by commas or enclosed in quotation marks and separated by spaces. To include the current Global system in the trace, specify Global. You can only specify the -@ flag in the Global system in a workload partition environment.
-a	Runs the trace daemon asynchronously (i.e. as a background task). Once trace has been started this way, you can use the trcon, trcoff, and trcstop commands to respectively start tracing, stop tracing, or exit the trace session. These commands are implemented as links to trace.
-A ProcessIDList	Traces only the processes and, optionally, their children specified with the ProcessIDList. A process ID is a decimal number. Multiple process IDs can either be separated by commas or enclosed in quotation marks and separated by spaces. The -A flag is only valid for trace channel `0`; the -A and -g flags are incompatible. All threads existing for the specified processes when tracing is started are traced. By default, if after the trace starts, the processes being traced create additional threads or processes, these are not traced unless the -P flag is specified.
-b	Allocate buffers from the kernel heap. If the requested buffer space can not be obtained from the kernel heap, the command fails. Restriction: The -b flag is only valid with the 32–bit kernel.
-B	Allocate buffers in separate segments. Restriction: The -B flag is only valid with the 32–bit kernel.
-c	Saves the trace log file, adding .old to its name.
-C [ CPUList \| all ]	Traces using one set of buffers per processor in the CPUList. The processors can be separated by commas, or enclosed in double quotation marks and separated by commas or blanks. To trace all processors, specify all. Since this flag uses one set of buffers per processor, and produces one file per processor, it can consume large amounts of memory and file space, and should be used with care. The files produced are named trcfile, trcfile-0, trcfile-1, etc., where 0, 1, etc. are the processor numbers. If -T or -L are specified, the sizes apply to each set of buffers and each file. On a uniprocessor system, you may specify `-C all`, but `-C` with a list of processor numbers is ignored. Prior to AIX 5.3, if -C is used to specify more than one processor, such as `-Call` or `-C "0 1"`, the associated buffers are not extracted from a system dump with trcdead. Attention: The -C flag can only be used by the root user.
-d	Disables the automatic start of trace data collection. Delays starting of trace data collection. Normally, the collection of trace data starts automatically when you issue the trace daemon. Use the trcon command to start the collection of trace data.
-e string-cmd	Configures Component Trace by running ctctrl with string-cmd as an argument before the trace is started. In other words, it runs ctctrl string-cmd. Passing multiple -e options is allowed and is equivalent to successively running the ctctrl command with each string-cmd of arguments. This option can be used to configure the system trace mode (by setting the system trace mode to On, changing the level of trace, and so on) for some components just before starting to trace the system.
-f	Runs trace in a single mode. Causes the collection of trace data to stop as soon as the in-memory buffer is filled up. The trace data is then written to the trace log. Use the trcon command to restart trace data collection and capture another full buffer of data. If you issue the trcoff subcommand before the buffer is full, trace data collection is stopped and the current contents of the buffer are written to the trace log.
-g	Starts a trace session on a generic trace channel (channels 1 through 7). This flag works only when trace is run asynchronously (-a). The return code of the command is the channel number; the channel number must subsequently be used in the generic trace subroutine calls. To stop the generic trace session, use the command trcstop -<channel_number>.
-h	Omits the header record from the trace log. Normally, the tracedaemon writes a header record with the date and time (from the date command) at the beginning of the trace log; the system name, version and release, the node identification, and the machine identification (from the uname -a command); and a user-defined message. At the beginning of the trace log, the information from the header record is included in the output of the trcrpt command.
-I	Trace interrupt events. When specified with -A or -t, the -I flag includes interrupt events along with the events for the processes or threads specified. When -I is specified, but neither -A nor -t is specified, only interrupt level events are traced. The -I flag is only valid for trace channel 0; the -I and -g flags are incompatible.
-j EventList	Specifies the user-defined events to collect trace data. The list items specified in the EventList parameter can either be separated by commas or enclosed in quotation marks and separated by commas or spaces. In AIX 6.1 and earlier releases, specifying a two-digit hook ID in the form hh specifies hh00, hh10,...,hhF0. Specifying a three-digit hook ID in the form hhh specifies hhh0. Specifying a four-digit hook ID in the form hhhh specifies hhhh. If any of these events is missing, the information reported by the trcrpt command will be incomplete. Consequently, when using the -j flag, include all these events in the EventList. If starting the trace with SMIT, or the -J flag, these events are in the tidhk group.
-J EventgroupList	Specifies the event groups to be included. The list items specified in the EventgroupList parameter can either be separated by commas or enclosed in quotation marks and separated by commas or spaces. The -J and -K flags work like -j and -k, except with event groups instead of individual hook IDs. You can specify each flag -j, -J, -k, and -K within the command.
-k EventgroupList	Specifies the user-defined events to exclude trace data. The list items specified in the EventgroupList parameter can either be separated by commas or enclosed in quotation marks and separated by commas or spaces. In AIX 6.1 and earlier releases, specifying a two-digit hook ID in the form hh specifies hh00, hh10,...,hhF0. Specifying a three-digit hook ID in the form hhh specifies hhh0. Specifying a four-digit hook ID in the form hhhh specifies hhhh. Tip: The following events are used to determine the pid, the cpuid, and the exec path name in the trcrpt report: `106 DISPATCH 10C DISPATCH IDLE PROCESS 134 EXEC SYSTEM CALL 139 FORK SYSTEM CALL 465 KTHREAD CREATE` If any of these events is missing, the information reported by the trcrpt command will be incomplete. When using the -k flag, do not include these events in the EventgroupList parameter. If starting the trace with SMIT, or the -J flag, these events are in the tidhk group.
-K EventgroupList	Specifies the event groups to be excluded. The list items specified in the EventgroupList parameter can either be separated by commas or enclosed in quotation marks and separated by commas or spaces. The -J and -K flags work like -j and -k, except with event groups instead of individual hook IDs. You can specify each flag -j, -J, -k, and -K within the command.
-l	Runs trace in a circular mode. The trace daemon writes the trace data to the trace log when the collection of trace data is stopped. Only the last buffer of trace data is captured. When you stop trace data collection using the trcoff command, restart it using the trconcommand.
-L Size	Overrides the default trace log file size of 1 MB with the value stated. Specifying a file size of zero sets the trace log file size to the default size. Note: In the circular and the alternate modes, the trace log file size must be at least twice the size of the trace buffer. In the single mode, the trace log file must be at least the size of the buffer. See the -T flag for information on controlling the trace buffer size.
-m Message	Specifies text to be included in the message field of the trace log header record.
-M	Dumps the address map of running processes into the trace. The -M flag must be specified if the trace file is to be processed by the tprof command.
-n	Adds information to the trace log header: lock information, hardware information, and, for each loader entry, the symbol name, address, and type.
-o Name	Overrides the /var/adm/ras/trcfile default trace log file and writes trace data to a user-defined file.
-o -	Overrides the default trace log name and writes trace data to standard output. The -c flag is ignored when using this flag. An error is produced if -o - and -C are specified.
-p	Includes the cpuid of the current processor with each hook. This flag is only valid for 64-bit kernel traces. Note: The trcrpt command can report the cpuid whether or not this option is specified.
-P propagation	The propagation is specified with the letters `p` for propagation across process creation, `t` for propagation across thread creation, and `n` for no propagation. Propagation across process creation implies propagation across thread creation. For example, if -A is specified to trace a process, all threads for that process that exist at the time the trace was started are traced. The -Pt flags causes all threads subsequently created by that process to be traced as well. If -Pp is specified, all processes and threads subsequently created by that process are traced. If -t all was specified to trace all threads, -P is ignored. The -P flag is only valid for trace channel `0`; the -P and -g flags are incompatible.
-r reglist	Optional, and only valid for a trace run on a 64-bit kernel. reglist items are separated by commas, or enclosed in quotation marks and separated by blanks. Up to 8 registers may be specified. Valid reglist values are: PURR - The PURR Register for this processor SPURR The SPURR register for this processor MCR0, MCR1, MCRA - the MCR Registers, 0, 1, and A PMC1, PMC2, ... PMC8 - PMC Registers 1 through 8. Restriction: Not all registers are valid for all processors.
-s	Stops tracing when the trace log fills. The trace daemon normally wraps the trace log when it fills up and continues to collect trace data. During asynchronous operation, this flag causes the trace daemon to stop trace data collection. (During interactive operation, the quit subcommand must be used to stop trace.)
-t ThreadIDList	Traces only the threads specified with the ThreadIDList parameter. A thread ID is a decimal number. Multiple thread IDs can either be separated by commas or enclosed in quotation marks and separated by spaces. Also, the thread list can be `all` or ``, indicating that all threads are to be traced. This is useful for tracing all thread-related events without tracing interrupt-related events. However, if -t all* and -I are both specified, this is the same as specifying neither one; all events are traced. Another way to say this is that trace and trace -It all are identical. The -t flag is only valid for trace channel `0`, the -t and -g flags are incompatible.
-T Size	Overrides the default trace buffer size of 128 KB with the value stated. You must be root to request more than 1 MB of buffer space. The maximum possible size is 268435184 bytes, unless the -f flag is used, in which case it is 536870368 bytes. The smallest possible size is 8192 bytes, unless the -f flag is used, in which case it is 16392 bytes. Sizes between 8192 and 16392 will be accepted when using the -f flag; however, the actual size used will be 16392 bytes. Note: In the circular and the alternate modes, the trace buffer size must be one-half or less the size of the trace log file. In the single mode, the trace log file must be at least the size of the buffer. See the -L flag for information on controlling the trace log file size. Also note that trace buffers use pinned memory, which means they are not pageable. Therefore, the larger the trace buffers, the less physical memory is available to applications. Unless the -b or -B flags are specified, the system attempts to allocate the buffer space from the kernel heap. If this request can not be satisfied, the system then attempts to allocate the buffers as separate segments. The -f flag actually uses two buffers, which behave as a single buffer (except that a buffer wraparound trace hook will be recorded when the first buffer is filled).
-W	Use the -W flag to include the workload partition's configured ID (CID) for the current process with each hook. This flag is only valid in the Global system in a workload partition environment. Tip: The trcrpt command can report the workload partition's CID whether or not this option is specified.
-x program-specification	Traces the specified program. The program-specification specifies a program and parameters as they would be when running the program from the shell, except that the program specification must be in quotes if more than just the program's name is given. The trace is stopped automatically when the program exits, and returns the program's return code. By default, any processes and threads created by the program are also traced; as if -Pp was specified. To change this behavior, use -Pn to specify no trace propagation, or -Pt to propagate trace only to threads created by the program's original process. Tip: The -x flag implies asynchronous tracing, as if the -a flag had also been specified.
-X program-specification	The -X flag works like the -x flag, except that the trace is not automatically stopped when the program exits. This is useful for tracing programs which fork processes, and then terminate, and you want these new processes traced as well.

Subcommands

When run interactively, trace recognizes the following subcommands:

Item	Description
trcon	Starts the collection of trace data.
trcoff	Stops the collection of trace data.
q or quit	Stops the collection of trace data and exits trace.
! Command	Runs the shell command specified by the Command parameter.
?	Displays the summary of trace subcommands.

Signals

The INTERRUPT signal acts as a toggle to start and stop the collection of trace data. Interruptions are set to SIG_IGN for the traced process.

Security

Attention RBAC users and Trusted AIX users: This command can perform privileged operations. Only privileged users can run privileged operations. For more information about authorizations and privileges, see Privileged Command Database in AIX Version 7.1 Security. For a list of privileges and the authorizations associated with this command, see the lssecattr command or the getcmdattr subcommand.

Examples

To use trace interactively, enter trace, (the > prompt is displayed), then specify the subcommands you want. For example, to trace system events during the run of the anycmd command, enter:
```
trace
> !anycmd
> q
```
To avoid delays when the command finishes, start trace asynchronously ( -a), using only one command line, enter:
```
trace -a; anycmd; trcstop
```
To trace the system itself for a period of 10 seconds, enter:
```
trace -a; sleep 10; trcstop
```
To output trace data to a specific trace log file (instead of the /var/adm/ras/trcfile default trace log file), :
```
trace -a -o /tmp/my_trace_log; anycmd; trcstop
```
To capture the execution of a cp command, excluding specific events from the collection process:
```
trace -a -k "20e,20f" -x "cp /bin/track /tmp/junk"
```
In the example above, the -k option suppresses the collection of events from the lockl and unlockl functions (20e and 20f events).

Also notice that the -x flag was used, so only hooks associated with the cp command process will be traced, and no interrupt activity will be traced.
To trace hook 234 and the hooks that will allow you to see the process names, use:
```
trace -a -j 234 -J tidhk
```
This traces the hooks in the event-group "tidhk" plus hook 234.
To have trace use one set of buffers per processor, specify:
```
trace -aC all
```
The files produced are /var/adm/ras/trcfile, /var/adm/ras/trcfile-0, /var/adm/ras/trcfile-1, etc. up to /var/adm/ras/trcfile-(n-1), where n is the number of procssors in the system.
Tip: trace -aCall -o mylog produces the files mylog, mylog-0, mylog-1, ...
To trace a program that starts a daemon process, and to continue tracing the daemon after the original program has finished, use
```
trace -X "mydaemon"
```
The trace must be stopped with trcstop.
To trace mydaemon, which is currently running, use:
```
trace -A mydaemon-process-id -Pp
```
Where mydaemon-process-id is the process for mydaemon as returned by the ps command. The -Pp flag tells trace to also trace any processes and threads created by mydaemon while the trace is running.
To capture the PURR, and PMC1 and PMC2, type:
```
trace -ar "PURR PMC1 PMC2"
```
To trace hooks 1A00,1A10,...,1AF0, DCA0 and 1AB1, enter:
```
trace -aj 1A,DCA,1AB1 
```

Files

Item	Description
/usr/include/sys/trcmacros.h	Defines trchook and utrchook macros.
/var/adm/ras/trcfile	Contains the default trace log file.