[Previous] [Contents] [Index] [Next]

Filtering

This chapter includes:

Overview

Gathering many events generates a lot of data, which requires memory and processor time. It also makes the task of interpreting the data more difficult.

Because the amount of data that the instrumented kernel generates can be overwhelming, the SAT supports several types of filters that you can use to reduce the amount of data to be processed:

Static rules filter
A simple filter that chooses events based on their type, class, or other simple criteria.
Dynamic rules filter
A more complex filter that lets you register a callback function that can decide — based on the state of your application or system, or on whatever criteria you choose — whether or not to log a given event.
Post-processing filter
A filter that you run after capturing event data. Like the dynamic rules filter, this can be as complex and sophisticated as you wish.

The static and dynamic rules filters affect the amount of data being logged into the kernel buffers; filtered data is discarded — you save processing time and memory, but there's a chance that some of the filtered data could have been useful.

In contrast, the post-processing facility doesn't discard data; it simply doesn't use it — if you've saved the data, you can use it later.

Filters

Overall view of the SAT and its filters.

Most of the events don't indicate what caused the event to occur. For example, an event for entering MsgSendv() doesn't indicate which thread in which process called it; you have to infer it during interpretation from a previous thread-running event. You have carefully choose what you filter to avoid losing this context.

The static rules filter

You can use the static rules filter to track or filter events for all classes, certain events in a class, or even events related to specific process and thread IDs. You can select events in an additive or subtractive manner; you can start with no events, and then add specific classes or events, or you can start with all events, and then exclude specific ones.

The static rules filter is the best, most efficient method of data reduction. It generally frees up the processor while significantly reducing the data rate. This filter is also useful for gathering large amounts of data periodically, or after many hours of logging without generating gigabytes of data in the interim.

You set up this filter using the following TraceEvent() commands:

_NTO_TRACE_ADDALLCLASSES, _NTO_TRACE_DELALLCLASSES
Emit or suppress tracing for all classes and events: 
TraceEvent(_NTO_TRACE_ADDALLCLASSES);
TraceEvent(_NTO_TRACE_DELALLCLASSES);
Note: The _NTO_TRACE_DELALLCLASSES command doesn't suppress the process- and thread-specific tracing that the _NTO_TRACE_SETCLASSPID, _NTO_TRACE_SETCLASSTID, _NTO_TRACE_SETEVENTPID, and _NTO_TRACE_SETEVENTTID commands set up. You need to clear their tracing separately, as shown below.
_NTO_TRACE_ADDCLASS, _NTO_TRACE_DELCLASS
Emit or suppress all trace events from a specific class: 
TraceEvent(_NTO_TRACE_ADDCLASS, class):
TraceEvent(_NTO_TRACE_DELCLASS, class):

For information about the different classes, see “Classes and events” in the Events and the Kernel chapter of this guide.

_NTO_TRACE_ADDEVENT, _NTO_TRACE_DELEVENT
Emit or suppress a specific event in a specific class: 
TraceEvent(_NTO_TRACE_ADDEVENT, class, event);
TraceEvent(_NTO_TRACE_DELEVENT, class, event);
_NTO_TRACE_SETCLASSPID, _NTO_TRACE_CLRCLASSPID
Emit or suppress all events from a specified process ID: 
TraceEvent(_NTO_TRACE_SETCLASSPID, int class, pid_t pid);
TraceEvent(_NTO_TRACE_CLRCLASSPID, int class);
_NTO_TRACE_SETCLASSTID, _NTO_TRACE_CLRCLASSTID
Emit or suppress all events from the specified process and thread IDs: 
TraceEvent(_NTO_TRACE_SETCLASSTID, int class, pid_t pid, tid_t tid);
TraceEvent(_NTO_TRACE_CLRCLASSTID, int class);
_NTO_TRACE_SETEVENTPID, _NTO_TRACE_CLREVENTPID
Emit or suppress a specific event for a specified process ID: 
TraceEvent(_NTO_TRACE_SETEVENTPID, int class, int event, pid_t pid);
TraceEvent(_NTO_TRACE_CLREVENTPID, int class, int event);
_NTO_TRACE_SETEVENTTID, _NTO_TRACE_CLREVENTTID
Emit or suppress a specific event for the specified process and thread IDs: 
TraceEvent(_NTO_TRACE_SETEVENTTID, int class, int event,
           pid_t pid, tid_t tid);
TraceEvent(_NTO_TRACE_CLREVENTTID, int class, int event);
Note: The _NTO_TRACE_SETCLASSPID, _NTO_TRACE_SETCLASSTID, _NTO_TRACE_SETEVENTPID, and _NTO_TRACE_SETEVENTTID commands apply only to these classes:
  • _NTO_TRACE_COMM
  • _NTO_TRACE_KERCALL, _NTO_TRACE_KERCALLENTER, _NTO_TRACE_KERCALLEXIT
  • _NTO_TRACE_SYSTEM
  • _NTO_TRACE_THREAD
  • _NTO_TRACE_VTHREAD

The instrumented kernel retains these settings, so you should be careful not to make any assumptions about the settings that are in effect when you set up your filters. For example, you might want to start by turning off all filtering: 

TraceEvent(_NTO_TRACE_DELALLCLASSES);
TraceEvent(_NTO_TRACE_CLRCLASSPID, _NTO_TRACE_KERCALL);
TraceEvent(_NTO_TRACE_CLRCLASSTID, _NTO_TRACE_KERCALL);
TraceEvent(_NTO_TRACE_CLRCLASSPID, _NTO_TRACE_THREAD);
TraceEvent(_NTO_TRACE_CLRCLASSTID, _NTO_TRACE_THREAD);
TraceEvent(_NTO_TRACE_CLRCLASSPID, _NTO_TRACE_VTHREAD);
TraceEvent(_NTO_TRACE_CLRCLASSTID, _NTO_TRACE_VTHREAD);
TraceEvent(_NTO_TRACE_CLRCLASSPID, _NTO_TRACE_SYSTEM);
TraceEvent(_NTO_TRACE_CLRCLASSTID, _NTO_TRACE_SYSTEM);
TraceEvent(_NTO_TRACE_CLRCLASSPID, _NTO_TRACE_COMM);
TraceEvent(_NTO_TRACE_CLRCLASSTID, _NTO_TRACE_COMM);

You can select events in an additive or subtractive manner; you can start with no events, and then add specific classes or events, or you can start with all events, and then exclude specific ones.

For an example using the static filter, see the five_events.c example in the Tutorials chapter.

The dynamic rules filter

The dynamic rules filter can do all the filtering that the static filter does — and more — but it isn't as quick. This filter lets you register functions (event handlers) that decide whether or not to log a given event.

Note: If you want to use a dynamic rules filter, be sure that you've also set up a static rules filter that logs the events you want to examine. For example, if you want to dynamically examine events in the _NTO_TRACE_THREAD class, also call: 
TraceEvent(_NTO_TRACE_ADDCLASS, _NTO_TRACE_THREAD):

For an example of using the dynamic rules filter, see the eh_simple.c example in the Tutorials chapter.

Setting up a dynamic rules filter

To set up the dynamic rules filter, you must call ThreadCtl() with the _NTO_TCTL_IO flag to get I/O privileges: 

if (ThreadCtl(_NTO_TCTL_IO, 0)!=EOK) {
   fprintf(stderr, "argv[0]: Failed to obtain I/O privileges\n");
   return (-1);
}

and then call TraceEvent() with one of these commands:

_NTO_TRACE_ADDCLASSEVHANDLER
Register a function to call whenever an event for the given class is emitted: 
TraceEvent(_NTO_TRACE_ADDCLASSEVHANDLER, class,
           int (*event_hdlr)(event_data_t*),
           event_data_t* data_struct);
_NTO_TRACE_ADDEVENTHANDLER
Register a function to call whenever an event for the given class and event type is emitted: 
TraceEvent(_NTO_TRACE_ADDEVENTHANDLER, class, event, 
           int (*event_hdlr)(event_data_t*), 
           event_data_t* data_struct);

The additional arguments are:

event_hdlr
A pointer to the function that you want to register. The prototype for the function is: 
int event_hdlr (event_data_t *event_data);
data_struct
A pointer to a locally defined data structure, of type event_data_t, where the kernel can store event data to pass to the event handler (see below).

Event handler

The dynamic filter is an event handler that works like an interrupt handler. When this filter is used, a section of your custom code is executed. The code can test for a set of conditions before determining whether the event should be stored.

Caution: The only library functions that you can call in your event handler are those that are safe to call from an interrupt handler. For a list of these functions, see the Summary of Safety Information appendix in the QNX Neutrino Library Reference. if you call an unsafe function — such as printf() — in your event handler, you'll crash your entire system.

If you want to log the current event, return a non-zero value; to discard the event, return 0. Here's a very simple event handler that says to log all of the given events: 

int event_handler(event_data_t* dummy_pt)
{
 return(1);
}
Note: If you use both types of dynamic filters (event handler and class event handler), and they both apply to a particular event, the event is logged if both event handlers return a non-zero value.

In addition to deciding whether or not the event should be logged, you can use the dynamic rules filter to output events to external hardware or to perform other tasks— it's up to you because it's your code. Naturally, you should write the code as efficiently as possible in order to minimize the overhead.

You can access the information about the intercepted event within the event handler by examining the event_data_t structure passed as an argument to the event handler. The layout of the event_data_t structure (declared in <sys/trace.h>) is as follows: 

/* event data filled by an event handler */
typedef struct 
{
   __traceentry header;      /* same as traceevent header    */
   _Uint32t*    data_array;  /* initialized by the user      */
   _Uint32t     el_num;      /* number of elements returned  */
   void*        area;        /* user data                    */
   _Uint32t     feature_mask;/* bits indicate valid features */
   _Uint32t     feature[_NTO_TRACE_FI_NUM]; /* feature array 
                                           - additional data */
} event_data_t;
Caution: The event_data_t structure includes a pointer to an array for the data arguments of the event. You must provide an array, and it must be large enough to hold the data for the event or events that you're handling (see the Current Trace Events and Data appendix). For example: 
event_data_t e_d_1;
_Uint32t     data_array_1[20]; /* 20 elements for potential args. */

e_d_1.data_array = data_array_1;

If you don't provide the data array, or it isn't big enough, your data segment could become corrupted.

You can use the following macros, defined in <sys/trace.h>, to work with the header of an event:

_NTO_TRACE_GETEVENT_C(c)
Get the class.
_NTO_TRACE_GETEVENT(c)
Get the type of event.
_NTO_TRACE_GETCPU(h)
Get the number of the CPU that the event occurred on.
_NTO_TRACE_SETEVENT_C(c,cl)
Set the class in the header c to be cl.
_NTO_TRACE_SETEVENT(c, e)
Set the event type in the header c to be e.

The bits of the feature_mask member are related to any additional features (arguments) that you can access inside the event handler. All standard data arguments — the ones that correspond to the data arguments of the trace event — are delivered without changes within the data_array.

There are two constants associated with each additional feature:

The currently defined features are:

Feature Parameter mask Index
Process ID _NTO_TRACE_FMPID _NTO_TRACE_FIPID
Thread ID _NTO_TRACE_FMTID _NTO_TRACE_FITID

If any particular bit of the feature_mask is set to 1, then you can access the feature corresponding to this bit within the feature array. Otherwise, you must not access the feature. For example, if the expression: 

feature_mask & _NTO_TRACE_FMPID

is TRUE, then you can access the additional feature corresponding to identifier _NTO_TRACE_FMPID as: 

my_pid = feature[_NTO_TRACE_FIPID];

Removing event handlers

To remove event handlers, call TraceEvent() with these commands:

_NTO_TRACE_DELCLASSEVHANDLER
Remove the function for the given class and event type: 
TraceEvent(_NTO_TRACE_DELCLASSEVHANDLER, class);
_NTO_TRACE_DELEVENTHANDLER
Remove the function for the given class and event type: 
TraceEvent(_NTO_TRACE_DELEVENTHANDLER, class, event);

The post-processing facility

The post-processing facility is different from the other filters in that it reacts to the events without permanently discarding them (or having to choose not to). Because the processing is done on the captured data, often saved as a file, you could make multiple passes on the same data without changing it—one pass could count the number of thread state changes, another pass could display all the kernel events.

The post-processing facility is really a collection of callback functions that decide what to do for each event. One example of post-processing is the traceprinter utility itself. It prints all the events instead of filtering them, but the principles are the same.

The source code for traceprinter is a good starting point for you if you want to create your own post-processing facility. As mentioned earlier, you can get it from Foundry27; go to http://community.qnx.com, log into your myQNX account, go to the project for the QNX Neutrino OS, and look for utils/t/traceprinter.

We'll look at traceprinter in more detail in the Interpreting Trace Data chapter.

[Previous] [Contents] [Index] [Next]