Transfers a user message to a remote task, obtaining the target address on the remote task from a user-specified header handler.
Availability Library (liblapi_r.a)
#include <lapi.h>
typedef void (compl_hndlr_t) (hndl, user_info);
lapi_handle_t *hndl; /* pointer to LAPI context passed in from LAPI_Amsend */
void *user_info; /* buffer (user_info) pointer passed in */
/* from header handler (void *(hdr_hndlr_t)) */
typedef void *(hdr_hndlr_t)(hndl, uhdr, uhdr_len, msg_len, comp_h, user_info);
lapi_handle_t *hndl; /* pointer to LAPI context passed in from LAPI_Amsend */
void *uhdr; /* uhdr passed in from LAPI_Amsend */
uint *uhdr_len; /* uhdr_len passed in from LAPI_Amsend */
ulong *msg_len; /* udata_len passed in fom LAPI_Amsend */
compl_hndlr_t **comp_h; /* function address of completion handler */
/* (void (compl_hndlr_t)) that needs to be filled */
/* out by this header handler function. */
void **user_info; /* pointer to the parameter to be passed */
/* in to the completion handler */
int LAPI_Amsend(hndl, tgt, hdr_hdl, uhdr, uhdr_len, udata, udata_len,
tgt_cntr, org_cntr, cmpl_cntr)
lapi_handle_t hndl;
uint tgt;
void *hdr_hdl;
void *uhdr;
uint uhdr_len;
void *udata;
ulong udata_len;
lapi_cntr_t *tgt_cntr;
lapi_cntr_t *org_cntr;
lapi_cntr_t *cmpl_cntr;include 'lapif.h'
INTEGER SUBROUTINE COMPL_H (hndl, user_info)
INTEGER hndl
INTEGER user_info
INTEGER FUNCTION HDR_HDL (hndl, uhdr, uhdr_len, msg_len, comp_h, user_info)
INTEGER hndl
INTEGER uhdr
INTEGER uhdr_len
INTEGER (KIND=LAPI_LONG_TYPE) :: msg_len
EXTERNAL INTEGER FUNCTION comp_h
TYPE (LAPI_ADDR_T) :: user_info
LAPI_AMSEND(hndl, tgt, hdr_hdl, uhdr, uhdr_len, udata, udata_len,
tgt_cntr, org_cntr, cmpl_cntr, ierror)
INTEGER hndl
INTEGER tgt
EXTERNAL INTEGER FUNCTION hdr_hdl
INTEGER uhdr
INTEGER uhdr_len
TYPE (LAPI_ADDR_T) :: udata
INTEGER (KIND=LAPI_LONG_TYPE) :: udata_len
INTEGER (KIND=LAPI_ADDR_TYPE) :: tgt_cntr
TYPE (LAPI_CNTR_T) :: org_cntr
TYPE (LAPI_CNTR_T) :: cmpl_cntr
INTEGER ierrorType of call: point-to-point communication (non-blocking)
Use this subroutine to transfer data to a target task, where it is desirable to run a handler on the target task before message delivery begins or after delivery completes. LAPI_Amsend allows the user to provide a header handler and optional completion handler. The header handler is used to specify the target buffer address for writing the data, eliminating the need to know the address on the origin task when the subroutine is called.
User data (uhdr and udata) are sent to the target task. Once these buffers are no longer needed on the origin task, the origin counter is incremented, which indicates the availability of origin buffers for modification. Using the LAPI_Xfer call with the LAPI_AM_XFER type provides the same type of transfer, with the option of using a send completion handler instead of the origin counter to specify buffer availability.
Upon arrival of the first data packet at the target, the user's header handler is invoked. Note that a header handler must be supplied by the user because it returns the base address of the buffer in which LAPI will write the data sent from the origin task (udata). See RSCT for AIX 5L™: LAPI Programming Guide for an optimization exception to this requirement that a buffer address be supplied to LAPI for single-packet messages.
The header handler also provides additional information to LAPI about the message delivery, such as the completion handler. LAPI_Amsend and similar calls (such as LAPI_Amsendv and corresponding LAPI_Xfer transfers) also allow the user to specify their own message header information, which is available to the header handler. The user may also specify a completion handler parameter from within the header handler. LAPI will pass the information to the completion handler at execution.
Note that the header handler is run inline by the thread running the LAPI dispatcher. For this reason, the header handler must be non-blocking because no other progress on messages will be made until it returns. It is also suggested that execution of the header handler be simple and quick. The completion handler, on the other hand, is normally enqueued for execution by a separate thread. It is possible to request that the completion handler be run inline. See RSCT for AIX 5L: LAPI Programming Guide for more information on inline completion handlers.
If a completion handler was not specified (that is, set to LAPI_ADDR_NULL in FORTRAN or its pointer set to NULL in C), the arrival of the final packet causes LAPI to increment the target counter on the remote task and send an internal message back to the origin task. The message causes the completion counter (if it is not NULL in C or LAPI_ADDR_NULL in FORTRAN) to increment on the origin task.
If a completion handler was specified, the above steps take place after the completion handler returns. To guarantee that the completion handler has executed on the target, you must wait on the completion counter. See RSCT for AIX 5L: LAPI Programming Guide for a time-sequence diagram of events in a LAPI_Amsend call.
User details
void *hdr_hndlr(lapi_handle_t *hndl, void *uhdr, uint *uhdr_len, ulong *msg_len,
compl_hndlr_t **cmpl_hndlr, void **user_info);Use of LAPI_Addr_set
Remote addresses are commonly exchanged by issuing a collective LAPI_Address_init call within a few steps of initializing LAPI. LAPI also provides the LAPI_Addr_set mechanism, whereby users can register one or more header handler addresses in a table, associating an index value with each address. This index can then be passed to LAPI_Amsend instead of an actual address. On the target side, LAPI will use the index to get the header handler address. Note that, if all tasks use the same index for their header handler, the initial collective communication can be avoided. Each task simply registers its own header handler address using the well-known index. Then, on any LAPI_Amsend calls, the reserved index can be passed to the header handler address parameter.
Role of the header handler
The user optionally returns the address of a completion handler function through the cmpl_hndlr parameter and a completion handler parameter through the user_info parameter. The address passed through the user_info parameter can refer to memory containing a datatype defined by the user and then cast to the appropriate type from within the completion handler if desired.
typedef void (compl_hndlr_t)(lapi_handle_t *hndl, void *completion_param);As mentioned above, the value returned by the header handler must be an address for writing the user data sent from the origin task. There is one exception to this rule. In the case of a single-packet message, LAPI passes the address of the packet in the receive FIFO, allowing the entire message to be consumed within the header handler. In this case, the header handler should return NULL (in C) or LAPI_ADDR_NULL (in FORTRAN) so that LAPI does not copy the message to a target buffer. See RSCT for AIX 5L: LAPI Programming Guide for more information (including a sample header handler that uses this method for fast retrieval of a single-packet message).
Passing additional information through lapi_return_info_t
LAPI allows additional information to be passed to and returned from the header handler by passing a pointer to lapi_return_info_t through the msg_len argument. On return from a header handler that is invoked by a call to LAPI_Amsend, the ret_flags member of lapi_return_info_t can contain one of these values: LAPI_NORMAL (the default), LAPI_SEND_REPLY (to run the completion handler inline), or LAPI_LOCAL_STATE (no reply is sent). The dgsp_handle member of lapi_return_info_t should not be used in conjunction with LAPI_Amsend.
For a complete description of the lapi_return_info_t type, see RSCT for AIX 5L: LAPI Programming Guide
Inline execution of completion handlers
Under normal operation, LAPI uses a separate thread for executing user completion handlers. After the final packet arrives, completion handler pointers are placed in a queue to be handled by this thread. For performance reasons, the user may request that a given completion handler be run inline instead of being placed on this queue behind other completion handlers. This mechanism gives users a greater degree of control in prioritizing completion handler execution for performance-critical messages.
LAPI places no restrictions on completion handlers that are run "normally" (that is, by the completion handler thread). Inline completion handlers should be short and should not block, because no progress can be made while the main thread is executing the handler. The user must use caution with inline completion handlers so that LAPI's internal queues do not fill up while waiting for the handler to complete. I/O operations must not be performed with an inline completion handler.
/* header handler routine to execute on target task */
void *hdr_hndlr(lapi_handle_t *hndl, void *uhdr, uint *uhdr_len,
ulong *msg_len, compl_hndlr_t **cmpl_hndlr,
void **user_info)
{
/* set completion handler pointer and other information */
/* return base address for LAPI to begin its data copy */
}
{
lapi_handle_t hndl; /* the LAPI handle */
int task_id; /* the LAPI task ID */
int num_tasks; /* the total number of tasks */
void *hdr_hndlr_list[NUM_TASKS]; /* the table of remote header handlers */
int buddy; /* the communication partner */
lapi_cntr_t cmpl_cntr; /* the completion counter */
int data_buffer[DATA_LEN]; /* the data to transfer */
.
.
.
/* retrieve header handler addresses */
LAPI_Address_init(hndl, (void *)&hdr_hndlr, hdr_hndlr_list);
/*
** up to this point, all instructions have executed on all
** tasks. we now begin differentiating tasks.
*/
if ( sender ) { /* origin task */
/* initialize data buffer, cmpl_cntr, etc. */
.
.
.
/* synchronize before starting data transfer */
LAPI_Gfence(hndl);
LAPI_Amsend(hndl, buddy, (void *)hdr_hndlr_list[buddy], NULL,
0,&(data_buffer[0]),DATA_LEN*(sizeof(int)),
NULL, NULL, cmpl_cntr);
/* Wait on completion counter before continuing. Completion */
/* counter will update when message completes at target. */
} else { /* receiver */
.
.
.
/* to match the origin's synchronization before data transfer */
LAPI_Gfence(hndl);
}
.
.
.
}For a complete program listing, see RSCT for AIX 5L: LAPI Programming Guide. Sample code illustrating the LAPI_Amsend call can be found in the LAPI sample files. See RSCT for AIX 5L: LAPI Programming Guide for more information about the sample programs that are shipped with LAPI.