vmstat Command

Purpose

Reports virtual memory statistics.

Syntax

vmstat [ -f ] [ -i ] [ -s ] [ -I[-W]] [ -t ] [ -v ] [ -h ] [ -w ] [ -l ] [-c] [ -@ wparname | ALL ] [ { -p | -P } pagesize | ALL ] ALL ] [ -S power] [ physicalvolume ... ] [ interval [ count ] ]

Note: Do not use the wparname parameter and the -i flag together inside workload partitions.

Description

The vmstat command reports statistics about kernel threads, virtual memory, disks, hypervisor pages, traps and processor activity. Reports generated by the vmstat command can be used to balance system load activity. These system-wide statistics (among all processors) are calculated as averages for values expressed as percentages, and as sums otherwise. The vmstat command might return inconsistent statistics because the statistics are not read atomically.

If you run the vmstat command without flags, the report contains a summary of the virtual memory activity since system startup. If you specify the -f flag, the vmstat command reports the number of forks since system startup. The physicalvolume parameter specifies the name of the physical volume.

The interval parameter specifies the amount of time in seconds between each report. If you do not specify the interval parameter, the vmstat command generates a single report that contains statistics for the time since system startup and then exits. You can specify the count parameter only with the interval parameter. If you specify the count parameter, its value determines the number of reports generated and the number of seconds apart. If you specify the interval parameter without the count parameter, reports are continuously generated. Do not specify a value of zero to the count parameter.

AIX® 4.3.3 and later contain enhancements to the method used to compute the percentage of processor time spent waiting on disk I/O (wio time). The method used in AIX 4.3.2 and earlier versions of the operating system can, under certain circumstances, give an inflated view of wio time on SMPs.

The method used in AIX 4.3.2 and earlier versions is as follows: At each clock interrupt on each processor (100 times a second per processor), a determination is made as to which of the four categories (usr/sys/wio/idle) to place the last 10 ms of time. If the processor was busy in usr mode at the time of the clock interrupt, then usr gets the clock tick added into its category. If the processor was busy in kernel mode at the time of the clock interrupt, then the sys category gets the tick. If the processor was not busy, a check is made to see if any I/O to disk is in progress. If any disk I/O is in progress, the wio category is incremented. If no disk I/O is in progress and the processor is not busy, the idle category gets the tick. The inflated view of wio time results from all idle processors being categorized as wio regardless of the number of threads waiting on I/O. For example, systems with just one thread doing I/O could report over 90 percent wio time regardless of the number of processors it has. The wio time is reported by the commands sar (%wio), vmstat (wa) and iostat (% iowait).

The kernel maintains statistics for kernel threads, paging, and interrupt activity, which the vmstat command accesses through the use of the perfstat kernel extension. The disk input/output statistics are maintained by device drivers. For disks, the average transfer rate is determined by using the active time and number of transfers information. The percent active time is computed from the amount of time the drive is busy during the report.

Beginning with AIX 5.3, the vmstat command reports the number of physical processors consumed (pc), and the percentage of entitlement consumed (ec), in Micro-Partitioning® environments. These metrics will only be displayed on Micro-Partitioning environments.

The report generated by the vmstat command contains a system configuration row and column headings. If the -@ flag is specified, the report will consist of system configuration and WPAR configuration. The system configuration row has the following values:

Item Description
lcpu Indicates the number of logical processors.
mem Indicates the amount of memory.
tmem Indicates the true memory size of the LPAR.
Note: This is available only when -c option is provided and Active Memory™ Expansion is enabled.
ent Displays only when the partition is running with shared processor.
drives Displays only when physical volume name is monitored.
WPARs Indicates the number of active workload partitions. It is displayed only when the -@ flag is specified.
memlim Indicates the limit of the memory resource of the workload partition. The limit is in megabytes (MB). This information is displayed only for the WPAR that has enforced memory resource limit.
cpulim Indicates the limit of processor resource of the workload partition in processor units. This information is displayed only for the WPAR that has enforced processor resource limit.
rset Indicates the type of the rset registry that is associated with a WPAR. The type can be regular or exclusive. This information is displayed only for the WPARs that are associated with an rset registry.
mmode Indicates memory mode. This metric is displayed automatically in a system with Active Memory Sharing enabled. This metric is also displayed when -c option is used.
mpsz Size of the memory pool in gigabytes. This metric is displayed only in shared-memory mode.

The column headings and their descriptions follow:

WPAR: Information about workload partitions. It displays only when the -@ flag is specified.

Item Description
WPAR Workload partition name.
Notes:
  1. The system WPAR name indicates system-wide statistics. The global WPAR name indicates the statistics belong to Global only.
  2. When the vmstat command is invoked with the -@ ALL option and the WPAR specific information is not available for a metric, then an en dash sign (-) is displayed instead of a value.
  3. When the vmstat command is invoked with the -@ wparname or invoked inside a WPAR, if the WPAR information is not available for a metric, then that metric is marked with the at sign (@), and the system-wide value is displayed for that metric.
  4. If a metric is not supported, then a en dash sign (-) is displayed instead of a value.

kthr: Information about kernel thread states.

Item Description
r Average number of runnable kernel threads over the sampling interval. Runnable threads consist of the threads that are ready but still waiting to run, and the threads that are already running.
b Average number of kernel threads placed in the Virtual Memory Manager (VMM) wait queue (awaiting resource, awaiting input/output) over the sampling interval.

Memory: Information about the usage of virtual and real memory. Virtual pages are considered active if they have been accessed. A page is 4096 bytes.

Item Description
avm Active virtual pages.
fre Size of the free list.
Note: A large portion of real memory is utilized as a cache for file system data. It is not unusual for the size of the free list to remain small.

Page: Information about page faults and paging activity. These are averaged over the interval and given in units per second.

Item Description
re Pager input/output list.
pi Pages paged in from paging space.
po Pages paged out to paging space.
fr Pages freed (page replacement).
sr Pages scanned by page-replacement algorithm.
cy Clock cycles by page-replacement algorithm.

Faults: Trap and interrupt rate averages per second over the sampling interval.

Item Description
in Device interrupts.
sy System calls.
cs Kernel thread context switches.

CPU: Breakdown of percentage usage of processor time.

Item Description
us User time.
sy System time.
id Processor idle time.
wa Processor idle time during which the system had outstanding disk/NFS I/O request(s). See detailed description above.
pc Number of physical processors consumed. Displayed only if the partition is running with shared processor.
ec The percentage of entitled capacity consumed. Displayed only if the partition is running with shared processor. Because the time base over which this data is computed can vary, the entitled capacity percentage can sometimes exceed 100%. This excess is noticeable only with small sampling intervals.
rc The percentage of processor resource that is consumed. This information is displayed only for the WPARs that have enforced processor resource limit.

Disk: Provides the number of transfers per second to the specified physical volumes that occurred in the sample interval. The physicalvolume parameter can be used to specify one to four names. Transfer statistics are given for each specified drive in the order specified. This count represents requests to the physical device. It does not imply an amount of data that was read or written. Several logical requests can be combined into one physical request. If the physicalvolume parameter is used, the physical volume names are printed at the beginning of command execution.

If the -I flag is specified, an I/O oriented view is presented with the following column changes.

Item Description
kthr The p column is displayed in addition to the r and b columns.
p
Number of threads waiting for I/O messages to raw devices. Raw devices are those devices that are directly attached to the system.

If the -W flag is specified along with the -I flag, an additional w column is also displayed along with the r, b, and p flags.

w
Number of threads that are waiting for the file system direct I/O and concurrent I/O (CIO).
page New fi and fo columns will be displayed instead of the re and cy columns.
fi
File page-ins per second.
fo
File page-outs per second.

If the -c flag is specified, Active Memory Expansion view is presented with the following column changes.

Item Description
memory The columns csz, cfr, and dxm will be displayed besides columns avm and fre.
csz
Current compressed pool size, in 4K page units.
cfr
Free pages available in compressed pool, in 4K page units.
dxm
Deficit in Expanded Memory Size, in 4K page units.
page New columns ci and co will be displayed instead of re and cy columns.
ci
Number of page-ins per second from compressed pool.
co
Number of page-outs per second to compressed pool.

If, while the vmstat command is running, there is a change in system configuration that will affect the output, vmstat prints a warning message about the configuration change. It then continues the output, after printing the updated system configuration information and the header.

If the -l flag is specified, an additional "large-page" section is displayed with the following columns:
alp
Indicates the number of large pages currently in use.
flp
Indicates the number of large pages on the large page freelist.
If the -p option is specified, additional lines of VMM statistics are displayed for the specified page sizes. With -I and -t options, the -p option produces an additional line for the specified page size. This line contains the following VMM statistics relevant to the specified page size:
  • avm
  • fre
  • re
  • fi
  • fo
  • pi
  • po
  • ci
  • co
  • fr
  • sr
  • cy
Notes:
  1. The display of the re, fi, fo, and cy options are affected by the -I option.
  2. The display of the re, ci, co, and cy options are affected by the -c option.
  3. If there is no resource control, then the avm and fre options are system-wide. Therefore with the -@ option set, both the avm and fre options will be marked with the at sign (@).
These VMM statistics are preceded by a psz column and followed by an siz column. The description of these two columns follows:
psz
Page size (for example, 4 KB, 64 KB).
siz
Number of frames of the specified page size that exist on the system.
With the -s option, the -p option produces a separate stanza of output that contains only the statistics relevant to the specified page size. This additional stanza is preceded by a page size header.
The -P option produces the following report for the specified page size:
pgsz
Indicates the page size (for example, 4 KB, 64 KB).
Memory
Indicates the memory statistics for the specified page sizes.
siz
The number of frames of the specified page size that exist on the system.
avm
Active virtual pages applicable to the specified page size.
fre
Size of the free list for the specified page size.
Page
Indicates the relevant page faults and paging activity for the specified page size. The page related columns re, pi, po, fr, sr, cy, fi, fo, ci, and co are also applicable to this report.

Flags

Note: If the -f (or -s) flag is entered on the command line, then the system will only accept the -f (or -s) flag and will ignore other flags. If both the -f and -s flags are specified, the system will accept only the first flag and ignore the second flag.
Item Description
-@ wparname Reports the Virtual Memory activity of a workload partition:
  • The -@ ALL option indicates that the report pertains to the system and global environment, in addition to all of the workload partitions in the system.
    Note: The values that are system-wide statistics are marked with dash sign (-) against the WPAR section.
  • The -@ wparname flag indicates that the activity is only for that workload partition. In a workload partition, if you specify the -@ flag, system-wide statistics and workload partition statistics are displayed. The system-wide statistics are marked with the at sign (@).
    Note: Do not use the -@ flag with any combination of the -i flag.
-c Displays memory compression statistics with the new columns of output, csz, cfr, and dxm under the heading memory, and columns ci and co under the heading page instead of the columns re and cy.
Note: This option is available only when Active Memory Expansion is enabled.
-f Reports the number of forks since system startup.
-i Displays the number of interrupts taken by each device since system startup.
Note: The -I, -t, -w, and -l flags are ignored when they are specified with the -i flag.
-I Displays I/O oriented view with the new columns of output, p under heading kthr,and columns fi and fo under heading page instead of the columns re and cy in the page heading.
-l Displays an additional "large-page" section with the alp and flp columns.
-p pagesize Appends the VMM statistics for the specified page size to the regular vmstat output.
-P pagesize Displays only the VMM statistics which are relevant for the specified page size.
-s Writes to standard output the contents of the sum structure, which contains an absolute count of paging events since system initialization. The -s flag can only be used with the -v flag. These events are described as follows:
address translation faults
Incremented for each occurrence of an address translation page fault. I/O may or may not be required to resolve the page fault. Storage protection page faults (lock misses) are not included in this count.
-s
backtracks
Incremented for each page fault that occurs while resolving a previous page fault. (The new page fault must be resolved first and then initial page faults can be backtracked.)
CPU context switches
Incremented for each processor context switch (dispatch of a new process).
decrementer interrupts
Incremented on each decrementer interrupt.
device interrupts
Incremented on each hardware interrupt.
executable-filled page faults
Incremented for each instruction page fault.
extend XPT waits
Incremented each time a process is waited by VMM due to a commit in progress for the segment being accessed.
free frame waits
Incremented each time a process requests a page frame, the free list is empty, and the process is forced to wait while the free list is replenished.
iodones
Incremented at the completion of each VMM I/O request.
mpc send interrupts
Incremented on each mpc send interrupt.
mpc receive interrupts
Incremented on each mpc receive interrupt.
page ins
Incremented for each page read in by the virtual memory manager. The count is incremented for page ins from page space and file space. Along with the page out statistic, this represents the total amount of real I/O initiated by the virtual memory manager.
page outs
Incremented for each page written out by the virtual memory manager. The count is incremented for page outs to page space and for page outs to file space. Along with the page in statistic, this represents the total amount of real I/O initiated by the virtual memory manager.
paging space page ins
Incremented for VMM initiated page ins from paging space only.
paging space page outs
Incremented for VMM initiated page outs to paging space only.
pages examined by the clock
VMM uses a clock-algorithm to implement a pseudo least recently used (lru) page replacement scheme. Pages are aged by being examined by the clock. This count is incremented for each page examined by the clock.
pages freed by the clock
Incremented for each page the clock algorithm selects to free from real memory.
pending I/O waits
Incremented each time a process is waited by VMM for a page-in I/O to complete.
-s
phantom interrupts
Incremented on each phantom interrupt
revolutions of the clock hand
Incremented for each VMM clock revolution (that is, after each complete scan of memory).
start I/Os
Incremented for each read or write I/O request initiated by VMM.
syscalls
Incremented for each system call.
total reclaims
Incremented when an address translation fault can be satisfied without initiating a new I/O request. This can occur if the page has been previously requested by VMM, but the I/O has not yet completed; or if the page was pre-fetched by VMM's read-ahead algorithm, but was hidden from the faulting segment; or if the page has been put on the free list and has not yet been reused.
traps
Not maintained by the operating system.
zero-filled page faults
Incremented if the page fault is to working storage and can be satisfied by assigning a frame and zero-filling it.
When the -c flag is specified along with the -s flag, the following additional metrics are displayed.
compressed pool page ins
Number of page-ins from Compressed Pool since system boot.
compressed pool page outs
Number of page-outs to Compressed Pool since system boot.
-s

When used with the -p pagesize option, the -s option appends the sum structure for the specified page size to the system-wide sum structure. This additional stanza is preceded by a page size header (for example, 4K pages). The following details are not be displayed in this pagesize-based stanza as these statistics are not related to page sizes:

  • Processor context switches
  • Device interrupts
  • Software interrupts
  • Decrementer interrupts
  • MPC-sent interrupts
  • MPC-received interrupts
  • Phantom interrupts
  • Traps
  • Syscalls
Notes:
  1. When the -s flag is used with the -@ ALL option, the system-wide statistics are repeated in the workload partition section.
  2. When the -s flag is used with the wparname option, all metrics are reported and the system-wide statistics are marked with the at sign (@).
  3. When the -s flag is used with the -l flag, the vmstat command displays the following metric:
    large-page hi water count
    Specifies the maximum value of the large-page inuse count.
-S power Multiplies the statistics of the processor with a value of 10power. The default value of the power is 0.
The following statistics are scaled:
  • us
  • sy
  • id
  • wa
  • pc
  • ec
Notes:
  1. Do not use the -S flag with the -f, -s, -i, -v, or -p flags.
  2. When the -S flag is specified, the us, sy, id, and wa statistics change. By default, the us, sy, id, and wa statistics are relative to the processor consumption of WPAR. When the -S flag is specified with a value of power that is not equal to zero, these statistics will be relative to system-wide processor consumption.
  3. The value of power for -S flag can be only between 0 and 3.
-t Prints the time-stamp next to each line of output of vmstat. The time-stamp is displayed in the HH:MM:SS format.
Note: Time stamp will not be printed if -f, -s, or -i flags are specified.
-v Writes to standard output various statistics maintained by the Virtual Memory Manager. The -v flag can only be used with the -s and the -h flags.

If you specify the -v flag, the following statistics are displayed:

compressed percentage
Percentage of memory used by compressed pages.
client filesystem I/Os blocked with no fsbuf
Number of client filesystem I/O requests blocked because no fsbuf was available. NFS (Network File System) and VxFS (Veritas) are client filesystems. Fsbuf are pinned memory buffers used to hold I/O requests in the filesystem layer.
client pages
Number of client pages.
compressed pages
Number of compressed memory pages.
external pager filesystem I/Os blocked with no fsbuf
Number of external pager client filesystem I/O requests blocked because no fsbuf was available. JFS2 is an external pager client filesystem. Fsbuf are pinned memory buffers used to hold I/O requests in the filesystem layer.
file pages
Number of 4 KB pages currently used by the file cache.
free pages
Number of free 4 KB pages.
filesystem I/Os blocked with no fsbuf
Number of filesystem I/O requests blocked because no fsbuf was available. Fsbuf are pinned memory buffers used to hold I/O requests in the filesystem layer.
lruable pages
Number of 4 KB pages considered for replacement. This number excludes the pages used for VMM internal pages, and the pages used for the pinned part of the kernel text.
maxclient percentage
Tuning parameter (managed using vmo) specifying the maximum percentage of memory which can be used for client pages.
maxperm percentage
Tuning parameter (managed using vmo) in percentage of real memory. This specifies the point above which the page stealing algorithm steals only file pages.
maxpin percentage
Tuning parameter (managed using vmo) specifying the percentage of real memory which can be pinned.
memory pages
Size of real memory in number of 4 KB pages.
memory pools
Tuning parameter (managed using vmo) specifying the number of memory pools.
minperm percentage
Tuning parameter (managed using vmo) in percentage of real memory. This specifies the point below which file pages are protected from the re-page algorithm.
numclient percentage
Percentage of memory occupied by client pages.
numperm percentage
Percentage of memory currently used by the file cache.
-v (Statistics displayed by -v, continued):
paging space I/Os blocked with no psbuf
Number of paging space I/O requests that are blocked because the psbuf space is not available. The psbufs space is pinned memory buffers that are used to hold I/O requests at the virtual memory manager layer.
pending disk I/Os blocked with no pbuf for rootvg
Number of pending disk I/O requests that are blocked because the pbuf buffer is not available. The buffers are pinned memory buffers that are used to hold I/O requests at the logical volume manager layer.
pinned pages
Number of pinned 4 KB pages.
remote pageouts scheduled
Number of pageouts scheduled for client file systems.

If you specify the -h flag with the -v flag, the following additional metrics are displayed:

Time resolving virtualized partition memory page faults
The total time that the virtual partition is blocked to wait for the resolution of its memory page fault. The time is measured in seconds, with millisecond granularity.
Virtualized partition memory page faults
The total number of virtual partition memory page faults that are recorded for the virtualized partition.
Number of 4 KB page frames loaned
The number of the 4 KB pages of partition's memory loaned to the hypervisor.
Percentage of partition memory loaned
The percentage of the partition's memory loaned to the hypervisor.
Notes:
  1. When the -v flag is used with the -@ ALL option, the system-wide statistics are not repeated in the workload partition section.
  2. When the -s flag is used with the wparname option, all metrics are reported and the system-wide statistics are marked with the at sign (@).
When the -c flag is specified along with the -v flag, the following additional metrics are displayed:
Compressed Pool Size
Size of Compressed Pool, in 4K page unit.
percentage of true memory used for compressed pool
Percentage of unexpanded memory used for compressed pool.
free pages in compressed pool (4K pages)
Number of free pages in compressed pool, in 4K page unit.
target memory expansion factor
Target memory expansion factor configured for the LPAR.
achieved memory expansion factor
Current memory expansion factor achieved.
-h Displays the hypv-page section that includes the hypervisor page information. The hypv-page section contains the following metrics:
hpi
Number of hypervisor page-in per second.
hpit
Average time spent in milliseconds per hypervisor page-in.
pmem
Amount of physical memory that is backing the logical memory of partitions. The value is measured in gigabytes.

If you specify the -h flag with the -v flag, the following metrics are displayed in addition to the metrics that are displayed using the -v flag:

Time resolving virtualized partition memory page faults
The total time that the virtual partition is blocked to wait for the resolution of its memory page fault. The time is measured in seconds, with millisecond granularity.
Virtualized partition memory page faults
The total number of virtual partition memory page faults that are recorded for the virtualized partition.
Number of 4 KB page frames loaned
The number of the 4 KB pages of the memory that is loaned to the hypervisor in the partition.
Percentage of partition memory loaned
The percentage of the memory loaned to the hypervisor in the partition.
-w Displays the report in wide mode.
-W Displays an additional field w in the kthr section. This option is allowed only with -I flag.
Notes:
  1. If Active Memory Expansion is enabled, the vmstat reports memory statistics in the expanded view. However, if the environment variable AME_MEMVIEW is set to TRUE, the memory statistics will represent the true view.
  2. The AME_MEMVIEW environment variable has no impact on memory statistics reported using the -c option.

Examples

  1. To display a summary of the statistics since boot, enter the following command: 
    vmstat
  2. To display five summaries at 2-second intervals, enter the following command: 
    vmstat 2 5
  3. To display a summary of the statistics since boot including statistics for logical disks scdisk13 and scdisk14, enter the following command: 
    vmstat scdisk13 scdisk14
  4. To display fork statistics, enter the following command: 
    vmstat -f
  5. To display the count of various events, enter the following command: 
    vmstat -s
  6. To display time-stamp next to each column of output of vmstat, enter the following command: 
    vmstat -t
  7. To display the I/O oriented view with an alternative set of columns, enter the following command: 
    vmstat -I
  8. To display all the VMM statistics available, enter the following command: 
    vmstat -vs
  9. To display the large-page section with the alp and flp columns at 8-second intervals, enter the following command: 
    vmstat -l 8
  10. To display the VMM statistics specific to a particular page size (in the example, 4 KB), enter the following command: 
    vmstat -p 4K
  11. To display the VMM statistics for all page sizes that are supported on the system, enter the following command: 
    vmstat -p ALL
    Or enter the following command: 
    vmstat -p all
  12. To display only the VMM statistics for a particular page size (in this example, 4 KB), enter the following command: 
    vmstat -P 4K
  13. To display only the per-page breakdown of VMM statistics for all supported page sizes, enter the following command: 
    vmstat -P ALL
    Or enter the following command: 
    vmstat -P all
  14. To display a summary of the statistics for all of the workload partitions after boot, enter the following command: 
    vmstat -@ ALL
  15. To display all of the VMM statistics available for all of the workload partitions, enter the following command: 
    vmstat –vs -@ ALL
  16. To display both WPAR and system-wide VMM statistics from a workload partition, enter the following command: 
    vmstat -@
  17. To multiply the processor values with 10 and display the results, enter the following command: 
    vmstat -S 1
  18. To display the statistics for the hypervisor page, enter the following command: 
    vmstat –h
  19. To display the information about pages that are loaned to the hypervisor, enter the following command: 
    vmstat -vh
  20. To display memory compression statistics (in an LPAR with Active Memory Expansion enabled), enter the following command: 
    vmstat -c
  21. To display memory compression statistics specific to per-pagesize (in an LPAR with Active Memory Expansion enabled), enter the following command: 
    vmstat –c –P ALL
  22. To append memory compression information to the statistics displayed by -soption (in an LPAR with Active Memory Expansion enabled), enter the following command: 
    vmstat –s -c
  23. To append memory compression information to the statistics displayed by -voption (in an LPAR with Active Memory Expansion enabled), enter the following command: 
    vmstat –v -c

Files

Item Description
/usr/bin/vmstat Contains the vmstat command.