This document describes how to check for resource bottlenecks using the Performance Toolbox for AIX. Resources on a system include memory, cpu, and Input/Output (I/O). This document covers bottlenecks across an entire system and is applicable to Performance Toolbox versions 2.1 and 2.2. This document does not address how to find the bottlenecks of a particular application. The following commands are described:
The Performance Toolbox package is a very powerful and diverse tool for identifying performance problems and potential performance bottlenecks. The following sections of this document will help you become familiar with what it has to offer and give you an idea of how to proceed with a strategy for monitoring system performance over a period of time. Please note that modifications to this basic strategy may have to be made in order to accommodate certain specific system environments.
Consult Line Performance Analysis - The AIX Support Family offers a system analysis with tuning recommendations. For more information contact your AIX support center.
Performance Toolbox for AIX Guide and Reference (SC23-2625) - This IBM publication covers the use of the Performance Toolbox for AIX licensed product.
Performance Tuning Guide (SC23-2365) - This IBM publication covers performance monitoring and tuning of AIX systems. Order through your local IBM representative.
All data is gathered by the xmservd daemon and forwarded to the program requesting the information.
For a listing of all the possible statistics and metrics which the xmservd daemon gathers, enter the following command (the listing is dependent upon each system's hardware and software configuration):
# xmpeek -l | pg
There are four tools available for monitoring system performance in real-time: chmon, xmperf, 3dmon and ptxrlog.
chmon -p3
xmperf
3dmon -c LAN
ptxrlog -f ptxrlog.cf -i 5
All four of these programs can be started from the command line. The commands 3dmon, chmon, and ptxrlog can be started from within xmperf under the Utilities Menu.
Another useful feature of Performance Toolbox is the ability to have the system monitor and react to specific situations. This is done with the filter daemon (filtd). A useful method is to utilize the scripts provided by installing bos.perf.pmr and having the filter daemon launch these scripts to gather related performance data whenever an alarm condition is detected.
NOTE: IBM's Consult Line services can be invoked to help analyze data gathered from these scripts. In addition, IBM offers a product called Performance Analyzer that utilizes filtd. It provides alarms that detect potential performance problems. See the "Related documentation" section at the end of this fax for more information.
# supplier: /usr/bin/filtd -p5
# Starting xmservd (Performance Toolbox Deamon) sleep 2 /usr/bin/xmpeek
xmquery dgram udp wait root /usr/bin/xmservd xmservd -p3 -l0
There are three types of recording files that can be created with Performance Toolbox: a binary recording file, an ASCII recording file, and a formatted ASCII recording file which can be read into a spreadsheet program.
The xmperf, 3dmon, ptxrlog, and the xmservd daemon itself can all be used to record performance data.
In order to record performance statistics with xmservd, create a xmservd.cf file in /etc/perf on the system you wish to monitor and record data. A sample xmservd.cf file is provided at the end of this document.
Collected data can be played back in xmperf or 3dplay (Version 2.2 only). The various commands beginning with the letters ptx allow for exporting collected data into a spreadsheet program. The azizo utility displays or prints recording files in graphical or tabulated formats. Files containing graphs are in PostScript format.
------------------------------------------------------------
#
# "SAMPLE filter.cf FILE"
#
# output files are created in /var/perf/tmp
# bos.perf.pmr (AIX 4.1 only) must be installed to run
# the scripts in /usr/sbin/perf/pmr. Personal scripts can
# be used in place of those provided with bos.perf.pmr.
#
diskmax = Disk_>_busy "Busy percent - most busy disk"
cpunum = CPU_#_user "Number of CPU's on the system"
Better number for 1 more than then # of processors on the machine
@highcpuload:[/usr/sbin/perf/pmr/tprof_ 10] \
{EXEPTION} \
(CPU_gluser CPU_glkern) > 90 && (Proc_runque > \
(DDS_IBM_Filters_cpunum 1)) \
DURATION 60 FREQUENCY 10 SEVERITY 1 \
"High CPU load"
@thrashing: [/usr/sbin/perf/pmr/vmstat_ 60] \
{EXEPTION} \
(Mem_Virt_pgspgout * 6) > Mem_Virt_steal \
DURATION 5 FREQUENCY 10 SEVERITY 2 \
"System is thrashing"
@lowfree: [/usr/sbin/perf/pmr/vmstat_ 60] \
{EXEPTION} \
Mem_Real_numfrb < 40 \
DURATION 5 FREQUENCY 10 SEVERITY 3 \
"Low free frames in RAM"
@lowpgsp: [/usr/sbin/lsps -a >> /var/perf/tmp/pagingspace
2>&1] \
{EXEPTION} \
PagSp_%totalused > 90 \
DURATION 60 FREQUENCY 10 SEVERITY 4 \
"Low paging space"
@wait: [/usr/sbin/perf/pmr/filemon_ 30] \
{EXEPTION} \
CPU_cpu0_wait > 0 && \
DDS_IBM_Filters_diskmax > 33 \
DURATION 30 FREQUENCY 10 SEVERITY 5 \
"I/O wait"
-------------------------------------------------------------
#
# "SAMPLE xmservd.cf FILE"
#
# Keep files at least 14 days and let each file contain
# one day's recordings
#
retain 14 1
#
# Set default sampling interval to 1 second
#
frequency 1000
#
# Statistics to record with default frequency
#
Proc/runque
Proc/swpque
CPU/gluser
CPU/glkern
CPU/glwait
Mem/Real/numfrb
Mem/Virt/pgspgin
Mem/Virt/pgspgout
#
# Statistics recorded every 5 minutes
#
Mem/Real/%comp 300000
Mem/Real/%noncomp 300000
PagSp/%totalused 300000
#
# Record every weekday from 7 am to 7pm
#
start 1-5 7 0 1-5 19 0
-------------------------------------------------------------
[ Doc Ref: 90605219014820 Publish Date: Oct. 19, 2000 4FAX Ref: 6835 ]