Problems with the /etc/utmp File in AIX 4

About this document

Corruption of the utmp file shows up in two ways:

1. The uptime and w commands show a time greater than 8000 days since the system was last booted.
2. Users are shown as still logged in when in fact they are not.

Both types of corruption have many causes because both AIX commands and third party applications write to the utmp file.

If record number 0 is overwritten by anyone (normally a third party program), the uptime shows up as greater than 8000 days.

To correct the invalid boot time you must reboot the system. The utmp file is recreated with each boot.

To attempt to discover who or what overwrote the first entry in the file, use the following command to create a readable version of the utmp file and look at record 0:

NOTE: The fwtmp command must first be installed. For AIX Version 4, install bos.acct.

/usr/sbin/acct/fwtmp < /etc/utmp  >/tmp/out

A valid entry looks something like this:

system boot 0 2 0000 0000 818538505 Sat Dec 9 13:48:25 CST 1995

Instead of the system boot entry, you will probably find an entry like:

jones pts/2 19193 7 0000 0000 818683926 Mon Dec 11 06:12:06 CST 1995

This output means that the time stamp was corrupted by whatever program jones on pts/2 used to login. A program should never overwrite the first two entries in the utmp file. You would have to talk with jones to see what he did. This is almost always caused by a third party program that is incorrectly writing to the utmp file or a corrupted file system where the data is invalid.

When a user logs into the system, the /usr/sbin/getty program writes an entry in /etc/utmp like:

AIX Version 4
sandy pts/23 pts/23 7 42300 0000 0000 818973357
[more data...]
 *                             *
Field #1 = user's name
Field #2 = /etc/inittab id
Field #3 = tty used to login  on
Field #4 = type of entry
Field #5 = PID (process id)

The types of entries can be seen by examining the /usr/include/utmp.h file under ut_type. Type 7 is a USER_PROCESS.

When a user logs out, it is the responsibility of the last process running to update the entry in the utmp file. After a logout, the entry should look like:

AIX Version 4:
pts/23 pts/23 8 42300 0000 0000 818973357
[more data...]
*              *

The user name is erased and the state is changed from 7 to 8 (DEAD_PROCESS).

The who command will only show entries that are in state 7.

1. Set up auditing on writes to the utmp file.
2. Have cron do the who command each minute and send the results to a file.
3. When you notice corruption with the who or w command, check the cron output files to determine when the corruption occurred.
4. Look in the audit log to determine what process was writing to the utmp file at the time the corruption occurred.

   This is an example of an audit log output:

   event      login  status   time                     command
   ---------- ------ -------  ----------------------   -------
   UTMP_WRITE root   OK       Tue Dec 19 17:00:29 1995 telnetd
   

   The example above shows that telnetd wrote to the file at 17:00:29.

For fixes related to utmp corruption, install the latest level of the following filesets:

bos.rte.misc_cmds
bos.rte.tty
devices.tty.rte
bos.net.tcp.server
bos.net.tcp.client

See the next section, "Fixes/Problems", for obtaining fixes.

Fixes for AIX Version 4 can be downloaded via the Internet with the FixDist service.

Rebooting clears the utmp file and is the recommended method of correcting the results of corruption.

The following is an awk script that can be used to attempt to clean out bad entries in the /etc/utmp file. It may not clean certain types of corruption and a reboot will be required to clean up the file.

WARNING: Since the utmp file is constantly being changed, there is always the possibility that an attempt at correction (other than by rebooting) may corrupt the /etc/utmp file.

#!/usr/bin/ksh
# utmp_clean.awk
# 12/12/95
# awk script to clean out entries in the /etc/utmp file
# that have no current matching correct process in the
# process table.
# This MUST be run by the root user, either from the
# command line or
# from the root crontab entry.
# 
if [ ! -s /usr/sbin/acct/fwtmp ]
then
# accounting not installed
   print "Accounting must be installed first,fwtmp file does not exist"
   exit
fi
#
SUM=1
NEWSUM=0
while [ "$SUM" != "$NEWSUM" ]
     do
          SUM=$(/usr/bin/sum /etc/utmp)
          /usr/sbin/acct/fwtmp </etc/utmp >/tmp/utmp.out
          ps au |awk '{print $2,$1,$7}' |grep -v USER >/tmp/ps.out
          NEWSUM=$(/usr/bin/sum /etc/utmp)
          # loop until the file is unchanged
          # on a busy system, this may take a long time.
        done
#
cat /tmp/utmp.out | awk '
# load the array
BEGIN {
counter=0
holder = ""
ss=1
while (ss == 1)
     {
     ss = (getline holder < "/tmp/ps.out")
     if (ss == 0)
          break
     n=split(holder,temp)
     combine=sprintf("%s %s",temp[2],temp[3])
     lookup[temp[1]]=combine
     }
} # end of BEGIN section
{
if ((length($4) == 1) && ($4 == 7))
  {

ps_name=lookup[$5]
if (length(ps_name)  > 0)
   {
    #found a ps table entry with same pid
    # entry needs to be checked for accuracy
    #only if the name and tty match, write the entry
    utmp_name=sprintf("%s %s",$1,$2)
     if (ps_name == utmp_name)
          print $0
    }
  }
else # Not an entry to look at, just pass it along
    {
           print $0
    }
}' > /tmp/utmp.tmp
/usr/sbin/acct/fwtmp -ic </tmp/utmp.tmp  >/tmp/utmp.new
# Only if the /etc/utmp file is still unchanged from when
# we last looked will the file be overwritten with the
# updated copy.
#           WARNING WARNING WARNING
# There is a chance that this step may corrupt the
# /etc/utmp file if a process changes it after we look
# and before we can write it.
CURRENTSUM=$(/usr/bin/sum /etc/utmp)
if [ "$CURRENTSUM" = "$SUM" ]
     then
     /usr/bin/cp /tmp/utmp.new /etc/utmp
     print "utmp successfully updated on "$(date)
     else
     print "utmp was too busy on "$(date)" to update now"
     print "try again later"
fi
rm /tmp/ps.out
rm /tmp/utmp.out
rm /tmp/utmp.tmp
rm /tmp/utmp.new

[ Doc Ref: 90605225614692     Publish Date: Jan. 29, 2001     4FAX Ref: 8243 ]