How time flies? It only seemed like yesterday that I installed CentOS 5.4 at a customer site. This week I needed to update rsync from 2.6.8 to what ever the latest version is. So I typed:

yum update rsync

and got the following message.

Loaded plugins: fastestmirror
rpmdb: unable to join the environment
error: db4 error(11) from dbenv->open: Resource temporarily unavailable
error: cannot open Packages index using db3 - Resource temporarily unavailable (11)
error: cannot open Packages database in /var/lib/rpm
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 309, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 157, in main
    base.getOptionsConfig(args)
  File "/usr/share/yum-cli/cli.py", line 187, in getOptionsConfig
    self.conf
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 664, in <lambda>
    conf = property(fget=lambda self: self._getConfig(),
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 239, in _getConfig
    self._conf = config.readMainConfig(startupconf)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 804, in readMainConfig
    yumvars['releasever'] = _getsysver(startupconf.installroot, startupconf.distroverpkg)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 877, in _getsysver
    idx = ts.dbMatch('provides', distroverpkg)
TypeError: rpmdb open failed

I also tried:

rpm -qa

which resulted in:

rpmdb: unable to join the environment
error: db4 error(11) from dbenv->open: Resource temporarily unavailable
error: cannot open Packages index using db3 - Resource temporarily unavailable (11)
error: cannot open Packages database in /var/lib/rpm

Even though the message says Resource temporarily unavailable, it was in fact always unavailable! Hunted around and found this article which helped a lot.

The problem is caused by an empty database index file in the RPM repository, probably caused by corruption. In my case it was because most of the index files were missing.

$ ls -l /var/lib/rpm
total 39656
-rw-r--r-- 1 root root  5107712 Nov 11 11:58 Basenames
-rw-r--r-- 1 root root    12288 Oct  5  2010 Conflictname
-rw-r--r-- 1 root root        0 Dec 12 11:42 __db.001
-rw-r--r-- 1 root root  1036288 Nov 11 11:58 Dirnames
-rw-r--r-- 1 root root  5259264 Nov 11 11:58 Filemd5s
-rw-r--r-- 1 root root    24576 Nov 11 11:58 Group
-rw-r--r-- 1 root root    16384 Nov 11 11:58 Installtid
-rw-r--r-- 1 root root    40960 Nov 11 11:58 Name
-rw-r--r-- 1 root root 30818304 Nov 11 11:58 Packages
-rw-r--r-- 1 root root   331776 Nov 11 11:58 Providename
-rw-r--r-- 1 root root   102400 Nov 11 11:58 Provideversion
-rw-r--r-- 1 root root    12288 Oct  5  2010 Pubkeys
-rw-r--r-- 1 root root   413696 Nov 11 11:58 Requirename
-rw-r--r-- 1 root root   208896 Nov 11 11:58 Requireversion
-rw-r--r-- 1 root root    81920 Nov 11 11:58 Sha1header
-rw-r--r-- 1 root root    45056 Nov 11 11:58 Sigmd5
-rw-r--r-- 1 root root    12288 Feb 16  2010 Triggername

So all I needed to do was remove all the index files and rebuild them.

rm -rf /var/lib/rpm/__db*
rpm -rebuilddb

The RPM repository now looks like:

-rw-r--r-- 1 root root  2768896 Jan  6 09:49 Basenames
-rw-r--r-- 1 root root    12288 Oct  5  2010 Conflictname
-rw-r--r-- 1 root root        0 Jan  6 09:49 __db.000
-rw-r--r-- 1 root root    24576 Jan  6 09:49 __db.001
-rw-r--r-- 1 root root  1318912 Jan  6 09:49 __db.002
-rw-r--r-- 1 root root   450560 Jan  6 09:49 __db.003
-rw-r--r-- 1 root root   978944 Jan  6 09:49 Dirnames
-rw-r--r-- 1 root root  5267456 Jan  6 09:49 Filemd5s
-rw-r--r-- 1 root root    24576 Jan  6 09:49 Group
-rw-r--r-- 1 root root    16384 Jan  6 09:49 Installtid
-rw-r--r-- 1 root root    45056 Jan  6 09:49 Name
-rw-r--r-- 1 root root 25427968 Jan  6 09:49 Packages
-rw-r--r-- 1 root root   335872 Jan  6 09:49 Providename
-rw-r--r-- 1 root root    98304 Jan  6 09:49 Provideversion
-rw-r--r-- 1 root root    12288 Oct  5  2010 Pubkeys
-rw-r--r-- 1 root root   409600 Jan  6 09:49 Requirename
-rw-r--r-- 1 root root   196608 Jan  6 09:49 Requireversion
-rw-r--r-- 1 root root    81920 Jan  6 09:49 Sha1header
-rw-r--r-- 1 root root    45056 Jan  6 09:49 Sigmd5
-rw-r--r-- 1 root root    12288 Feb 16  2010 Triggername

I re-issued the update command:

yum update rsync

and it successfully installed rsync 3.0.6-4.el5_7.1.

Interestingly, CentOS 5.4 has been deprecated and so this version of rsync was for CentOS 5.7, but it didn't need to install any dependencies.