Mr N

If you are a systems UNIX administrator, sooner or later you are going to have a catastrophic disaster which will for ever alter your way of working. UNIX is unforgiving when it comes to deleting files.

The UNIX file system works faster and more efficiently that a windows file system. Notably, the disk never needs de-fragmenting and the disk caching is better. The bad point about it being better is that if you delete anything it is almost impossible to get it back!

There are several common mistakes that you have to watch out for: accidentally deleting something important, accidentally deleting system tools or accidentally writing into a device file which doesn’t point to what you thought it did.

Accidentally deleting something important

Is the file still in use by any other program? If yes then read on, if no then it’s either back to those backup tapes or you could have a go at Carlo Wood’s HOWTO recover deleted files on an ext3 file system which is not for the faint hearted!

One thing worth noting is that when a file is open the space can not be reclaimed by the system. Unix allows many pointers to a file. They may or may not exist in the file system. Under windows you can’t delete a file that another user is writing to, but under Unix you can. …and believe it or not it’s really handy. No more of those famous “can’t delete folder - file in use” messages from Windows.

In fact the best way to create a temporary file that only you can write to, is to open a new file then delete it from the file system. There is no longer a global reference (file system entry) to your file so no one can read it. When you close the file the OS reclaims the space. If your process dies then the file is closed for you by the OS and is therefore automatically tidied away. Cool!

The first method of getting a deleted file back is the easiest and will work in most cases.

1. Find the process id of the program that has your file open. You can use one of the following help programs to help find a process that has your deleted file open (locked):

   lsof

   or

   ls -l /proc/*/fd | grep myfilename

2. Change directory into the /proc filesystem for that process.

   cd /proc/12345/fd

3. In the directory lists there will be a symbolic link that points to your deleted file. eg

   total 4
   lrwx—— 1 root root 64 Aug 11 17:33 0 -> /dev/pts/0
   lrwx—— 1 root root 64 Aug 11 17:33 1 -> /dev/pts/0
   lrwx—— 1 root root 64 Aug 11 17:33 2 -> /dev/pts/0
   lr-x—— 1 root root 64 Aug 11 17:33 3 -> /tmp/hello.txt (deleted)

4. Simply copy the file somewhere else using the copy command:

   cp 3 /tmp/hello.txt.undeleted

I once deleted a load of database files in an Oracle instance. Luckily Oracle was still running, so I was able to re-link the files without any damage. If the file is too big or being written to, it might not be practical to copy it away. The following set of instructions show how to re-link the file back into the file system.

1. Run the program lsof and search the output for the line containing the name of the file you have just deleted.

   # lsof | grep -e hello.txt -e COMMAND
   COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
   cat 1002 root 3r REG 3,3 1500015 3866793 /home/hello.txt (deleted)

2. The number next to the file name is the inode number and represents the location of the first part of the file in the file system structure (3866793).
3. Now that we have the inode of the file we want to restore, we can re-link it to the file system using the e2fsprogs suite.

It is definitely worth noting that if you start tampering with your file systems you are on your own. This blog is advisory and if you accidentally make your problems worse then don’t blame me.

1. Firstly you need the device name of the file system your deleted file is on. Change to the directory you want to restore the file to and use the df command with a dot for the current directory. Note you can only restore files on the same file system. Once it has been restored, you can copy it anywhere you like.

   # df .
   Filesystem 1k-blocks Used Available Use% Mounted on
   /dev/hda3 38535468 9966368 26611580 28% /

2. Next load in the file system debugger in write mode:

   # debugfs -w /dev/hda3
   debugfs 1.26 (3-Feb-2002)
   debugfs:

3. Change to the folder you want to restore the file to:

   debugfs: pwd
   [pwd] INODE: 2 PATH: /
   [root] INODE: 2 PATH: /
   debugfs: cd /home
   debugfs: pwd
   [pwd] INODE: 3866727 PATH: /home
   [root] INODE: 2 PATH: /
   debugfs: ln <3866793> hello.txt

4. Now link the first inode of the delete file back to the file system.

   debugfs: ls -l
   3866727 40755 0 0 4096 12-Aug-2009 10:13 .
   2 40755 0 0 4096 1-Apr-2009 00:04 ..
   debugfs: ln <3866793> hello.txt
   debugfs: ls -l
   3866727 40755 0 0 4096 12-Aug-2009 10:13 .
   2 40755 0 0 4096 1-Apr-2009 00:04 ..
   3866793 100644 0 0 1500015 12-Aug-2009 10:13 hello.txt

   The ‘<’ and ‘>’ tells ln that the source file is actually an inode and not a regular file.
5. Changes are made to the file system straight away, so all you have to do is exit:

   debugfs: q
   #

Accidentally deleting a file
When you are looking after development machines or machines where there is a lot of interactive use then you will often find that there is no process to lock the file when it is accidentally deleted. If this happens, then you need to have a copy of all the inodes in advance. Each night as part of a cron job (scheduled task) I would do a full listing of all the files on the computer using:

ls -Rail / > /ls-Rail.txt

This gives a recursive listing of all files (including hidden files) with their inodes from the root.

If you delete a lot of files then you may want to set the file system to read-only mode, so that the system can’t overwrite any of the deleted blocks. This will give you a bit of time to repair the file system.

mount -n -o remount -o ro /dev/hda1 /home

where

• -n : don’t write /etc/fstab.
• -o remount : lets you change options without unmounting the file system. Often you can’t unmount anyway it because it is in use.
• -o ro : sets the file system to read-only mode.
• /dev/hda1 : device file to be mounted.
• /home : folder to place the file system.

When you have made repairs you can change the file system back to read-write with:

mount -n -o remount /dev/hda1 /home

Catastrophic disaster

I once tried to create a tar file on a USB stick, but accidentally used the root file system device file instead of the USB device file. This overwrote all the superblocks in the file allocation table. Now the only copy was in memory! The filesystem degraded over time as the superblocks were read from disk during the normal running of the system.

Most of /bin was gone, so the plan became “save as many as you can". My other partitions were unaffected because they had separate superblock structures. The file lister ls was gone along with half the other file system tools, so not only was I buggered, I was also blind!

Krishna Kunchith said that the echo command is a shell command as well as an actual command program, so when my echo command program died, the shell echo gave me eyes.

echo *

prints all the files which match the while card, you can also use the wild card system to do file list filtering:

echo h*

cd is also a shell command. So now I had eyes and legs, and for the most part that’s all you need!

Oh dear!

If your here, then you really ought to have a long think about your disaster and recovery backup procedure! Good luck!