Data Archiving

  1. Archiving Scheme
  2. Tape and Log File Naming Scheme
  3. Standard Tape Commands
  4. Doing/Checking the Backup
  5. Backup Scripts
  6. Troubleshooting
  7. DVD Nightly Backup
  8. Archiving Data for Transport to your Home Institution
  9. Recovering Raw Data
  10. Revision History
Back to BolocamWebPage
Back to ExpertManual


Bolocam raw data tape database: tape names and where they are: xls, pdf.

Archiving Scheme

Please refer to the Data Handling page for an explanation of the different data types.

On a daily basis, we archive the raw data to tape; this is part of the Daily Observing Tasks and the detailed instructions are provided below.  Two tape copies are made.  One copy will remain at the summit and the second copy will go to Hilo for safekeeping.

The tape archive is not intended to be the method for transporting data to your home institution.  Tape systems are expensive and there are many different standards, so it's obviously not economical for each observer to buy a drive of the same kind as we use!  Instead, you will use DVD, external hard drives, your laptop, or the network.

In principle, you only need to transport the raw data.  However, they typical observer will not want to set up to run merge at his home institution.  If you have had no problems with processing the data during your run, you can just archive the merged or the sliced data and carry that back with you.  There is no point in archiving the later stages in the analysis as you will likely reprocess the sliced data and regenerate those files at your home institution.

Note that, once your run is over, the disk space you have been using is fair game.  We will not delete data immediately, but don't expect your data to remain on the summit computers for more than a week or two.  You should try to transport it off or archive it on a daily basis as it is being taken.  We have enough disk space to hold up to about a month of data, but we don't want to get into the business of having to contact a long history of past observers before deleting data.  If there are special circumstances, let the Bolocam support person know explicitly and you will be contacted before your data is deleted.

One particular case for which we are happy to keep data on disk is if you had data stream problems that require manual repair or remerging.  It will be much easier to remerge this data on kilauea if it is still on disk, so inform the Bolocam support person of the problem days and he will make sure those data remain on disk until they are remerged.

Tape and Log File Naming Scheme

Tapes are usually named obsrun_YYYYMM_tapeT_copyC, where YYYYMM is the year and month of the start of the run, T is the tape number (one run may require multiple tapes) and C is the copy number (we make two copies).  The naming is not known to the tape itself (the tapes do not have volume labels), but serves as a means to keep things organized.

When backup_day does a backup, the log file is named

    allegro:/home/observer/backup/log/backup_day_YYYYMMDD_yyyymmdd_hhmm

where YYYYMMDD is the date of the data being backed up and yyyymmdd and hhmm give the date and time of the start of the backup.  All dates and times are UT because allegro's clock is set to UT.

For each tape, we usually create a file with the same name as the tape in

    allegro:/home/observer/backup/log

and list in that file the backups done to that tape by the log file name.

Standard Tape Commands

The generic unix/linux tape utility program is called mt (for magnetic tape).  It lets you check tape status and navigate through a tape.  Its syntax is

     mt -f TAPE command (count)

where TAPE is the tape device name, command is one of a variety of tape commands, and count is parameter necessary for some of the commands.  The tape drive on allegro is /dev/nst0.  Some typical commands you will use are

Commmand
 Explanation
mt -f /dev/nst0 rewind rewinds the tape
mt -f /dev/nst0 offline rewinds and ejects the tape
mt -f /dev/nst0 status checks status -- is there a tape in the drive, where on the tape is the head positioned (file number, block number, partition number)

Many more commands are available; type man mt to see them.

Doing/Checking the Backup

  1. Where: the tape drive sits on the floor next to allegro.

  2. Which tape:

  3. At any terminal logged in to allegro as observer, cd to ~/backup/scripts/ and type

         backup_day YYYYMMDD >& temp1.log &

    where YYYYMMDD is the UT day of the data just taken.
     
  4. You can monitor the backup by typing

         tail -f temp1.log

    Note the name of the log file mentioned in temp1.log -- you will check it later.  To see the entire log file, type

         more temp1.log

  5. When the backup is done, the tape will be automatically rewound.  If the backup ends with more than 30 GB on the tape, then the tape will automatically ejected. 

    NOTE: if you try to back up to a tape that already has more than 30 GB on it, then the tape will be ejected without writing the backup.  This will be clearly indicated in temp1.log, so check temp1.log to make sure that the backup was actually written!  If not, you need to repeat with a blank tape.

  6. Check the log file indicated in temp1.log for errors.  If temp1.log is not available, just look for files beginning with backup_day_YYYYMMDD in the /home/observer/backup/log/ directory.  One quick way to check the log file is to type

         grep "tar:" logfilename

    which will list any errors.  Note that you will usually get the message

         Removing leading `/' from member names

    when you do this; ignore this message.  Notify the Bolocam support person of any errors.  You can check the amount of data written by just scrolling to the bottom of the log file (e.g., type cat logfilename); there will be a line near the end of the form

         Total bytes written: 2423296000 (2.3GB, 2.7MB/s)

    You should get 2.5 to 3 GB for a full night of data. 

  7. If the copy 1 backup has completed successfully, you can start the copy 2 backup.  Eject copy 1 using the mt -f /dev/nst0 offline command (can be done from any directory), then insert copy 2 and repeat the above, changing temp1.log to temp2.log in the above instructions.  Note that, if copy 1 was full and you had to start a new tape, then this will be true for copy 2 also and you can just start a fresh tape for copy 2 without trying to write to the full tape.

  8. If your backup filled up the tape, inform the Bolocam support person so he can run a verification on the backup.  Provide the full tape name, including the copy number, and where the tape is located.  Once the full tape is verified, copy 1 will go into the summit archive and copy 2 will go to Hilo.

Backup Scripts

A set of backup scripts have been written to make doing and checking backups relatively easy.

backup_day YYYYMMDD

This is the script used above, most users will need no others.  It backs up the specified day, appending to the end of the tape.

backup_day_overwrite YYYYMMDD N

This is a useful but dangerous script.  It backs up the specified day, but it fast forwards past the first N backups written to the tape and begins writing.  So it can overwrite data on the tape.  You should only use this script when you want to intentionally overwrite a backup (e.g., there was some problem with a backup and you want to try to rewrite it).  Since tapes are not random access devices,  be aware that overwriting will destroy not only the data that is overwritten, but any data after it also -- the new backup will write an end-of-data mark at the end of the backup it does, so any data past that point on the tape is likely inaccessible.

multiday_backup
This isn't really a formal script; it's just an example of a script that can string together a set of backup_day commands.  You may want to do this if you need to catch up on a few days of backups, or you ran into a corruption problem with a tape and want to write a new copy of the tape.  Feel free to modify as necessary.

multiday_backup_overwrite

This is like multiday_backup, but uses backup_day_overwrite.  Again, use with extreme care!

check_backup

This is a script to do a verification of tape.  It spins through the tape, doing tar -t on each archive on the tape to check that the archive is fully readable.  For each file that it checks, it creates a log file with the name

LOG_STUB_YYYYMMDD_HHMM

where LOG_STUB is the calling argument and YYYYMMDD_HHMM is the date that the verification of the particular file starts.  A global log with filename

LOG_STUB

is also written; it contains the position at start and end and size for each file on the tape.  It is sensible to use following for LOG_STUB:

for copy1: verify1/TAPE_NAME
for copy2: verify2/TAPE_NAME

where TAPE_NAME is the tape name of the form obsrun_YYYYMM_tapeT_copyC.

The main difficulty with the routine is that, in order to do an explicit comparison of the verification logs to the original logs, you have to figure out which backup logs correspond to which files.  You can usually do this by comparing the Total bytes written: line each of the original files (use grep to find the lines) to the file size in the global log file LOG_STUB.  Either the file sizes for the two copies written on a given day are identical, in which case it doesn't really matter which one you compare to, or the file sizes are different and it will be possible to use the file size to determine which log goes with which tape. 

However, it's not really necessary to do such an explicit comparison.  If the verification executes without any errors and the files sizes are vaguely correct (1-3 GB per day), then the archives on tape are readable and you shouldn't have any worries.

Troubleshooting

In general, the only problem you will have will be problems writing or reading tapes -- the backup routines themselves are both simple enough and have been used enough that they are robust.  If you have trouble writing a backup, you can try writing it again using backup_day_overwrite.  If it still fails, try a new tape; remember, you will need to rewrite all the backups on the tape, you can use multiday_backup to do this.

If you still have problems, something is probably wrong with the drive and you should contact the Bolocam support person.  Until the tape drive is back in action, you can write DVDs (one pair of duplicate copies per night, instructions given below) as a stopgap. 

DVD Nightly Backup

If the tape drive is not functioning, you should back up to DVD.  This is not a permanent archiving solution, but a stopgap until the tape drive functions again.
  1. Blank DVDs are available in the the file cabinet in the computer room.  If the number of DVDs is running low, contact the Bolocam support person.  Give the DVD a label of the form

    bolocam_YYYYMMDD_copyN

    where YYYYMMDD is the UT day for the data being backed up and N is a copy number (you should make 2 copies).  You should back up only one night to a given DVD to minimize confusion (and for possible future automation of backup recovery).

  2. /data00 on allegro should already be mounted on puuoo.  If not, become root and type mount /data00. 

  3. You have to use the DVD writer (top slot) on puuoo.  To make the backup, start k3b.  It will start automatically if you insert a blank DVD in the writer; give it 30 seconds or so to see the DVD.  If it does not start automatically, you may type

    > k3b &

    at a shell prompt.

  4. Open the project bolocam_dvd_backup_template.k3b (in the /home/kilauea/bolocam/ directory). 

  5. There will be 4 directories in the project: merged, rawdir, encdir, headerdir.  Go to /data00 and look for the corresponding directories (except for merged) with the appropriate date.  Use F5 to refresh the k3b listing.  Drag and drop each dated folder to the DVD; for example from /data00/encdir to encdir on the DVD, etc.  Hit burn when all are copied over.  Accept the defaults.  The program will show you progress.  Note that you are not backing up the merged directory.

Archiving Data for Transport to your Home Institution

You have some alternatives as to how to archive your Bolocam data for transport to your home institution.  In general, you should take the merged and/or sliced data.  You will likely reclean the data at your home institution, so there's no point in take cleaned data or maps.  The nightly backups ensure that, if your merged or sliced data has problems, the raw data can be reloaded to disk and remerged or resliced.

Here are your options:

Recovering Raw Data

If you have lost your copy of the data, or need to remerge some of the raw data, contact the Bolocam support person.  Your data may still be on disk on allegro or kilauea; it can certainly be spooled off tape onto disk so it can be remerged or copied across the network.

Revision History

Questions or comments? Contact the Bolocam support person.