This guide will attempt to outline the use of the Open XDMoD shredder command line utility. The shredder is responsible for loading data from resource manager log files into the Open XDMoD databases. If you are running Slurm, you should first read the Slurm Notes.
In order to make data available to the Open XDMoD portal you will need to use the shredder utility. If you followed the install guide, you will have already used the shredder to populate your database. In addition to the install process, this program is typically used once a day to add jobs from the the previous day to the database.
To display the shredder help text from the command line:
$ xdmod-shredder -h
By default the Open XDMoD shredder only outputs what it considers to be warnings or errors. If you would like to see informational output about what is being performed, use the verbose option:
$ xdmod-shredder -v ...
Debugging output is also available:
$ xdmod-shredder --debug ...
You must specify a resource name (the name of your cluster) during the shredding process. This name must match the name that you provided during the setup process. When using Slurm this name must match the cluster name.
$ xdmod-shredder -r mycluster ...
You must specify the format of the log files you are shredding. This is
dependant on the resource manager you use.
For TORQUE and OpenPBS use
pbs, for Sun Grid Engine use
sge, for Univa Grid Engine 8.2+ use
uge, for Slurm use
slurm and for LSF use
$ xdmod-shredder -f pbs ... $ xdmod-shredder -f sge ... $ xdmod-shredder -f uge ... $ xdmod-shredder -f slurm ... $ xdmod-shredder -f lsf ...
Files may be shredded one at a time:
$ xdmod-shredder -i file ...
An entire directory of files can be shredded, but the names of the files
must be formatted as
YYYYMMDD (e.g. 20120101).
The log file for the current day will be ignored (along with any files that correspond to future dates). This is intended to prevent the shredding of partial log files.
If the database is empty all files that meet the above constraints will be shredded. If there is data in the database, only files dated after the date of the most recent job will be shredded.
$ xdmod-shredder -d directory ...