Warning
There are known security vulnerabilities in Open XDMoD versions ≤11.0.2. We strongly encourage upgrading immediately to the latest version of Open XDMoD, 11.0.3, which contains fixes for these vulnerabilities.
To upgrade version 8.0 to 11.0.3, you will first need to upgrade to version 8.1.0, then upgrade to version 8.5.0, then upgrade to version 9.0.0, then upgrade to version 9.5.0, then upgrade to version 10.0.3, then upgrade to version 10.5.1, then upgrade to version 11.0.3.
If you cannot upgrade immediately, you can follow the instructions below to manually patch your installation as a temporary workaround before upgrading later.
Workaround instructions
- First, make sure you are on the correct web page for your version of Open XDMoD. The web page you are on now is for version 8.0. Other versions' pages are linked below:
- Download the patch file for your version:
- Copy the patch file to your Open XDMoD web server.
-
Make sure you have the
patchcommand installed; it can be installed withdnf install patch. -
Apply the patch by running the command below, replacing
[PATH_TO_XDMOD_SHARE_DIR]with the path to the share directory (/usr/share/xdmodfor RPM-based installs,/opt/xdmod/shareor another location for source code installs) and replacing[PATH_TO_PATCH_FILE]with the path to the patch file.# patch -p1 -d [PATH_TO_XDMOD_SHARE_DIR] < [PATH_TO_PATCH_FILE]
- Upgrade to the latest version of Open XDMoD as soon as you can.
Storage (Beta)
NOTE: Storage metrics are currently considered beta quality.
These instructions use the file paths from the RPM installation. If you’ve installed from source they will need to be adjusted accordingly.
Setup
Add Storage Resource
Add a storage resource using the xdmod-setup script or by manually modifying
/etc/xdmod/resources.json. If you use the xdmod-setup script be sure to
select a storage resource type or you will need to manually add the roles
configuration as described in the following section.
The resource name (also referred to as the resource code; not the formal name) must then be used in the JSON storage input files described below.
Update Roles Configuration
If you did not use the xdmod-setup script you must enable the storage query
descripters manually. This can be done by creating a new file,
roles.d/storage.json, with the appropriate contents. An example is shown
below and can be copied from /usr/share/xdmod/templates/roles.d/storage.json.
{
"+roles": {
"+default": {
"+query_descripters": [
{
"realm": "Storage",
"group_by": "none"
},
{
"realm": "Storage",
"group_by": "resource"
},
{
"realm": "Storage",
"group_by": "resource_type"
},
{
"realm": "Storage",
"group_by": "mountpoint"
},
{
"realm": "Storage",
"group_by": "person"
},
{
"realm": "Storage",
"group_by": "pi"
},
{
"realm": "Storage",
"group_by": "username"
},
{
"realm": "Storage",
"group_by": "nsfdirectorate"
},
{
"realm": "Storage",
"group_by": "parentscience"
},
{
"realm": "Storage",
"group_by": "fieldofscience"
}
]
},
"+pub": {
"+query_descripters": [
{
"realm": "Storage",
"group_by": "none"
},
{
"realm": "Storage",
"group_by": "resource"
},
{
"realm": "Storage",
"group_by": "resource_type"
},
{
"realm": "Storage",
"group_by": "mountpoint"
},
{
"realm": "Storage",
"group_by": "person"
},
{
"realm": "Storage",
"group_by": "pi"
},
{
"realm": "Storage",
"group_by": "username",
"disable": true
},
{
"realm": "Storage",
"group_by": "nsfdirectorate"
},
{
"realm": "Storage",
"group_by": "parentscience"
},
{
"realm": "Storage",
"group_by": "fieldofscience"
}
]
}
}
}
After adding the file you must update the ACL database tables:
$ acl-config && acl-import
Input Format
NOTE: The thresholds and usage numbers are all measured in bytes. Mountpoint names are currently limited to 255 characters.
[
{
"resource": "nfs",
"mountpoint": "/home",
"user": "jdoe",
"pi": "pi_username",
"dt": "2017-01-01T00:00:00",
"soft_threshold": 1000000,
"hard_threshold": 1200000,
"file_count": 10000,
"logical_usage": 100000,
"physical_usage": 100000
},
...
]
Data Ingestion
All of the following commands must be executed in the order specified below to fully ingest storage data into the data warehouse.
Ingest all files in the /path/to/storage/logs directory:
$ /usr/share/xdmod/tools/etl/etl_overseer.php \
-d STORAGE_LOG_DIRECTORY=/path/to/storage/logs \
-p xdmod.staging-ingest-storage
NOTE: The above command will ingest all files in the /path/to/storage/logs
directory even if they have already been ingested.
Ingest and aggregate data:
$ /usr/share/xdmod/tools/etl/etl_overseer.php \
-p xdmod.staging-ingest-common \
-p xdmod.hpcdb-ingest-common \
-p xdmod.hpcdb-ingest-storage
$ /usr/share/xdmod/tools/etl/etl_overseer.php \
-a xdmod.hpcdb-xdw-ingest.resource \
-a xdmod.hpcdb-xdw-ingest.field-of-science \
-a xdmod.hpcdb-xdw-ingest.field-of-science-hierarchy \
-a xdmod.hpcdb-xdw-ingest.organization \
-a xdmod.hpcdb-xdw-ingest.pi-person \
-a xdmod.hpcdb-xdw-ingest.person \
-a xdmod.hpcdb-xdw-ingest.people-under-pi \
-a xdmod.hpcdb-xdw-ingest.principal-investigator \
-a xdmod.hpcdb-xdw-ingest.resource-type \
-a xdmod.hpcdb-xdw-ingest.system-account
$ /usr/share/xdmod/tools/etl/etl_overseer.php \
-p xdmod.xdw-ingest-storage \
-p xdmod.xdw-aggregate-storage
Rebuild filter lists after aggregation:
$ /usr/bin/xdmod-build-filter-lists -r Storage
