Instructions on how to setup and run PyConform independent of the workflow

You will need to enter your experiment into the experiment database before proceeding with the below instructions.

The database is found at https://csegweb.cgd.ucar.edu/expdb2.0

One Time Setup

  1. Run the below command
pip install -i https://pypi.python.org/pypi --user  dreqPy==01.00.31
  1. Create a file called ~/.subversion/cmip6.conf and add the following
[svn]
password: your_svn_password

Then run

chmod 600 ~/.subversion/cmip6.conf

Instructions for Running PyConform on Cheyenne

  1. Setup the post-processing environment variables:
setenv POSTPROCESS_PATH /glade/u/home/mickelso/CESM_postprocessing_3/
setenv POSTPROCESS_PATH_GEYSER /glade/u/home/mickelso/CESM_postprocessing_3/
  1. Make sure you have the following modules loaded
intel/17.0.1
python/2.7.16
netcdf/4.6.3
mpt/2.19
  1. cd into your case directory
  2. Run the following commands
alias cesm_pp_activate 'source $POSTPROCESS_PATH/cesm-env2/bin/activate.csh'
cesm_pp_activate
create_postprocess --caseroot [fully-qualified-path-to-caseroot]
deactivate
./archive_metadata --query_cmip6 [casename that matches name in database]  db.json --user [your-username] --password --expType CMIP6
  1. Take a look at the $CASE/archive_files/db.json file to verify the information is correct.
  2. cd into the $CASE/postprocess directory that was created in the above set.
  3. Open the file env_postprocess.xml and edit the following XML variables:
CASEROOT                   (the full path to your case directory)
PP_CASE_PATH               (your_case_directory/postprocess)
CASE                       (your case name)
DOUT_S_ROOT                (the full path to your raw CESM output, usually in the scratch/username/archive directory)
TIMESERIES_OUTPUT_ROOTDIR  (full path to the timeseries output)
  1. Open the file env_conform.xml and edit the following XML variables:
CONFORM_CESM_DEFINITIONS   (modify on if running on a land only or WACCM experiment)
CONFORM_EXP_NAM            (the official and exact cmip6 experiment name, see the database for this name)
CONFORM_JSON_DIRECTORY     (this should be something similar to your_case_directory/postprocess/)
CONFORM_OUTPUT_DIR         (where you want pyconform to put the output)
  1. Run the command
./iconform

on the command line from the postprocess directory. This should create a directory in your postprocessing directory called PyConform_input. make sure there are files in this directory and make sure your logs/iconform.<date> file does not contain errors that the top of this file. Warning should be okay.

  1. If the above step passes the two checks, open the xconform file and make sure the pbs settings at the top look okay.
  2. Run this command to submit the job to the queue:
qsub xconform
  1. Your output should appear in the directory you set the variable CONFORM_OUTPUT_DIR to.
  2. If no output is produced, check the logs/xconform.<date> for error messages. PyConform can be difficult to debug because of its internal graph structures and its ability to catch certain errors accurately. If you run into issues, you can email mickelso.at.ucar.edu with your error information.