Source codes and model outputs of AWIESM-wiso

Technique notes for AWIESM-wiso Xiaoxu Shi May 11, 2023 The AWIESM-wiso can be run under the framework of ESM-tools, here is an instruction on how to use ESM-tools to compile AWIESM-wiso and run PI and MH experiments. 1 Install ESM-tools First, make sure you add the following lines to one of your login or profile files, i.e. ∼/.bash profile, ∼/.bashrc, ∼/.profile, etc.: export PATH=$PATH: /.local/bin export LC ALL=en US.UTF-8 export LANG=en US.UTF-8 You can choose to source now your login or profile file, so that the module and export commands are run (e.g. source ∼/.bash profile). To use the ESM-Tools, written in Python, clone this repository: git clone https://github.com/esm-tools/esm tools.git Then, run the install.sh: ./install.sh You should now have the command line tools esm master and esm runscripts. For more details on how to install ESM-tools, please refer to https://github.com/esm-tools/esm tools. 2 Download and compile AWIESM2.1-wiso Create a folder called awiesm-2.1-wiso, in this folder, download the model codes and runscripts (codes runscripts.tar.gz) from zenodo (in the same repository where this file is in): Unzip codes runscripts.tar.gz, you can then have 4 new folders called: fesom-2.0-wiso containing the sourcec codes of the ocean component oasis containing the source codes of the coupler namelist AWIESM2-beta containing namelists of the model run-script containing two example run-scripts for pre-industrial and mid-Holocene simulations For the atmosphere component ECHAM6, please first apply for a personal user license at https://code.mpimet.mpg.de/projects/mpiesm-license, then the isotope-enabled version can be used upon request to [email protected]. Once the legal access to the the atmosphere component is achieved, please save it in a new folder called echam-6.3.05p2-wiso in awiesm-2.1-wiso. You need to rename fesom-2.0-wiso to fesom-2.1 as this is required for the compilation using ESM-tools. Now go to the parent folder, i.e., awiesm-2.1-wiso, and compile the model with the following command: esm master comp-awiesm-2.1-wiso 1 3 Make changes to AWIESM2.1-wiso You can modify the source code of all model components which are stored at: ECHAM: awiesm-2.1-wiso/echam-6.3.05p2-wiso/src/echam JSBACH: awiesm-2.1-wiso/echam-6.3.05p2-wiso/src/src jsbach3 Radiation: awiesm-2.1-wiso/echam-6.3.05p2-wiso/src/rad src FESOM: awiesm-2.1-wiso/fesom-2.1/src There are several ways to re-compile the model after making changes, one can go to the ”parent” folder of awiesm-2.1-wiso and run esm master comp-awiesm-2.1-wiso It will re-compile all model components and often takes a long time. If you would like to compile just one model component (either atmosphere or ocean), just go into the folder awiesm-2.1-wiso and you can see the following files: comp-echam-6.3.05p2-wiso script.sh comp-fesom-2.1-wiso script.sh The script comp-echam-6.3.05p2-wiso script.sh is for compiling the atmosphere component (ECHAM, JSBACH, radiation) and the comp-fesom-2.1-wiso script.sh is for the ocean (FESOM). After running the script, you need to replace the binary file: cp echam-6.3.05p2-wiso/bin/echam6 bin/ cp fesom-2.1/bin/fesom.x bin/ 4 Run AWIESM2.1-wiso Set a path as your working folder, where your experiments should run and produce outputs. In this path, you can start simulations. 4.1 Pre-industrial (PI) control simulation An run-script example to run a PI simulation is here: run-script/pi.yaml Here is a detailed explanation of the script: general: this section define general information of the run. setup name: ”awiesm” default, don’t change version: ”2.1-wiso” default, don’t change with wiso: False do you want to run the simulation with isotopes compute time: ”01:55:00” the maximum compute time. The run will stop once the time limit is reached. initial date: ”2000-01-01” The first model year/month/day of the simulation, the year must be large than (or equal to) 1000 final date: ”2300-12-31” The last model year/month/day of the simulation base dir: ”/home/a/a270064/bb1029/AWIESM2.1-EXP/” Please change it to the path of your working folder!!! nmonth: 0 default, only change it if you would like to restart the simulation every month, instead of every year. nyear: 1 the run restarted every model year account: ”ab0246” your working project, where the CPU consumed from. use venv: False default, don’t change echam: this section define general information of the atmosphere configuration. lresume: 0 0 means a cold start, 1 means a warm start (continued from a previous run). If lresume=1, you need to define the following 4 lines. ini parent exp id: ”pi fix” the name of the experiment you would like to restart from (only valid of lresume=1). 2 fake initial date: ”2133-01-01” The restart date, it tells esm.tool which restart files should be read from the previous simulation (only valid of lresume=1). ini parent date: ”$(( ${fake initial date}-${echam.time step}seconds ))” default, don’t change (only valid of lresume=1). ini parent dir: ”/home/a/a270064/bb1029/awiesm exp/pi fix70/restart/echam/” the folder containing the echam restart files (only valid of lresume=1). restart unit: ”years” default, only change it if you would like to restart the simulation every month, instead of every year. nproca: 48 default, don’t change nprocb: 16 default, don’t change nprocar: 0 switch off concurrent radiation, default, don’t change nprocbr: 0 switch off concurrent radiation, default, don’t change namelist changes: the following controls the namelists namelist.echam: the following controls namelist.echam parctl: the following controls the ”parctl” part of namelist.echam nproca: 48 default, don’t change nprocb: 16 default, don’t change nprocar: 0 switch off concurrent radiation, default, don’t change nprocbr: 0 switch off concurrent radiation, default, don’t change runctl: the following controls the ”runctl” part of namelist.echam default output: True default, don’t change putdata: [ 1, ’months’, ’last’, 0 ] time frequency of model output, higher frequency can make the model slower. radctl: the following controls the ”radctl” part of namelist.echam lrad async: False switch off concurrent radiation, default, don’t change awiesm: this section define general information of the model version and path. postprocessing: true default, don’t change scenario: ”PALEO” default, don’t change model dir: ”/home/a/a270064/esm tools/awiesm-2.1-wiso/” change it to the path of your model, it tells esm-tool where to read the binary files of the model. fesom: this section define general information of the ocean configuration. lresume: 0 0 means a cold start, 1 means a warm start (continued from a previous run). If lresume=1, you need to define the following 3 lines. ini parent exp id: pi fix the name of the experiment you would like to restart from (only valid of lresume=1). ini parent dir: ”/home/a/a270064/bb1029/awiesm exp/from ollie/pi fix/restart/fesom/” the folder containing the echam restart files (only valid of lresume=1). ini parent date: ”2113-12-31” The restart date, it tells esm.tool which restart files should be read from the previous simulation (only valid of lresume=1). nproc: 384 how many CPUs are used for fesom version: ”2.1-wiso” default, don’t change res: ”CORE2” default, don’t change ALE scheme: ’linfs’ use the non-linear free surface scheme, default, don’t change pool dir: ”/pool/data/AWICM/FESOM2” default, don’t change mesh dir: ” /home/a/a270064/bb1029/inputs/awicm2 final384/” default, only change it if you intend to use a different mesh. namelist dir: ”/home/a/a270064/ab0246/files for Xiaoxu Notes/namelist AWIESM2beta/” the path to default fesom namelists. restart rate: 1 restart unit: ”y” These two lines means the run will be restarted every 1 year. You can change it as you want. It should be consistent with the information defined in the general section. restart first: 1 default, don’t change namelist changes: the following controls the namelists namelist.config: the following controls namelist.config paths: the following controls the ”paths” part of namelist.config 3 ClimateDataPath: ”/home/a/a270064/bb1029/inputs/hydrography/” where to read the initial conditions for a cold-start run, which is usually stored in phc3.0 winter.nc (for temperature and salinity), and in wiso.nc (for isotopes). geometry: the following controls the ”geometry” part of namelist.config force rotation: False does fesom need to rotate the mesh inside the mode? If the mesh is already rotated, then set it to False, otherwise to True. ncar bulk z wind: remove from namelist default, don’t change ncar bulk z tair: remove from namelist default, don’t change ncar bulk z shum: remove from namelist default, don’t change jsbach: this section define general information of the land surface module JSBACH. input sources: the following line defines the input file for JSBACH jsbach 1850: ”/home/a/a270064/bb1029/inputs/mesh core2/tarfilesT63/input/jsbach /jsbach T63CORE2 11tiles 5layers 1850.nc” define the input file for JSBACH oasis3mct: this section define general information of the coupler OASIS lresume: 0 0 means a cold start, 1 means a warm start (continued from a previous run). If lresume=1, you need to define the following 3 lines. Note that lresume=0 leads to slow model speed for the first model year. ini parent exp id: ”mh fix” doesn’t matter ini parent date: ”22731231” doesn’t matter ini restart dir: ”/home/a/a270064/bb1029/AWIESM2.1-EXP/pi beta1/restart/oasis3mct/” It defines the path to the restart files, esm-tool will copy a2o flux and o2a flux from this folder. After necessary modification to the run-script (i.e., pi.yaml), you can start your PI simulation with: esm runscripts pi.yaml -e [exp name] Replace [exp name] with the name of your experiment. 4.2 Mid-Holocene (MH) simulation To perfrom MH or LIG simulations, one can stick to the run-script of PI, just with some minor changes in the echam section: For MH (example file: run-script/mh.yaml): runctl: default output: True putdata: [ 1, ’months’, ’last’, 0 ] l orbvsop87: False set it to False so the orbital parameters (cecc, cobld, clonp) can be updated from the following settings. radctl: lrad async: False CO2VMR: 264.36e-6 set CO2 concentration CH4VMR: 0.56466e-6 set CH4 concentration N2OVMR: 0.26065e-6 set N2O concentration cecc: 0.0186818 set eccentricity cobld: 24.104767 set obliquity clonp: 180.917664 set perihelion yr perp: -99999 this line is necessary, I don’t know why. jsbach: input sources: jsbach 1850: ”/home/a/a270064/bb1029/inputs/mesh core2 /jsbach T63CORE2 11tiles 5layers natural-veg.nc” For paleo simulation, crops and pastures should be removed from the vegetation. 4 5 Check your simulations 5.1 Status of the jobs Run squeue -u [user id] to see the status of your simulations. In my case: [a270064@levante1 work]$squeue -u a270064 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 2741450 compute lig beta a270064 R 2:34 9 l[10309-10311,10316-10321] 2741439 compute pi beta a270064 R 3:35 9 l[30113-30121] It means 2 jobs ”lig beta” and ”pi beta” are running with the job ID of 2741450 and 2741439 respectively. You can see the experiment folders ”lig beta” and ”pi beta” in the [base dir] defined in general section of the run-script. Here is a list of sub-folders in your experiment folder: analysis: not important, it’s empty bin: it contains the binary files of the model config: history run-scripts, namelists etc of each model year. couple: not important, it’s empty forcing: history forcing data of each model year input: necessary input files to run the model log: history log files of each model year mon: not important, it’s empty outdata: important, it contains the model output data restart: history restart files of each model year scripts: your main run-script and some other files generated by esm-tool src: not important, it’s empty unknown: a lot of unknown files, clean it regularly as it occupy a lot of space viz: not important, it’s empty Besides, you might see a lot of run * folders, where the experiment is/was running. Please delete those run * folders regularly, except the last run * folder (as your experiment is now running in it). 5.2 Cancel a simulation To kill a job, just use scancel [job id] 5.3 Continue a simulation If your job is canceled and your would like to resume it, go to your experiment folder, then go to the sub-folder ”scripts”, you can see your run-script there, submit it. 5.4 If a simulation stops automatically It might means that the final date (defined in the general section of your run-script) is reached, your can prolong the final date and continue the run. It can also means that the upper time limit is reached, in this case just increase the time limit compute time (defined in the general section of your run-script). Errors can also lead to break of simulations, for troubleshooting please refer to the next section. 6 Troubleshooting Some useful log files for you to check the simulation: run */work/atmout it gives information about the echam component run */log/*log it gives information about both fesom and echam History logs for each integrated model year can be found in log of your experiment folder. 5 Here I list the most common errors according to my experience: 6.1 Overflow Go to the last run * folder, check work/atmout, if you see an error message related to ”overflow”, then you need to increase ”enstdif” in your namelist.echam: vi work/namelist.echam add the following in namelist.echam: dynctl enstdif = 1.00001 / then: cd ../scripts sbatch *.run 6.2 Job neither runs nor stops If you find that your job hang before the integration until the time limit is reached, and log files give no error message, it is hard to know the reason. I only know that it might be caused by using a rotated/unrotated fesom mesh but set force rotation (in fesom section of the run-script) to True/False, just correct the setting and try again. 6.3 Blow up If your log file tell you the model blows up, you can see a blow-up NetCDF file in run */work, please check the data and see what is the problem (mostly due to salinity/temperature/sea surface height exceeding the a certain range). Reduce the time step of fesom might help to solve the problem. It the error occurs at the beginning of a cold-start simulation, try the following: (1) make sure you have lresume=0 in oasis3mct section of your run-script. (If lresume=1, it might lead to blowup error if the restart files are from a simulation using a different ocean mesh or runing with a different number of CPUs). (2) If (1) does not help, then it might means that the temperature/salinity gradient in your initial condition is too sharp in some area, horizontally or vertically. Try to smooth your initial condition. 6.4 oclook: no ocean cell found for HD grid cell Please refer to section ??. 6.5 liquid water content above field capacity FATAL ERROR in digest evapotrans crash Please refer to section ??. 6.6 cuadjtq (1): lookup table overflow FATAL ERROR in cuadjtq (1): lookup table overflow with WARNING! high wind speed in the highest vertical layer. Please refer to section ??. 6.7 1385423270394660728 If the log file gives you an error number of 1385423270394660728. It means that the number of the surface nodes defined in your run-script is not identical to the mesh you are using. You need to define it correctly (in fesom section: nx: [number of nodes]). 6 6.8 1385423270818802975 This error number means at least one of your restart files for the coupler (oasis) is missing. (1) If it happens in the first model year, please check the ini restart dir in oasis3mct section of your run-script to see if there are o2a flux and a2o flux files in the path. (2) If it happens in the second model year, it is likely due to the fact that esm-tool did not correctly produce a2o flux for the first model year. I don’t know how to solve it technically but my trick is to manually copy a a2o flux file from other experiment and paste it into restart/oasis3mct. Note that if your run is with isotopes, then the a2o flux must be copied from another run with isotopes too. 7 Good luck! 7