'ERROR: trcdata_init: failed to allocate x array' in SE case

[James King]

Hi all,

I've come across an error which I haven't seen before and which hasn't been discussed in the forums elsewhere. This is a MUSICA case running the FCnudged compset in CESM2.2.2. The case compiles and submits as normal, but crashes fairly quickly when loading in emissions files. Below is the relevant output from the atm log:

srf_emis_inti: n_emis_files = 87
(GETFIL): attempting to find local file CAMS-GLOB-ANT_Glb_ne0np4.UK_JAK_ne30x16_SO2_anthro-ag-ship-res_surface_v3.1_c20240607.nc
(GETFIL): using /work/n02/n02/jking/cesm/MUSICA_REPO/ne0np4.UK_JAK_ne30x16/CAMS_data/CAMS-GLOB-ANT_Glb_ne0np4.UK_JAK_ne30x16_SO2_anthro-ag-ship-res_surface_v3.1_c20240607.nc
trcdata_init: data type: CYCLICAL file: /work/n02/n02/jking/cesm/MUSICA_REPO/ne0np4.UK_JAK_ne30x16/CAMS_data/CAMS-GLOB-ANT_Glb_ne0np4.UK_JAK_ne30x16_SO2_anthro-ag-ship-res_surface_v3.1_c20240607.nc
(GETFIL): attempting to find local file CAMS-GLOB-ANT_Glb_ne0np4.UK_JAK_ne30x16_SO2_anthro-ag-ship-res_surface_v3.1_c20240607.nc
(GETFIL): using /work/n02/n02/jking/cesm/MUSICA_REPO/ne0np4.UK_JAK_ne30x16/CAMS_data/CAMS-GLOB-ANT_Glb_ne0np4.UK_JAK_ne30x16_SO2_anthro-ag-ship-res_surface_v3.1_c20240607.nc
open_trc_datafile: /work/n02/n02/jking/cesm/MUSICA_REPO/ne0np4.UK_JAK_ne30x16/CAMS_data/CAMS-GLOB-ANT_Glb_ne0np4.UK_JAK_ne30x16_SO2_anthro-ag-ship-res_surface_v3.1_c20240607.nc
trcdata_init: file%ps allocation error = 5014
ERROR: trcdata_init: failed to allocate x array

I'm not sure what this is - it looks like either some kind of memory error, or some objection to the properties of that input file. Said file looks fine to me, having the expected variables and dimensions, including (crucially) the same number of 'ncol' as the SE grid on which I'm trying to run the case, so I'm not sure what I'm missing.

Any advice is as ever much appreciated.

James

 

[James King]

Bump in case anyone has any ideas!

 

[yhanw]

Hello James,
I encountered exactly the same error and found your thread here. Do you happen to have any updates? I'd appreciate it if you can share any clues.
Thanks!
Yuhan

 

[rrbuchholz]

Hi James,

Is it just that file, or does it occur with other input emission files as well? i.e. is you skip that file in the user_nl_cam do you still get an error?
Are you using CONUS RR setup, or NE30?
Did you create the file using the python regridding tool? Sometimes the regridding script can fail without throwing an error and leave an empty array (but creates a file with correct dimensions). Are there real values within the emission file (not all zeros or NaNs).

Best,
Rebecca

 

[James King]

Hi,

I've tried with different files and keep running into the same problem, including with files that work fine for a colleague. The files in question do have data in them, and I'm trying to use an ne30 grid over the UK. What does this error mean and does anyone have any suggestions about resolving it? As it is I am unable to actually use MUSICA at all, as this error is always thrown by an ext_frc_specifier file, whether this is a standard 1 degree fv file or one that has been regridded to the MUSICA grid, either using Python or the tools provided in the IPT repo.

Thanks,

James

 

[rrbuchholz]

Hi James,

Here is what I would try to diagnose the error:

• Do the other log files have any information in them?
• Is there an earlier WARNING in the atm log file that might be helpful?
• Can you get an out of the box MUSICA simulation to run?
• Are there any hidden characters in your user_nl_cam file?
• Where is the first point of failure? i.e. if you remove /work/n02/n02/jking/cesm/MUSICA_REPO/ne0np4.UK_JAK_ne30x16/CAMS_data/CAMS-GLOB-ANT_Glb_ne0np4.UK_JAK_ne30x16_SO2_anthro-ag-ship-res_surface_v3.1_c20240607.nc from the user_nl_cam, can you get past the error?

Let me know how it goes.
Best,
Rebecca

 

[rrbuchholz]

I chatted with Shawn and he wondered if it may be that the model needs more resources (specifically memory). He suggested:

increase the number of nodes by a couple in env_mach_pes.xml to get a memory bump and see if that works

 

[James King]

Hi Rebecca,

Thanks for the suggestions! In response to your questions:

• There aren't any error messages in any of the other log files other than the atm.log, which terminates with the same error as the cesm.log during the process of reading in the external forcing specifiers, e.g
  • (GETFIL): attempting to find local file CAMS-GLOB-AIR_v2.1_2000-2020_ne0np4.UKnIR_ne30x16_SO2_c20250623.nc (GETFIL): using /work/n02/n02/shared/CESM/UKnIR_test/emissions/CAMS/CAMS-GLOB-AIR_v2.1_2000-2020_ne0np4.UKnIR_ne30x16_SO2_c20250623.nc
    open_trc_datafile: /work/n02/n02/shared/CESM/UKnIR_test/emissions/CAMS/CAMS-GLOB-AIR_v2.1_2000-2020_ne0np4.UKnIR_ne30x16_SO2_c20250623.nc
    trcdata_init: file%ps allocation error = 5014
    ERROR: trcdata_init: failed to allocate x array
• The only warning in the atm log relates to the CFL stability test rather than the input files, and the guide for MUSICA physics setup says I can safely ignore the particular warning here.
• I've previously been able to run out-of-the-box configurations, and I can also run FHIST using this user-generated grid, so the problem does seem to be chemistry-related.
• I don't think there are any hidden characters in user_nl_cam - I didn't put any in!
• If I comment out the exf_frc_specifier files, the run fails with a segmentation fault, with no errors in the atm.log and this in the cesm.log:
  • Reading zbgc_nml
    MCT::m_Router::initp_: GSMap indices not increasing...Will correct
    MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
    MCT::m_Router::initp_: RGSMap indices not increasing...Will correct
    MCT::m_Router::initp_: GSMap indices not increasing...Will correct
    
    Program received signal SIGSEGV: Segmentation fault - invalid memory reference
    Backtrace for this error:
    #0 0x1507b8d9fd4f in ???
    #1 0x951a30 in ???
    #2 0x956951 in ???
    #3 0x95b3b1 in ???
    #4 0xc399a5 in ???
    #5 0xa28ae7 in ???
    #6 0x5ed2dc in ???
    #7 0x4a4230 in ???
    #8 0x4a0703 in ???
    #9 0x4269f4 in ???
    #10 0x41a66e in ???
    #11 0x40b187 in ???
    #12 0x1507b8d8a29c in ???
    #13 0x40b319 in ???
    at ../sysdeps/x86_64/start.S:120

I will throw a few more nodes at it and see if that does anything. Currently I am getting the error with 9 nodes on ARCHER2, where each node has 128 processor cores and 256GB of memory.

Thanks,

James

 

[James King]

Quick update - I can now get the case to run using the raw 1deg CMIP6 ext_frc_specifier and srf_emis_specifier files. I've not made any changes (which is simultaneously helpful and unhelpful), but have been made aware of some maintenance issues with the HPC which are currently being investigated. I will keep the thread updated once I have tested with the files I need, which are 2022 regridded MERRA-2 atmospheric nudging and regridded SSP emissions.

 

[rrbuchholz]

OK - good to know! Keep us updated.

Regarding the hidden characters, it sounds like this wasn't the issue if it is now running. But FYI if you use something like notepad, that can automatically format things like quotes, so it can automatically add hidden characters without you knowing... just something to be aware of. You can check for hidden characters in an editor like vi with the command

:set list

 

[James King]

Currently the case can run with regridded meteorology for nudging and standard 1 degree emission files. However, the idea ./cof MUSICA is to be able to look at chemistry in higher resolution, and I get this exact same error when I try and run with emissions files that have been regridded and renamed from the raw SSP3 input files using the NCL scripts provided in the IPT repo (IPT/Emissions/CMIP at master · NCAR/IPT). The MUSICA wiki mentions a Python regridding/renaming routing but the linked URL returns a 404 error (https://ncar.github.io/CAM-chem/examples/functions/Regridding.html).

An example of the error in the atm log when I try and run with regridded emissions files:

srf_emis_inti: n_emis_files = 52
(GETFIL): attempting to find local file emissions-cmip6-SSP_C2H4_other_surface_surface_1750_2015_ne0np4.UKnIR_ne30x16_c20250527.nc
(GETFIL): using /work/n02/n02/jking/cesm/MUSICA_REPO/ne0np4.UKnIR_ne30x16/emis/CMIP6/emissions-cmip6-SSP_C2H4_other_surface_surface_1750_2015_ne0np4.UKnIR_ne30x16_c20250527.nc
trcdata_init: data type: INTERP_MISSING_MONTHS file: /work/n02/n02/jking/cesm/MUSICA_REPO/ne0np4.UKnIR_ne30x16/emis/CMIP6/emissions-cmip6-SSP_C2H4_other_surface_surface_1750_2015_ne0np4.UKnIR_ne30x16_c20250527.nc
(GETFIL): attempting to find local file emissions-cmip6-SSP_C2H4_other_surface_surface_1750_2015_ne0np4.UKnIR_ne30x16_c20250527.nc
(GETFIL): using /work/n02/n02/jking/cesm/MUSICA_REPO/ne0np4.UKnIR_ne30x16/emis/CMIP6/emissions-cmip6-SSP_C2H4_other_surface_surface_1750_2015_ne0np4.UKnIR_ne30x16_c20250527.nc
open_trc_datafile: /work/n02/n02/jking/cesm/MUSICA_REPO/ne0np4.UKnIR_ne30x16/emis/CMIP6/emissions-cmip6-SSP_C2H4_other_surface_surface_1750_2015_ne0np4.UKnIR_ne30x16_c20250527.nc
trcdata_init: file%ps allocation error = 5014
ERROR: trcdata_init: failed to allocate x array

 

[rrbuchholz]

Hi James,

The correct link for regridding is here:

Python Resources for chemistry modeling

I think I updated the broken link, but let me know if you find another.

Here is the updated IPT code location

MUSICA-Tools/IPT at main · NCAR/MUSICA-Tools

Tools to be used with atmospheric chemistry modeling - NCAR/MUSICA-Tools

github.com

Cheers,
Rebecca