Scheduled Downtime
On Tuesday 24 October 2023 @ 5pm MT the forums will be in read only mode in preparation for the downtime. On Wednesday 25 October 2023 @ 5am MT, this website will be down for maintenance and expected to return online later in the morning.
Normal Operations
The forums are back online with normal operations. If you notice any issues or errors related to the forums, please reach out to help@ucar.edu

ERROR: (shr_stream_getCalendar) ERROR: nf90_open file /media/dell/data/cesm/inputdata /lmwg/atm_forcing.datm7.GSWP3.0.5d.v1.c170516/Precip/clmforc

kaeritai

New Member
hi dear all,
I am trying to run CLM5 and found the below error: I guess it was form reading the NC file and thus,
I checked the files and make sure that they are not empty. Can anybody help me with this ? Thanks!

-------------------------------------------------------------------------------------------------------------------------------------
Invalid PIO rearranger comm max pend req (comp2io), 0
Resetting PIO rearranger comm max pend req (comp2io) to 64
PIO rearranger options:
comm type =
p2p

comm fcd =
2denable

max pend req (comp2io) = 0
enable_hs (comp2io) = T
enable_isend (comp2io) = F
max pend req (io2comp) = 64
enable_hs (io2comp) = F
enable_isend (io2comp) = T
(seq_comm_setcomm) init ID ( 1 GLOBAL ) pelist = 0 47 1 ( npes = 48) ( nthreads = 1)( suffix =)
(seq_comm_setcomm) init ID ( 2 CPL ) pelist = 0 47 1 ( npes = 48) ( nthreads = 1)( suffix =)
(seq_comm_setcomm) init ID ( 5 ATM ) pelist = 0 47 1 ( npes = 48) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 6 CPLATM ) join IDs = 2 5 ( npes = 48) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 3 ALLATMID ) join multiple comp IDs ( npes = 48) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 4 CPLALLATMID ) join IDs = 2 3 ( npes = 48) ( nthreads = 1)
(seq_comm_setcomm) init ID ( 9 LND ) pelist = 0 47 1 ( npes = 48) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 10 CPLLND ) join IDs = 2 9 ( npes = 48) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 7 ALLLNDID ) join multiple comp IDs ( npes = 48) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 8 CPLALLLNDID ) join IDs = 2 7 ( npes = 48) ( nthreads = 1)
(seq_comm_setcomm) init ID ( 13 ICE ) pelist = 0 47 1 ( npes = 48) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 14 CPLICE ) join IDs = 2 13 ( npes = 48) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 11 ALLICEID ) join multiple comp IDs ( npes = 48) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 12 CPLALLICEID ) join IDs = 2 11 ( npes = 48) ( nthreads = 1)
(seq_comm_setcomm) init ID ( 17 OCN ) pelist = 0 47 1 ( npes = 48) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 18 CPLOCN ) join IDs = 2 17 ( npes = 48) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 15 ALLOCNID ) join multiple comp IDs ( npes = 48) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 16 CPLALLOCNID ) join IDs = 2 15 ( npes = 48) ( nthreads = 1)
(seq_comm_setcomm) init ID ( 21 ROF ) pelist = 0 47 1 ( npes = 48) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 22 CPLROF ) join IDs = 2 21 ( npes = 48) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 19 ALLROFID ) join multiple comp IDs ( npes = 48) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 20 CPLALLROFID ) join IDs = 2 19 ( npes = 48) ( nthreads = 1)
(seq_comm_setcomm) init ID ( 25 GLC ) pelist = 0 47 1 ( npes = 48) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 26 CPLGLC ) join IDs = 2 25 ( npes = 48) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 23 ALLGLCID ) join multiple comp IDs ( npes = 48) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 24 CPLALLGLCID ) join IDs = 2 23 ( npes = 48) ( nthreads = 1)
(seq_comm_setcomm) init ID ( 29 WAV ) pelist = 0 47 1 ( npes = 48) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 30 CPLWAV ) join IDs = 2 29 ( npes = 48) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 27 ALLWAVID ) join multiple comp IDs ( npes = 48) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 28 CPLALLWAVID ) join IDs = 2 27 ( npes = 48) ( nthreads = 1)
(seq_comm_setcomm) init ID ( 33 ESP ) pelist = 0 0 1 ( npes = 1) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 34 CPLESP ) join IDs = 2 33 ( npes = 48) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 31 ALLESPID ) join multiple comp IDs ( npes = 1) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 32 CPLALLESPID ) join IDs = 2 31 ( npes = 48) ( nthreads = 1)
(seq_comm_printcomms) 1 0 48 1 GLOBAL:
(seq_comm_printcomms) 2 0 48 1 CPL:
(seq_comm_printcomms) 3 0 48 1 ALLATMID:
(seq_comm_printcomms) 4 0 48 1 CPLALLATMID:
(seq_comm_printcomms) 5 0 48 1 ATM:
(seq_comm_printcomms) 6 0 48 1 CPLATM:
(seq_comm_printcomms) 7 0 48 1 ALLLNDID:
(seq_comm_printcomms) 8 0 48 1 CPLALLLNDID:
(seq_comm_printcomms) 9 0 48 1 LND:
(seq_comm_printcomms) 10 0 48 1 CPLLND:
(seq_comm_printcomms) 11 0 48 1 ALLICEID:
(seq_comm_printcomms) 12 0 48 1 CPLALLICEID:
(seq_comm_printcomms) 13 0 48 1 ICE:
(seq_comm_printcomms) 14 0 48 1 CPLICE:
(seq_comm_printcomms) 15 0 48 1 ALLOCNID:
(seq_comm_printcomms) 16 0 48 1 CPLALLOCNID:
(seq_comm_printcomms) 17 0 48 1 OCN:
(seq_comm_printcomms) 18 0 48 1 CPLOCN:
(seq_comm_printcomms) 19 0 48 1 ALLROFID:
(seq_comm_printcomms) 20 0 48 1 CPLALLROFID:
(seq_comm_printcomms) 21 0 48 1 ROF:
(seq_comm_printcomms) 22 0 48 1 CPLROF:
(seq_comm_printcomms) 23 0 48 1 ALLGLCID:
(seq_comm_printcomms) 24 0 48 1 CPLALLGLCID:
(seq_comm_printcomms) 25 0 48 1 GLC:
(seq_comm_printcomms) 26 0 48 1 CPLGLC:
(seq_comm_printcomms) 27 0 48 1 ALLWAVID:
(seq_comm_printcomms) 28 0 48 1 CPLALLWAVID:
(seq_comm_printcomms) 29 0 48 1 WAV:
(seq_comm_printcomms) 30 0 48 1 CPLWAV:
(seq_comm_printcomms) 31 0 1 1 ALLESPID:
(seq_comm_printcomms) 32 0 48 1 CPLALLESPID:
(seq_comm_printcomms) 33 0 1 1 ESP:
(seq_comm_printcomms) 34 0 48 1 CPLESP:
(t_initf) Read in prof_inparm namelist from: drv_in
(t_initf) Using profile_disable= F
(t_initf) profile_timer= 4
(t_initf) profile_depth_limit= 4
(t_initf) profile_detail_limit= 2
(t_initf) profile_barrier= F
(t_initf) profile_outpe_num= 1
(t_initf) profile_outpe_stride= 0
(t_initf) profile_single_file= F
(t_initf) profile_global_stats= T
(t_initf) profile_ovhd_measurement= F
(t_initf) profile_add_detail= F
(t_initf) profile_papi_enable= F
ERROR:
(shr_stream_getCalendar) ERROR: nf90_open file /media/dell/data/cesm/inputdata
/lmwg/atm_forcing.datm7.GSWP3.0.5d.v1.c170516/Precip/clmforc.GSWP3.c2011.0.5x0.
5.Prec.2010-01.nc
Image PC Routine Line Source
cesm.exe 00000000010F0A5D shr_abort_mod_mp_ 114 shr_abort_mod.F90
cesm.exe 0000000001241069 shr_stream_mod_mp 2021 shr_stream_mod.F90
cesm.exe 00000000012527AB shr_stream_mod_mp 648 shr_stream_mod.F90
cesm.exe 0000000001234174 shr_strdata_mod_m 1303 shr_strdata_mod.F90
cesm.exe 00000000004FEE2B datm_shr_mod_mp_d 154 datm_shr_mod.F90
cesm.exe 00000000004F35CE atm_comp_mct_mp_a 141 atm_comp_mct.F90
cesm.exe 0000000000439AF3 component_mod_mp_ 267 component_mod.F90
cesm.exe 00000000004287BB cime_comp_mod_mp_ 1175 cime_comp_mod.F90
cesm.exe 0000000000436B8C MAIN__ 92 cime_driver.F90
cesm.exe 000000000041945D Unknown Unknown Unknown
libc-2.31.so 00001519528CF083 __libc_start_main Unknown Unknown
cesm.exe 000000000041937E Unknown Unknown Unknown
Abort(1001) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1001) - process 0
 

oleson

Keith Oleson
CSEG and Liaisons
Staff member
The traceback seems to indicate a problem opening the precip netcdf file to get the calendar, possibly here in shr_stream_getCalendar:

if (rcode /= nf90_noerr) call shr_sys_abort(subname//' ERROR: nf90_open file '//trim(filename))
rCode = nf90_inq_varid(fid,trim(strm%domTvarName),vid)

What version of CLM/CESM are you using (the output of git describe)?

I'd do an ncdump -k to make sure it's not a NETCDF-4 file. And do an ncdump -h on the file and attach the output please.
 

kaeritai

New Member
The traceback seems to indicate a problem opening the precip netcdf file to get the calendar, possibly here in shr_stream_getCalendar:

if (rcode /= nf90_noerr) call shr_sys_abort(subname//' ERROR: nf90_open file '//trim(filename))
rCode = nf90_inq_varid(fid,trim(strm%domTvarName),vid)

What version of CLM/CESM are you using (the output of git describe)?

I'd do an ncdump -k to make sure it's not a NETCDF-4 file. And do an ncdump -h on the file and attach the output please.
hi dear Oleson,
thank you for your reply. I used CLM5 and the netcdf are downloaded from the CESM website and is classic. I am not sure ;but I changed the mode of all
the data as chmod 777 and seems resolved the above problem . Please note that all forcing data are shared by all users and I am not root.

However, I got another bug (as below). I guess there is NaNs for the variable Faxa_rainc? Thus, I tried cdo fillmiss2 as from https://bb.cgd.ucar.edu/cesm/threads/error-m50-component_mod-check_fields-nan-found-in-atm-instance.4773/ but still failed.

Can you help me with this? thank you.
------------------------------------------------------------------------------------------------------------------------
Invalid PIO rearranger comm max pend req (comp2io), 0
Resetting PIO rearranger comm max pend req (comp2io) to 64
PIO rearranger options:
comm type =
p2p

comm fcd =
2denable

max pend req (comp2io) = 0
enable_hs (comp2io) = T
enable_isend (comp2io) = F
max pend req (io2comp) = 64
enable_hs (io2comp) = F
enable_isend (io2comp) = T
(seq_comm_setcomm) init ID ( 1 GLOBAL ) pelist = 0 35 1 ( npes = 36) ( nthreads = 1)( suffix =)
(seq_comm_setcomm) init ID ( 2 CPL ) pelist = 0 35 1 ( npes = 36) ( nthreads = 1)( suffix =)
(seq_comm_setcomm) init ID ( 5 ATM ) pelist = 0 35 1 ( npes = 36) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 6 CPLATM ) join IDs = 2 5 ( npes = 36) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 3 ALLATMID ) join multiple comp IDs ( npes = 36) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 4 CPLALLATMID ) join IDs = 2 3 ( npes = 36) ( nthreads = 1)
(seq_comm_setcomm) init ID ( 9 LND ) pelist = 0 35 1 ( npes = 36) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 10 CPLLND ) join IDs = 2 9 ( npes = 36) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 7 ALLLNDID ) join multiple comp IDs ( npes = 36) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 8 CPLALLLNDID ) join IDs = 2 7 ( npes = 36) ( nthreads = 1)
(seq_comm_setcomm) init ID ( 13 ICE ) pelist = 0 35 1 ( npes = 36) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 14 CPLICE ) join IDs = 2 13 ( npes = 36) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 11 ALLICEID ) join multiple comp IDs ( npes = 36) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 12 CPLALLICEID ) join IDs = 2 11 ( npes = 36) ( nthreads = 1)
(seq_comm_setcomm) init ID ( 17 OCN ) pelist = 0 35 1 ( npes = 36) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 18 CPLOCN ) join IDs = 2 17 ( npes = 36) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 15 ALLOCNID ) join multiple comp IDs ( npes = 36) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 16 CPLALLOCNID ) join IDs = 2 15 ( npes = 36) ( nthreads = 1)
(seq_comm_setcomm) init ID ( 21 ROF ) pelist = 0 35 1 ( npes = 36) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 22 CPLROF ) join IDs = 2 21 ( npes = 36) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 19 ALLROFID ) join multiple comp IDs ( npes = 36) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 20 CPLALLROFID ) join IDs = 2 19 ( npes = 36) ( nthreads = 1)
(seq_comm_setcomm) init ID ( 25 GLC ) pelist = 0 35 1 ( npes = 36) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 26 CPLGLC ) join IDs = 2 25 ( npes = 36) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 23 ALLGLCID ) join multiple comp IDs ( npes = 36) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 24 CPLALLGLCID ) join IDs = 2 23 ( npes = 36) ( nthreads = 1)
(seq_comm_setcomm) init ID ( 29 WAV ) pelist = 0 35 1 ( npes = 36) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 30 CPLWAV ) join IDs = 2 29 ( npes = 36) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 27 ALLWAVID ) join multiple comp IDs ( npes = 36) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 28 CPLALLWAVID ) join IDs = 2 27 ( npes = 36) ( nthreads = 1)
(seq_comm_setcomm) init ID ( 33 ESP ) pelist = 0 0 1 ( npes = 1) ( nthreads = 1)( suffix =)
(seq_comm_joincomm) init ID ( 34 CPLESP ) join IDs = 2 33 ( npes = 36) ( nthreads = 1)
(seq_comm_jcommarr) init ID ( 31 ALLESPID ) join multiple comp IDs ( npes = 1) ( nthreads = 1)
(seq_comm_joincomm) init ID ( 32 CPLALLESPID ) join IDs = 2 31 ( npes = 36) ( nthreads = 1)
(seq_comm_printcomms) 1 0 36 1 GLOBAL:
(seq_comm_printcomms) 2 0 36 1 CPL:
(seq_comm_printcomms) 3 0 36 1 ALLATMID:
(seq_comm_printcomms) 4 0 36 1 CPLALLATMID:
(seq_comm_printcomms) 5 0 36 1 ATM:
(seq_comm_printcomms) 6 0 36 1 CPLATM:
(seq_comm_printcomms) 7 0 36 1 ALLLNDID:
(seq_comm_printcomms) 8 0 36 1 CPLALLLNDID:
(seq_comm_printcomms) 9 0 36 1 LND:
(seq_comm_printcomms) 10 0 36 1 CPLLND:
(seq_comm_printcomms) 11 0 36 1 ALLICEID:
(seq_comm_printcomms) 12 0 36 1 CPLALLICEID:
(seq_comm_printcomms) 13 0 36 1 ICE:
(seq_comm_printcomms) 14 0 36 1 CPLICE:
(seq_comm_printcomms) 15 0 36 1 ALLOCNID:
(seq_comm_printcomms) 16 0 36 1 CPLALLOCNID:
(seq_comm_printcomms) 17 0 36 1 OCN:
(seq_comm_printcomms) 18 0 36 1 CPLOCN:
(seq_comm_printcomms) 19 0 36 1 ALLROFID:
(seq_comm_printcomms) 20 0 36 1 CPLALLROFID:
(seq_comm_printcomms) 21 0 36 1 ROF:
(seq_comm_printcomms) 22 0 36 1 CPLROF:
(seq_comm_printcomms) 23 0 36 1 ALLGLCID:
(seq_comm_printcomms) 24 0 36 1 CPLALLGLCID:
(seq_comm_printcomms) 25 0 36 1 GLC:
(seq_comm_printcomms) 26 0 36 1 CPLGLC:
(seq_comm_printcomms) 27 0 36 1 ALLWAVID:
(seq_comm_printcomms) 28 0 36 1 CPLALLWAVID:
(seq_comm_printcomms) 29 0 36 1 WAV:
(seq_comm_printcomms) 30 0 36 1 CPLWAV:
(seq_comm_printcomms) 31 0 1 1 ALLESPID:
(seq_comm_printcomms) 32 0 36 1 CPLALLESPID:
(seq_comm_printcomms) 33 0 1 1 ESP:
(seq_comm_printcomms) 34 0 36 1 CPLESP:
(t_initf) Read in prof_inparm namelist from: drv_in
(t_initf) Using profile_disable= F
(t_initf) profile_timer= 4
(t_initf) profile_depth_limit= 4
(t_initf) profile_detail_limit= 2
(t_initf) profile_barrier= F
(t_initf) profile_outpe_num= 1
(t_initf) profile_outpe_stride= 0
(t_initf) profile_single_file= F
(t_initf) profile_global_stats= T
(t_initf) profile_ovhd_measurement= F
(t_initf) profile_add_detail= F
(t_initf) profile_papi_enable= F
ERROR:
component_mod:check_fields NaN found in ATM instance: 1 field Faxa_rainc 1d
global index: 385
 

oleson

Keith Oleson
CSEG and Liaisons
Staff member
There isn't enough information here for me to diagnose the problem. Normally you shouldn't have to do any filling on GSWP3 data that is provided. Are you running a regional or single-point simulation for a location for which there isn't any valid GSWP3 data?
It would be helpful if you could provide the information requested here:

 

oleson

Keith Oleson
CSEG and Liaisons
Staff member
I guess I don't understand how you are getting NaN or missing values for that field since the GSWP3 data is global and there shouldn't be any missing values and certainly not any NaNs. Have you successfully run the model previously on your machine?
Have you searched the file you are getting the error from for NaNs or missing values?
 

kaeritai

New Member
There isn't enough information here for me to diagnose the problem. Normally you shouldn't have to do any filling on GSWP3 data that is provided. Are you running a regional or single-point simulation for a location for which there isn't any valid GSWP3 data?
It would be helpful if you could provide the information requested here:

sorry for my vague questions. I used CLM 5.0.0.3 and the compset of I1850Clm50Bgc on a GLOBAL scale.
Please note that I checked the precipitation data and found no NaN.

I show the NC file detailes:

dimensions:
time = UNLIMITED ; // (248 currently)
lon = 720 ;
lat = 360 ;
scalar = 1 ;
variables:
float time(time) ;
time:standard_name = "time" ;
time:units = "days since 2010-01-01 00:00:00" ;
time:calendar = "365_day" ;
time:axis = "T" ;
float LONGXY(lat, lon) ;
LONGXY:long_name = "longitude" ;
LONGXY:units = "degrees_east" ;
LONGXY:mode = "time-invariant" ;
float LATIXY(lat, lon) ;
LATIXY:long_name = "latitude" ;
LATIXY:units = "degrees_north" ;
LATIXY:mode = "time-invariant" ;
float EDGEE(scalar) ;
EDGEE:long_name = "eastern edge in atmospheric data" ;
EDGEE:units = "degrees_east" ;
EDGEE:mode = "time-invariant" ;
float EDGEW(scalar) ;
EDGEW:long_name = "western edge in atmospheric data" ;
EDGEW:units = "degrees_east" ;
EDGEW:mode = "time-invariant" ;
float EDGES(scalar) ;
EDGES:long_name = "southern edge in atmospheric data" ;
EDGES:units = "degrees_north" ;
EDGES:mode = "time-invariant" ;
float EDGEN(scalar) ;
EDGEN:long_name = "northern edge in atmospheric data" ;
EDGEN:units = "degrees_north" ;
EDGEN:mode = "time-invariant" ;
float PRECTmms(time, lat, lon) ;
PRECTmms:long_name = "PRECTmms total precipitation" ;
PRECTmms:units = "mm H2O / sec" ;
PRECTmms:_FillValue = 1.e+36f ;
PRECTmms:missing_value = 1.e+36f ;
PRECTmms:mode = "time-dependent" ;

// global attributes:
:CDI = "Climate Data Interface version 1.9.9rc1 (https://mpimet.mpg.de/cdi)" ;
:Conventions = "CF-1.6" ;
:history = "Thu Apr 06 16:29:54 2023: cdo fillmiss2 clmforc.GSWP3.c2011.0.5x0.5.Prec.2010-01.nc clmforc.GSWP3.c2011.0.5x0.5.Prec.2010-01.nc1" ;
:case_title = "CRUNCEP 6-Hourly Atmospheric Forcing: Precipitation" ;
:CDO = "Climate Data Operators version 1.9.9rc1 (https://mpimet.mpg.de/cdo)" ;
}
 

oleson

Keith Oleson
CSEG and Liaisons
Staff member
I don't see any problems with that file unfortunately. If you provide the exact steps you took up to the point you got the error, I can try to replicate it here. However, we haven't had any similar problems with global simulations using GSWP3, so it may be a problem unique to your system.
I've also queried a user who had a similar problem here, maybe we will get an answer there:

 
Top