cesm build fails

[Shruti]

Hello,

I was trying to run the following case on cesm 2.1.2 version.
./create_newcase --case newcase6 --compset B1850 --res f19_g17 --mach homebrew

I referred a link which had similar error to the ones i faced. And tried building cesm on the same lines

Build error with 'ocn.base.tavg.csh' error · Issue #74 · ESCOMP/CESM

Summary of Issue: Build Error with the following messages below Expected behavior and actual behavior: It won't build. No problem with Case.Setup . Steps to reproduce the problem. Include externals...

github.com

But i am still not able to build succesfully. It gives error for cesm build.

PFA the build file and the macros file.
Ca you please let me know what other approach i need to try.

 

[jedwards]

relocation truncated to fit: R_X86_64_PC32 against symbol
Indicates that the build requirements exceed the memory model limitations. Try an A compset instead.

 

[Shruti]

Hello,

Is there a way to resolve the issue-"relocation truncated to fit: R_X86_64_PC32 against symbol"
B1850 seems to be one of the standard compsets for testing. But it doesnt seem to be working for me.

 

[Shruti]

Hello,

I added the "-mcmodel=medium" option to run the above compset.
The run seems to be successful, but the log/timing files are not generated.
Even tried to run PET.f09_g17.B1850 test.
But the "run" seems to be in pending state(test execution was ongoing for ~2 hour).

TestStatus contents are as follows:

PASS PET.f09_g17.B1850.amd_gnu CREATE_NEWCASE
PASS PET.f09_g17.B1850.amd_gnu XML
PASS PET.f09_g17.B1850.amd_gnu SETUP
PASS PET.f09_g17.B1850.amd_gnu SHAREDLIB_BUILD time=71
PASS PET.f09_g17.B1850.amd_gnu MODEL_BUILD time=62
PASS PET.f09_g17.B1850.amd_gnu SUBMIT
PEND PET.f09_g17.B1850.amd_gnu RUN


Please let me know what options should i look into.

 

[jedwards]

I suggested earlier that you try an A or X compset, have you done that? If the run appears successful but the test didn't complete - did you build and install cprnc? Check the file TestStatus.out in the case directory for errors.

 

[Shruti]

Yes, I have already tried with A and X compset. It works fine.
Cprnc is installed and the path is placed in CCSM_CPRNC variable.
I tried with --wait option as well as without it.
- The one with --wait option seems to be running for a long time(more than 2 hours) with no exit status.
- The one without --wait option seems to be reach. Last few lines of TestStatus.log file is as follows:

Creating component namelists
Finished creating component namelists
Check case OK
Submitting job script sbatch --time 00:30:00 -q debug .case.test --skip-preview-namelist
Submitted job id is 718
Submitted job case.test with id 718
submit_jobs case.test
Submit job case.test

What other options should i check?

 

[jedwards]

Please attach the TestStatus.log from the failing test.

 

[Shruti]

I ran a fresh PET test today, the test was pending for more than 4 hours.
PFA TestStatus.log. Any suggestions would be appreciated

 

[jedwards]

No errors here - is the job in the queue or hung while running? If it's in the queue and not starting you may need to ask your sys-ad why.

 

[Shruti]

But if for other compsets, tests execute and move to completion and this does not then what can be concluded?

 

[jedwards]

You didn't answer the question - is the job hung while running or waiting in the queue?

 

[Shruti]

I am not sure if this is the right way, but I checked the running applications using "top".command.
Nothing supposedly related to CESM or slurm or sbatch(i have used slurm for queuing) is been displayed.
Is this the right way to check? Can you please guide me on the same.

 

[jedwards]

squeue is the command you are looking for.

 

[Shruti]

Hi,

This is the output of squeue. Pasting last few lines of the output here.

690 long run.case PD 0:00 1 (DependencyNeverSatisfied)
691 long gfs_post PD 0:00 1 (Dependency)
693 long run.case PD 0:00 1 (DependencyNeverSatisfied)
694 long gfs_post PD 0:00 1 (Dependency)
695 long run.cb2 PD 0:00 9 (PartitionNodeLimit)
696 long st_archi PD 0:00 1 (Dependency)
697 long test.SMS PD 0:00 9 (PartitionNodeLimit)
698 long test.PET PD 0:00 9 (PartitionNodeLimit)
699 long run.cb5 PD 0:00 6 (PartitionNodeLimit)
700 long st_archi PD 0:00 1 (Dependency)
703 long run.cb6 PD 0:00 6 (PartitionNodeLimit)
704 long st_archi PD 0:00 1 (Dependency)
717 long test.PET PD 0:00 9 (PartitionNodeLimit)
718 long test.PET PD 0:00 9 (PartitionNodeLimit)
720 long st_archi PD 0:00 1 (DependencyNeverSatisfied)
741 long test.NOD PD 0:00 4 (PartitionNodeLimit)
742 long test.PET PD 0:00 9 (PartitionNodeLimit)
743 long test.PET PD 0:00 9 (PartitionNodeLimit)
744 long test.PET PD 0:00 9 (PartitionNodeLimit)

Execution is ongoing and is stuck for around 15 min now. The output is the same as above

 

[jedwards]

We cannot answer questions about the details of your queueing system - you should talk to your system administrators.

 

[Shruti]

Hi,

This issue still seems to remain unsolved for me.
The execution doesn't seem to be starting as per the "squeue" command.
So instead I tried executing "mpirun" manually(instead of ./case.submit).
It gave the following error:
"Fortran runtime error: Cannot open file './timing/checkpoints/model_timing_00010102_00000_stats': No such file or directory"

So i created the required file and then the execution went for completion.
It is generating the timing files as attached.
But these files are not similar to the ones generated by other compset( A and X).
Any other idea which can work?

Can you please suggest any other compset which uses Lapack and Blas libraries?