MPI VERSION : CRAY MPICH2 XT version 5.1.3 (ANL base 1.2.1p1) BUILD INFO : Built Tue Dec 7 18:06:42 2010 (svn rev 8190) PE 0: MPICH environment settings: PE 0: MPICH_ENV_DISPLAY = 1 PE 0: MPICH_VERSION_DISPLAY = 1 PE 0: MPICH_ABORT_ON_ERROR = 1 PE 0: MPICH_CPUMASK_DISPLAY = 0 PE 0: MPICH_RANK_REORDER_METHOD = 1 PE 0: MPICH_RANK_REORDER_DISPLAY = 0 PE 0: MPICH_MEMCPY_MEM_CHECK = 0 PE 0: MPICH_MAX_THREAD_SAFETY = single PE 0: MPICH_COLL_SYNC = 0 PE 0: MPICH_MSG_QUEUE_DBG = 0 PE 0: MPICH_CPU_YIELD = 0 PE 0: MPICH_MSGS_PER_PROC = 160000 PE 0: MPICH/SMP environment settings: PE 0: MPICH_SMP_OFF = 0 PE 0: MPICH_SMPDEV_BUFS_PER_PROC = 32 PE 0: MPICH_SMP_SINGLE_COPY_SIZE = 131072 PE 0: MPICH_SMP_SINGLE_COPY_OFF = 0 PE 0: MPICH/PORTALS environment settings: PE 0: MPICH_MAX_SHORT_MSG_SIZE = 960000 PE 0: MPICH_UNEX_BUFFER_SIZE = 629145600 PE 0: MPICH_PTL_UNEX_EVENTS = 90000 PE 0: MPICH_PTL_OTHER_EVENTS = 2048 PE 0: MPICH_PTL_MEMD_LIMIT = 2048 PE 0: MPICH_VSHORT_OFF = 0 PE 0: MPICH_MAX_VSHORT_MSG_SIZE = 1024 PE 0: MPICH_VSHORT_BUFFERS = 32 PE 0: MPICH_PTL_EAGER_LONG = 0 PE 0: MPICH_PTL_MATCH_OFF = 0 PE 0: MPICH_PTL_SEND_CREDITS = 703 (-1) PE 0: MPICH/COLLECTIVE environment settings: PE 0: MPICH_FAST_MEMCPY = 0 PE 0: MPICH_COLL_OPT_OFF = 0 PE 0: MPICH_BCAST_ONLY_TREE = 1 PE 0: MPICH_ALLTOALL_SHORT_MSG = 1024 PE 0: MPICH_REDUCE_SHORT_MSG = 65536 PE 0: MPICH_REDUCE_LARGE_MSG = 131072 PE 0: MPICH_ALLREDUCE_LARGE_MSG = 262144 PE 0: MPICH_ALLGATHER_VSHORT_MSG = 2048 PE 0: MPICH_ALLGATHERV_VSHORT_MSG= 2048 PE 0: MPICH_ALLTOALLVW_FCSIZE = 32 PE 0: MPICH_ALLTOALLVW_SENDWIN = 20 PE 0: MPICH_ALLTOALLVW_RECVWIN = 20 PE 0: MPICH_SCATTERV_MAX_PROCS = 512 PE 0: MPICH/MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 0 PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH_MPIIO_META_CHUNK_SIZE = 500 PE 0: MPICH_MPIIO_SEND_CHUNK_SIZE = 100000 PE 0: MPICH_MPIIO_HINTS = NULL (seq_comm_setcomm) initialize ID ( 7 GLOBAL ) pelist = 0 63 1 ( npes = 64) ( nthreads = 1) (seq_comm_setcomm) initialize ID ( 2 ATM ) pelist = 0 63 1 ( npes = 64) ( nthreads = 1) (seq_comm_setcomm) initialize ID ( 1 LND ) pelist = 0 63 1 ( npes = 64) ( nthreads = 1) (seq_comm_setcomm) initialize ID ( 4 ICE ) pelist = 0 63 1 ( npes = 64) ( nthreads = 1) (seq_comm_setcomm) initialize ID ( 5 GLC ) pelist = 0 63 1 ( npes = 64) ( nthreads = 1) (seq_comm_setcomm) initialize ID ( 3 OCN ) pelist = 0 63 1 ( npes = 64) ( nthreads = 1) (seq_comm_setcomm) initialize ID ( 6 CPL ) pelist = 0 63 1 ( npes = 64) ( nthreads = 1) (seq_comm_joincomm) initialize ID ( 8 CPLATM ) join IDs = 6 2 ( npes = 64) ( nthreads = 1) (seq_comm_joincomm) initialize ID ( 9 CPLLND ) join IDs = 6 1 ( npes = 64) ( nthreads = 1) (seq_comm_joincomm) initialize ID ( 10 CPLICE ) join IDs = 6 4 ( npes = 64) ( nthreads = 1) (seq_comm_joincomm) initialize ID ( 11 CPLOCN ) join IDs = 6 3 ( npes = 64) ( nthreads = 1) (seq_comm_joincomm) initialize ID ( 12 CPLGLC ) join IDs = 6 5 ( npes = 64) ( nthreads = 1) (seq_comm_printcomms) ID layout : global pes vs local pe for each ID gpe LND ATM OCN ICE GLC CPL GLOBAL CPLATM CPLLND CPLICE CPLOCN CPLGLC nthrds --- ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ 0 : 0 0 0 0 0 0 0 0 0 0 0 0 1 1 : 1 1 1 1 1 1 1 1 1 1 1 1 1 2 : 2 2 2 2 2 2 2 2 2 2 2 2 1 3 : 3 3 3 3 3 3 3 3 3 3 3 3 1 4 : 4 4 4 4 4 4 4 4 4 4 4 4 1 12 : 12 12 12 12 12 12 12 12 12 12 12 12 1 13 : 13 13 13 13 13 13 13 13 13 13 13 13 1 14 : 14 14 14 14 14 14 14 14 14 14 14 14 1 15 : 15 15 15 15 15 15 15 15 15 15 15 15 1 16 : 16 16 16 16 16 16 16 16 16 16 16 16 1 17 : 17 17 17 17 17 17 17 17 17 17 17 17 1 18 : 18 18 18 18 18 18 18 18 18 18 18 18 1 19 : 19 19 19 19 19 19 19 19 19 19 19 19 1 20 : 20 20 20 20 20 20 20 20 20 20 20 20 1 28 : 28 28 28 28 28 28 28 28 28 28 28 28 1 29 : 29 29 29 29 29 29 29 29 29 29 29 29 1 30 : 30 30 30 30 30 30 30 30 30 30 30 30 1 31 : 31 31 31 31 31 31 31 31 31 31 31 31 1 32 : 32 32 32 32 32 32 32 32 32 32 32 32 1 33 : 33 33 33 33 33 33 33 33 33 33 33 33 1 34 : 34 34 34 34 34 34 34 34 34 34 34 34 1 35 : 35 35 35 35 35 35 35 35 35 35 35 35 1 36 : 36 36 36 36 36 36 36 36 36 36 36 36 1 44 : 44 44 44 44 44 44 44 44 44 44 44 44 1 45 : 45 45 45 45 45 45 45 45 45 45 45 45 1 46 : 46 46 46 46 46 46 46 46 46 46 46 46 1 47 : 47 47 47 47 47 47 47 47 47 47 47 47 1 48 : 48 48 48 48 48 48 48 48 48 48 48 48 1 49 : 49 49 49 49 49 49 49 49 49 49 49 49 1 50 : 50 50 50 50 50 50 50 50 50 50 50 50 1 51 : 51 51 51 51 51 51 51 51 51 51 51 51 1 52 : 52 52 52 52 52 52 52 52 52 52 52 52 1 53 : 53 53 53 53 53 53 53 53 53 53 53 53 1 54 : 54 54 54 54 54 54 54 54 54 54 54 54 1 55 : 55 55 55 55 55 55 55 55 55 55 55 55 1 56 : 56 56 56 56 56 56 56 56 56 56 56 56 1 60 : 60 60 60 60 60 60 60 60 60 60 60 60 1 61 : 61 61 61 61 61 61 61 61 61 61 61 61 1 62 : 62 62 62 62 62 62 62 62 62 62 62 62 1 63 : 63 63 63 63 63 63 63 63 63 63 63 63 1 37 : 37 37 37 37 37 37 37 37 37 37 37 37 1 38 : 38 38 38 38 38 38 38 38 38 38 38 38 1 39 : 39 39 39 39 39 39 39 39 39 39 39 39 1 5 : 5 5 5 5 5 5 5 5 5 5 5 5 1 6 : 6 6 6 6 6 6 6 6 6 6 6 6 1 7 : 7 7 7 7 7 7 7 7 7 7 7 7 1 40 : 40 40 40 40 40 40 40 40 40 40 40 40 1 8 : 8 8 8 8 8 8 8 8 8 8 8 8 1 41 : 41 41 41 41 41 41 41 41 41 41 41 41 1 42 : 42 42 42 42 42 42 42 42 42 42 42 42 1 43 : 43 43 43 43 43 43 43 43 43 43 43 43 1 (t_initf) Read in prof_inparm namelist from: drv_in 9 : 9 9 9 9 9 9 9 9 9 9 9 9 1 10 : 10 10 10 10 10 10 10 10 10 10 10 10 1 11 : 11 11 11 11 11 11 11 11 11 11 11 11 1 21 : 21 21 21 21 21 21 21 21 21 21 21 21 1 22 : 22 22 22 22 22 22 22 22 22 22 22 22 1 23 : 23 23 23 23 23 23 23 23 23 23 23 23 1 57 : 57 57 57 57 57 57 57 57 57 57 57 57 1 58 : 58 58 58 58 58 58 58 58 58 58 58 58 1 59 : 59 59 59 59 59 59 59 59 59 59 59 59 1 24 : 24 24 24 24 24 24 24 24 24 24 24 24 1 25 : 25 25 25 25 25 25 25 25 25 25 25 25 1 26 : 26 26 26 26 26 26 26 26 26 26 26 26 1 27 : 27 27 27 27 27 27 27 27 27 27 27 27 1 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 8 MB memory dealloc in MB is 0.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4090.01 8 MB memory dealloc in MB is 0.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 Memory block size conversion in bytes is 4090.01 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 64 pes participating in computation ----------------------------------- TASK# NAME 0 nid01702 1 nid01702 2 nid01702 3 nid01702 4 nid01703 5 nid01703 6 nid01703 7 nid01703 8 nid01704 9 nid01704 10 nid01704 11 nid01704 12 nid01730 13 nid01730 14 nid01730 15 nid01730 16 nid01731 17 nid01731 18 nid01731 19 nid01731 20 nid01732 21 nid01732 22 nid01732 23 nid01732 24 nid01733 25 nid01733 26 nid01733 27 nid01733 28 nid01734 29 nid01734 30 nid01734 31 nid01734 32 nid01735 33 nid01735 34 nid01735 35 nid01735 36 nid01736 37 nid01736 38 nid01736 39 nid01736 40 nid01737 41 nid01737 42 nid01737 43 nid01737 44 nid01738 45 nid01738 46 nid01738 47 nid01738 48 nid01739 49 nid01739 50 nid01739 51 nid01739 52 nid01740 53 nid01740 54 nid01740 55 nid01740 56 nid01741 57 nid01741 58 nid01741 59 nid01741 60 nid01742 61 nid01742 62 nid01742 63 nid01742 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 29 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 20 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 50 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 13 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 10 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 63 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 45 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 42 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 12 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 30 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 26 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 5 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 37 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 24 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 8 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 51 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 15 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 28 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 61 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 14 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 31 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 60 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 19 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 32 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 16 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 62 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 35 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 34 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 33 _pmii_daemon(SIGCHLD): [NID 01702] [c13-0c1s1n2] [Fri May 13 17:42:21 2011] PE 3 exit signal Aborted ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc _pmii_daemon(SIGCHLD): [NID 01730] [c13-0c2s0n2] [Fri May 13 17:42:21 2011] PE 13 exit signal Aborted ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc _pmii_daemon(SIGCHLD): [NID 01735] [c13-0c2s1n3] [Fri May 13 17:42:21 2011] PE 33 exit signal Aborted ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 48 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc _pmii_daemon(SIGCHLD): [NID 01742] [c13-0c2s3n2] [Fri May 13 17:42:21 2011] PE 63 exit signal Aborted [NID 01702] 2011-05-13 17:41:57 Apid 5200925: initiated application termination aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 58 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 39 _pmii_daemon(SIGCHLD): [NID 01734] [c13-0c2s1n2] [Fri May 13 17:42:21 2011] PE 31 exit signal Aborted aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 22 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 47 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 11 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 54 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 43 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 27 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 18 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 36 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 49 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 21 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 17 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 53 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 56 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 46 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 40 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 25 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 9 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 44 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 7 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 38 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 23 ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 59 _pmii_daemon(SIGCHLD): [NID 01739] [c13-0c2s2n3] [Fri May 13 17:42:21 2011] PE 50 exit signal Aborted ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc _pmii_daemon(SIGCHLD): [NID 01732] [c13-0c2s1n0] [Fri May 13 17:42:21 2011] PE 20 exit signal Aborted aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 6 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 55 _pmii_daemon(SIGCHLD): [NID 01738] [c13-0c2s2n2] [Fri May 13 17:42:21 2011] PE 47 exit signal Aborted aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 41 _pmii_daemon(SIGCHLD): [NID 01704] [c13-0c1s2n0] [Fri May 13 17:42:21 2011] PE 9 exit signal Aborted ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc _pmii_daemon(SIGCHLD): [NID 01737] [c13-0c2s2n1] [Fri May 13 17:42:21 2011] PE 42 exit signal Aborted _pmii_daemon(SIGCHLD): [NID 01703] [c13-0c1s1n3] [Fri May 13 17:42:21 2011] PE 6 exit signal Aborted aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 52 _pmii_daemon(SIGCHLD): [NID 01736] [c13-0c2s2n0] [Fri May 13 17:42:21 2011] PE 38 exit signal Aborted ENDRUN:GETFIL: FAILED to get N20TRAERCN_f19_g16_01.cam2.i.1978-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 57 _pmii_daemon(SIGCHLD): [NID 01740] [c13-0c2s3n0] [Fri May 13 17:42:21 2011] PE 54 exit signal Aborted _pmii_daemon(SIGCHLD): [NID 01733] [c13-0c2s1n1] [Fri May 13 17:42:21 2011] PE 24 exit signal Aborted _pmii_daemon(SIGCHLD): [NID 01731] [c13-0c2s0n3] [Fri May 13 17:42:21 2011] PE 17 exit signal Aborted _pmii_daemon(SIGCHLD): [NID 01741] [c13-0c2s3n1] [Fri May 13 17:42:21 2011] PE 58 exit signal Aborted Application 5200925 exit codes: 134 Application 5200925 resources: utime 0, stime 0