MPI VERSION : CRAY MPICH2 XT version 5.1.3 (ANL base 1.2.1p1) BUILD INFO : Built Tue Dec 7 18:06:42 2010 (svn rev 8190) PE 0: MPICH environment settings: PE 0: MPICH_ENV_DISPLAY = 1 PE 0: MPICH_VERSION_DISPLAY = 1 PE 0: MPICH_ABORT_ON_ERROR = 1 PE 0: MPICH_CPUMASK_DISPLAY = 0 PE 0: MPICH_RANK_REORDER_METHOD = 1 PE 0: MPICH_RANK_REORDER_DISPLAY = 0 PE 0: MPICH_MEMCPY_MEM_CHECK = 0 PE 0: MPICH_MAX_THREAD_SAFETY = single PE 0: MPICH_COLL_SYNC = 0 PE 0: MPICH_MSG_QUEUE_DBG = 0 PE 0: MPICH_CPU_YIELD = 0 PE 0: MPICH_MSGS_PER_PROC = 160000 PE 0: MPICH/SMP environment settings: PE 0: MPICH_SMP_OFF = 0 PE 0: MPICH_SMPDEV_BUFS_PER_PROC = 32 PE 0: MPICH_SMP_SINGLE_COPY_SIZE = 131072 PE 0: MPICH_SMP_SINGLE_COPY_OFF = 0 PE 0: MPICH/PORTALS environment settings: PE 0: MPICH_MAX_SHORT_MSG_SIZE = 960000 PE 0: MPICH_UNEX_BUFFER_SIZE = 629145600 PE 0: MPICH_PTL_UNEX_EVENTS = 90000 PE 0: MPICH_PTL_OTHER_EVENTS = 2048 PE 0: MPICH_PTL_MEMD_LIMIT = 2048 PE 0: MPICH_VSHORT_OFF = 0 PE 0: MPICH_MAX_VSHORT_MSG_SIZE = 1024 PE 0: MPICH_VSHORT_BUFFERS = 32 PE 0: MPICH_PTL_EAGER_LONG = 0 PE 0: MPICH_PTL_MATCH_OFF = 0 PE 0: MPICH_PTL_SEND_CREDITS = 703 (-1) PE 0: MPICH/COLLECTIVE environment settings: PE 0: MPICH_FAST_MEMCPY = 0 PE 0: MPICH_COLL_OPT_OFF = 0 PE 0: MPICH_BCAST_ONLY_TREE = 1 PE 0: MPICH_ALLTOALL_SHORT_MSG = 1024 PE 0: MPICH_REDUCE_SHORT_MSG = 65536 PE 0: MPICH_REDUCE_LARGE_MSG = 131072 PE 0: MPICH_ALLREDUCE_LARGE_MSG = 262144 PE 0: MPICH_ALLGATHER_VSHORT_MSG = 2048 PE 0: MPICH_ALLGATHERV_VSHORT_MSG= 2048 PE 0: MPICH_ALLTOALLVW_FCSIZE = 32 PE 0: MPICH_ALLTOALLVW_SENDWIN = 20 PE 0: MPICH_ALLTOALLVW_RECVWIN = 20 PE 0: MPICH_SCATTERV_MAX_PROCS = 512 PE 0: MPICH/MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 0 PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH_MPIIO_META_CHUNK_SIZE = 500 PE 0: MPICH_MPIIO_SEND_CHUNK_SIZE = 100000 PE 0: MPICH_MPIIO_HINTS = NULL (seq_comm_setcomm) initialize ID ( 7 GLOBAL ) pelist = 0 63 1 ( npes = 64) ( nthreads = 1) (seq_comm_setcomm) initialize ID ( 2 ATM ) pelist = 0 63 1 ( npes = 64) ( nthreads = 1) (seq_comm_setcomm) initialize ID ( 1 LND ) pelist = 0 63 1 ( npes = 64) ( nthreads = 1) (seq_comm_setcomm) initialize ID ( 4 ICE ) pelist = 0 63 1 ( npes = 64) ( nthreads = 1) (seq_comm_setcomm) initialize ID ( 5 GLC ) pelist = 0 63 1 ( npes = 64) ( nthreads = 1) (seq_comm_setcomm) initialize ID ( 3 OCN ) pelist = 0 63 1 ( npes = 64) ( nthreads = 1) (seq_comm_setcomm) initialize ID ( 6 CPL ) pelist = 0 63 1 ( npes = 64) ( nthreads = 1) (seq_comm_joincomm) initialize ID ( 8 CPLATM ) join IDs = 6 2 ( npes = 64) ( nthreads = 1) (seq_comm_joincomm) initialize ID ( 9 CPLLND ) join IDs = 6 1 ( npes = 64) ( nthreads = 1) (seq_comm_joincomm) initialize ID ( 10 CPLICE ) join IDs = 6 4 ( npes = 64) ( nthreads = 1) (seq_comm_joincomm) initialize ID ( 11 CPLOCN ) join IDs = 6 3 ( npes = 64) ( nthreads = 1) (seq_comm_joincomm) initialize ID ( 12 CPLGLC ) join IDs = 6 5 ( npes = 64) ( nthreads = 1) (seq_comm_printcomms) ID layout : global pes vs local pe for each ID gpe LND ATM OCN ICE GLC CPL GLOBAL CPLATM CPLLND CPLICE CPLOCN CPLGLC nthrds --- ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ 0 : 0 0 0 0 0 0 0 0 0 0 0 0 1 1 : 1 1 1 1 1 1 1 1 1 1 1 1 1 2 : 2 2 2 2 2 2 2 2 2 2 2 2 1 3 : 3 3 3 3 3 3 3 3 3 3 3 3 1 4 : 4 4 4 4 4 4 4 4 4 4 4 4 1 5 : 5 5 5 5 5 5 5 5 5 5 5 5 1 6 : 6 6 6 6 6 6 6 6 6 6 6 6 1 7 : 7 7 7 7 7 7 7 7 7 7 7 7 1 8 : 8 8 8 8 8 8 8 8 8 8 8 8 1 9 : 9 9 9 9 9 9 9 9 9 9 9 9 1 10 : 10 10 10 10 10 10 10 10 10 10 10 10 1 11 : 11 11 11 11 11 11 11 11 11 11 11 11 1 12 : 12 12 12 12 12 12 12 12 12 12 12 12 1 20 : 20 20 20 20 20 20 20 20 20 20 20 20 1 21 : 21 21 21 21 21 21 21 21 21 21 21 21 1 22 : 22 22 22 22 22 22 22 22 22 22 22 22 1 23 : 23 23 23 23 23 23 23 23 23 23 23 23 1 24 : 24 24 24 24 24 24 24 24 24 24 24 24 1 28 : 28 28 28 28 28 28 28 28 28 28 28 28 1 29 : 29 29 29 29 29 29 29 29 29 29 29 29 1 30 : 30 30 30 30 30 30 30 30 30 30 30 30 1 36 : 36 36 36 36 36 36 36 36 36 36 36 36 1 37 : 37 37 37 37 37 37 37 37 37 37 37 37 1 38 : 38 38 38 38 38 38 38 38 38 38 38 38 1 39 : 39 39 39 39 39 39 39 39 39 39 39 39 1 40 : 40 40 40 40 40 40 40 40 40 40 40 40 1 41 : 41 41 41 41 41 41 41 41 41 41 41 41 1 42 : 42 42 42 42 42 42 42 42 42 42 42 42 1 43 : 43 43 43 43 43 43 43 43 43 43 43 43 1 44 : 44 44 44 44 44 44 44 44 44 44 44 44 1 52 : 52 52 52 52 52 52 52 52 52 52 52 52 1 53 : 53 53 53 53 53 53 53 53 53 53 53 53 1 54 : 54 54 54 54 54 54 54 54 54 54 54 54 1 55 : 55 55 55 55 55 55 55 55 55 55 55 55 1 56 : 56 56 56 56 56 56 56 56 56 56 56 56 1 57 : 57 57 57 57 57 57 57 57 57 57 57 57 1 58 : 58 58 58 58 58 58 58 58 58 58 58 58 1 59 : 59 59 59 59 59 59 59 59 59 59 59 59 1 60 : 60 60 60 60 60 60 60 60 60 60 60 60 1 45 : 45 45 45 45 45 45 45 45 45 45 45 45 1 46 : 46 46 46 46 46 46 46 46 46 46 46 46 1 47 : 47 47 47 47 47 47 47 47 47 47 47 47 1 61 : 61 61 61 61 61 61 61 61 61 61 61 61 1 62 : 62 62 62 62 62 62 62 62 62 62 62 62 1 63 : 63 63 63 63 63 63 63 63 63 63 63 63 1 48 : 48 48 48 48 48 48 48 48 48 48 48 48 1 31 : 31 31 31 31 31 31 31 31 31 31 31 31 1 25 : 25 25 25 25 25 25 25 25 25 25 25 25 1 26 : 26 26 26 26 26 26 26 26 26 26 26 26 1 27 : 27 27 27 27 27 27 27 27 27 27 27 27 1 49 : 49 49 49 49 49 49 49 49 49 49 49 49 1 32 : 32 32 32 32 32 32 32 32 32 32 32 32 1 50 : 50 50 50 50 50 50 50 50 50 50 50 50 1 33 : 33 33 33 33 33 33 33 33 33 33 33 33 1 51 : 51 51 51 51 51 51 51 51 51 51 51 51 1 34 : 34 34 34 34 34 34 34 34 34 34 34 34 1 35 : 35 35 35 35 35 35 35 35 35 35 35 35 1 (t_initf) Read in prof_inparm namelist from: drv_in 13 : 13 13 13 13 13 13 13 13 13 13 13 13 1 14 : 14 14 14 14 14 14 14 14 14 14 14 14 1 15 : 15 15 15 15 15 15 15 15 15 15 15 15 1 16 : 16 16 16 16 16 16 16 16 16 16 16 16 1 17 : 17 17 17 17 17 17 17 17 17 17 17 17 1 18 : 18 18 18 18 18 18 18 18 18 18 18 18 1 19 : 19 19 19 19 19 19 19 19 19 19 19 19 1 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 Memory block size conversion in bytes is 4090.01 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory dealloc in MB is 0.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 64 pes participating in computation ----------------------------------- TASK# NAME 0 nid00309 1 nid00309 2 nid00309 3 nid00309 4 nid00310 5 nid00310 6 nid00310 7 nid00310 8 nid00311 9 nid00311 10 nid00311 11 nid00311 12 nid00312 13 nid00312 14 nid00312 15 nid00312 16 nid00313 17 nid00313 18 nid00313 19 nid00313 20 nid00314 21 nid00314 22 nid00314 23 nid00314 24 nid00315 25 nid00315 26 nid00315 27 nid00315 28 nid00316 29 nid00316 30 nid00316 31 nid00316 32 nid00317 33 nid00317 34 nid00317 35 nid00317 36 nid00318 37 nid00318 38 nid00318 39 nid00318 40 nid00319 41 nid00319 42 nid00319 43 nid00319 44 nid00320 45 nid00320 46 nid00320 47 nid00320 48 nid00321 49 nid00321 50 nid00321 51 nid00321 52 nid00322 53 nid00322 54 nid00322 55 nid00322 56 nid00323 57 nid00323 58 nid00323 59 nid00323 60 nid00324 61 nid00324 62 nid00324 63 nid00324 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 8 MB memory alloc in MB is 8.00 Memory block size conversion in bytes is 4090.01 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4090.01 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 8 MB memory alloc in MB is 8.00 8 MB memory dealloc in MB is 0.00 Memory block size conversion in bytes is 4088.02 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 42 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 46 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 30 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 21 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 27 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 44 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 14 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 54 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 43 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 34 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 16 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 62 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 56 ENDRUN: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 38 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 28 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 26 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 8 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 45 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 23 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 5 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 61 GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 40 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 15 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 60 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 35 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 10 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 47 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 39 ENDRUN: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 18 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 29 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 63 GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 32 GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 11 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 33 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 9 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 31 GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 6 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 7 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 ENDRUN: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 50 GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 51 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc _pmii_daemon(SIGCHLD): [NID 00309] [c2-0c1s5n1] [Tue Jun 28 15:25:13 2011] PE 1 exit signal Aborted _pmii_daemon(SIGCHLD): [NID 00311] [c2-0c1s5n3] [Tue Jun 28 15:25:13 2011] PE 8 exit signal Aborted _pmii_daemon(SIGCHLD): [NID 00316] [c2-0c1s7n0] [Tue Jun 28 15:25:13 2011] PE 28 exit signal Aborted _pmii_daemon(SIGCHLD): [NID 00324] [c2-0c2s1n0] [Tue Jun 28 15:25:13 2011] PE 62 exit signal Aborted [NID 00309] 2011-06-28 15:25:13 Apid 5740223: initiated application termination _pmii_daemon(SIGCHLD): [NID 00310] [c2-0c1s5n2] [Tue Jun 28 15:25:13 2011] PE 4 exit signal Aborted _pmii_daemon(SIGCHLD): [NID 00317] [c2-0c1s7n1] [Tue Jun 28 15:25:13 2011] PE 34 exit signal Aborted ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 48 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 58 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 37 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 22 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 13 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 53 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 36 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 24 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 52 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 19 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 20 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 12 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 41 ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc _pmii_daemon(SIGCHLD): [NID 00318] [c2-0c1s7n2] [Tue Jun 28 15:25:13 2011] PE 38 exit signal Aborted ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 55 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 25 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 17 aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 59 _pmii_daemon(SIGCHLD): [NID 00314] [c2-0c1s6n2] [Tue Jun 28 15:25:13 2011] PE 20 exit signal Aborted _pmii_daemon(SIGCHLD): [NID 00312] [c2-0c1s6n0] [Tue Jun 28 15:25:13 2011] PE 13 exit signal Aborted ENDRUN: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 57 GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc _pmii_daemon(SIGCHLD): [NID 00322] [c2-0c2s0n2] [Tue Jun 28 15:25:13 2011] PE 53 exit signal Aborted _pmii_daemon(SIGCHLD): [NID 00323] [c2-0c2s0n3] [Tue Jun 28 15:25:13 2011] PE 58 exit signal Aborted aborting job: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 49 _pmii_daemon(SIGCHLD): [NID 00321] [c2-0c2s0n1] [Tue Jun 28 15:25:13 2011] PE 51 exit signal Aborted ENDRUN: GETFIL: FAILED to get NF2005AERAMIPO_f19_f19_02.cam2.i.2006-01-01-00000.nc _pmii_daemon(SIGCHLD): [NID 00319] [c2-0c1s7n3] [Tue Jun 28 15:25:13 2011] PE 43 exit signal Aborted _pmii_daemon(SIGCHLD): [NID 00313] [c2-0c1s6n1] [Tue Jun 28 15:25:13 2011] PE 17 exit signal Aborted _pmii_daemon(SIGCHLD): [NID 00315] [c2-0c1s6n3] [Tue Jun 28 15:25:14 2011] PE 25 exit signal Aborted Application 5740223 exit codes: 134 Application 5740223 exit signals: Killed Application 5740223 resources: utime 0, stime 0