Invalid PIO rearranger comm max pend req (comp2io), 0 Resetting PIO rearranger comm max pend req (comp2io) to 64 PIO rearranger options: comm type = p2p comm fcd = 2denable max pend req (comp2io) = 0 enable_hs (comp2io) = T enable_isend (comp2io) = F max pend req (io2comp) = 64 enable_hs (io2comp) = F enable_isend (io2comp) = T (seq_comm_setcomm) init ID ( 1 GLOBAL ) pelist = 0 1023 1 ( npes = 1024) ( nthreads = 1)( suffix =) (seq_comm_setcomm) init ID ( 2 CPL ) pelist = 0 895 1 ( npes = 896) ( nthreads = 1)( suffix =) (seq_comm_setcomm) init ID ( 5 ATM ) pelist = 0 895 1 ( npes = 896) ( nthreads = 1)( suffix =) (seq_comm_joincomm) init ID ( 6 CPLATM ) join IDs = 2 5 ( npes = 896) ( nthreads = 1) (seq_comm_jcommarr) init ID ( 3 ALLATMID ) join multiple comp IDs ( npes = 896) ( nthreads = 1) (seq_comm_joincomm) init ID ( 4 CPLALLATMID ) join IDs = 2 3 ( npes = 896) ( nthreads = 1) (seq_comm_setcomm) init ID ( 9 LND ) pelist = 0 255 1 ( npes = 256) ( nthreads = 1)( suffix =) (seq_comm_joincomm) init ID ( 10 CPLLND ) join IDs = 2 9 ( npes = 896) ( nthreads = 1) (seq_comm_jcommarr) init ID ( 7 ALLLNDID ) join multiple comp IDs ( npes = 256) ( nthreads = 1) (seq_comm_joincomm) init ID ( 8 CPLALLLNDID ) join IDs = 2 7 ( npes = 896) ( nthreads = 1) (seq_comm_setcomm) init ID ( 13 ICE ) pelist = 256 511 1 ( npes = 256) ( nthreads = 1)( suffix =) (seq_comm_joincomm) init ID ( 14 CPLICE ) join IDs = 2 13 ( npes = 896) ( nthreads = 1) (seq_comm_jcommarr) init ID ( 11 ALLICEID ) join multiple comp IDs ( npes = 256) ( nthreads = 1) (seq_comm_joincomm) init ID ( 12 CPLALLICEID ) join IDs = 2 11 ( npes = 896) ( nthreads = 1) (seq_comm_setcomm) init ID ( 17 OCN ) pelist = 896 1023 1 ( npes = 128) ( nthreads = 1)( suffix =) (seq_comm_joincomm) init ID ( 18 CPLOCN ) join IDs = 2 17 ( npes = 1024) ( nthreads = 1) (seq_comm_jcommarr) init ID ( 15 ALLOCNID ) join multiple comp IDs ( npes = 128) ( nthreads = 1) (seq_comm_joincomm) init ID ( 16 CPLALLOCNID ) join IDs = 2 15 ( npes = 1024) ( nthreads = 1) (seq_comm_setcomm) init ID ( 21 ROF ) pelist = 0 255 1 ( npes = 256) ( nthreads = 1)( suffix =) (seq_comm_joincomm) init ID ( 22 CPLROF ) join IDs = 2 21 ( npes = 896) ( nthreads = 1) (seq_comm_jcommarr) init ID ( 19 ALLROFID ) join multiple comp IDs ( npes = 256) ( nthreads = 1) (seq_comm_joincomm) init ID ( 20 CPLALLROFID ) join IDs = 2 19 ( npes = 896) ( nthreads = 1) (seq_comm_setcomm) init ID ( 25 GLC ) pelist = 0 31 1 ( npes = 32) ( nthreads = 1)( suffix =) (seq_comm_joincomm) init ID ( 26 CPLGLC ) join IDs = 2 25 ( npes = 896) ( nthreads = 1) (seq_comm_jcommarr) init ID ( 23 ALLGLCID ) join multiple comp IDs ( npes = 32) ( nthreads = 1) (seq_comm_joincomm) init ID ( 24 CPLALLGLCID ) join IDs = 2 23 ( npes = 896) ( nthreads = 1) (seq_comm_setcomm) init ID ( 29 WAV ) pelist = 0 31 1 ( npes = 32) ( nthreads = 1)( suffix =) (seq_comm_joincomm) init ID ( 30 CPLWAV ) join IDs = 2 29 ( npes = 896) ( nthreads = 1) (seq_comm_jcommarr) init ID ( 27 ALLWAVID ) join multiple comp IDs ( npes = 32) ( nthreads = 1) (seq_comm_joincomm) init ID ( 28 CPLALLWAVID ) join IDs = 2 27 ( npes = 896) ( nthreads = 1) (seq_comm_setcomm) init ID ( 33 ESP ) pelist = 0 0 1 ( npes = 1) ( nthreads = 1)( suffix =) (seq_comm_joincomm) init ID ( 34 CPLESP ) join IDs = 2 33 ( npes = 896) ( nthreads = 1) (seq_comm_jcommarr) init ID ( 31 ALLESPID ) join multiple comp IDs ( npes = 1) ( nthreads = 1) (seq_comm_joincomm) init ID ( 32 CPLALLESPID ) join IDs = 2 31 ( npes = 896) ( nthreads = 1) (seq_comm_printcomms) 1 0 1024 1 GLOBAL: (seq_comm_printcomms) 2 0 896 1 CPL: (seq_comm_printcomms) 3 0 896 1 ALLATMID: (seq_comm_printcomms) 4 0 896 1 CPLALLATMID: (seq_comm_printcomms) 5 0 896 1 ATM: (seq_comm_printcomms) 6 0 896 1 CPLATM: (seq_comm_printcomms) 7 0 256 1 ALLLNDID: (seq_comm_printcomms) 8 0 896 1 CPLALLLNDID: (seq_comm_printcomms) 9 0 256 1 LND: (seq_comm_printcomms) 10 0 896 1 CPLLND: (seq_comm_printcomms) 11 256 -1 1 ALLICEID: (seq_comm_printcomms) 12 0 896 1 CPLALLICEID: (seq_comm_printcomms) 13 256 -1 1 ICE: (seq_comm_printcomms) 14 0 896 1 CPLICE: (seq_comm_printcomms) 15 896 -1 1 ALLOCNID: (seq_comm_printcomms) 16 0 1024 1 CPLALLOCNID: (seq_comm_printcomms) 17 896 -1 1 OCN: (seq_comm_printcomms) 18 0 1024 1 CPLOCN: (seq_comm_printcomms) 19 0 256 1 ALLROFID: (seq_comm_printcomms) 20 0 896 1 CPLALLROFID: (seq_comm_printcomms) 21 0 256 1 ROF: (seq_comm_printcomms) 22 0 896 1 CPLROF: (seq_comm_printcomms) 23 0 32 1 ALLGLCID: (seq_comm_printcomms) 24 0 896 1 CPLALLGLCID: (seq_comm_printcomms) 25 0 32 1 GLC: (seq_comm_printcomms) 26 0 896 1 CPLGLC: (seq_comm_printcomms) 27 0 32 1 ALLWAVID: (seq_comm_printcomms) 28 0 896 1 CPLALLWAVID: (seq_comm_printcomms) 29 0 32 1 WAV: (seq_comm_printcomms) 30 0 896 1 CPLWAV: (seq_comm_printcomms) 31 0 1 1 ALLESPID: (seq_comm_printcomms) 32 0 896 1 CPLALLESPID: (seq_comm_printcomms) 33 0 1 1 ESP: (seq_comm_printcomms) 34 0 896 1 CPLESP: Setting mpi info: romio_ds_read=disable Setting mpi info: romio_ds_write=disable OCN : pio_numiotasks = 4 OCN : pio_stride = 32 OCN : pio_root = 1 OCN : pio_iotype = 5 OCN : pio_numiotasks = 4 OCN : pio_stride = 32 OCN : pio_rearranger = 1 OCN : pio_root = 1 OCN : pio_iotype = 5 Setting mpi info: romio_ds_read=disable Setting mpi info: romio_ds_write=disable Setting mpi info: romio_ds_read=disable Setting mpi info: romio_ds_write=disable Setting mpi info: romio_ds_read=disable Setting mpi info: romio_ds_write=disable Setting mpi info: romio_ds_read=disable Setting mpi info: romio_ds_write=disable Setting mpi info: romio_ds_read=disable Setting mpi info: romio_ds_write=disable Setting mpi info: romio_ds_read=disable Setting mpi info: romio_ds_write=disable Setting mpi info: romio_ds_read=disable Setting mpi info: romio_ds_write=disable (t_initf) Read in prof_inparm namelist from: drv_in (t_initf) Using profile_disable= F (t_initf) profile_timer= 4 (t_initf) profile_depth_limit= 4 (t_initf) profile_detail_limit= 2 (t_initf) profile_barrier= F (t_initf) profile_outpe_num= 1 (t_initf) profile_outpe_stride= 0 (t_initf) profile_single_file= F (t_initf) profile_global_stats= T (t_initf) profile_ovhd_measurement= F (t_initf) profile_add_detail= F (t_initf) profile_papi_enable= F ICE : pio_numiotasks = 8 ICE : pio_stride = 32 ICE : pio_root = 1 ICE : pio_iotype = 5 ICE : pio_numiotasks = 8 ICE : pio_stride = 32 ICE : pio_rearranger = 1 ICE : pio_root = 1 ICE : pio_iotype = 5 Setting mpi info: romio_ds_read=disable Setting mpi info: romio_ds_write=disable 896 pes participating in computation ----------------------------------- TASK# NAME 0 c49-3.cluster 1 c49-3.cluster 2 c49-3.cluster 3 c49-3.cluster 4 c49-3.cluster 5 c49-3.cluster 6 c49-3.cluster 7 c49-3.cluster 8 c49-3.cluster 9 c49-3.cluster 10 c49-3.cluster 11 c49-3.cluster 12 c49-3.cluster 13 c49-3.cluster 14 c49-3.cluster 15 c49-3.cluster 16 c49-3.cluster 17 c49-3.cluster 18 c49-3.cluster 19 c49-3.cluster 20 c49-3.cluster 21 c49-3.cluster 22 c49-3.cluster 23 c49-3.cluster 24 c49-3.cluster 25 c49-3.cluster 26 c49-3.cluster 27 c49-3.cluster 28 c49-3.cluster 29 c49-3.cluster 30 c49-3.cluster 31 c49-3.cluster 32 c49-4.cluster 33 c49-4.cluster 34 c49-4.cluster 35 c49-4.cluster 36 c49-4.cluster 37 c49-4.cluster 38 c49-4.cluster 39 c49-4.cluster 40 c49-4.cluster 41 c49-4.cluster 42 c49-4.cluster 43 c49-4.cluster 44 c49-4.cluster 45 c49-4.cluster 46 c49-4.cluster 47 c49-4.cluster 48 c49-4.cluster 49 c49-4.cluster 50 c49-4.cluster 51 c49-4.cluster 52 c49-4.cluster 53 c49-4.cluster 54 c49-4.cluster 55 c49-4.cluster 56 c49-4.cluster 57 c49-4.cluster 58 c49-4.cluster 59 c49-4.cluster 60 c49-4.cluster 61 c49-4.cluster 62 c49-4.cluster 63 c49-4.cluster 64 c49-5.cluster 65 c49-5.cluster 66 c49-5.cluster 67 c49-5.cluster 68 c49-5.cluster 69 c49-5.cluster 70 c49-5.cluster 71 c49-5.cluster 72 c49-5.cluster 73 c49-5.cluster 74 c49-5.cluster 75 c49-5.cluster 76 c49-5.cluster 77 c49-5.cluster 78 c49-5.cluster 79 c49-5.cluster 80 c49-5.cluster 81 c49-5.cluster 82 c49-5.cluster 83 c49-5.cluster 84 c49-5.cluster 85 c49-5.cluster 86 c49-5.cluster 87 c49-5.cluster 88 c49-5.cluster 89 c49-5.cluster 90 c49-5.cluster 91 c49-5.cluster 92 c49-5.cluster 93 c49-5.cluster 94 c49-5.cluster 95 c49-5.cluster 96 c49-6.cluster 97 c49-6.cluster 98 c49-6.cluster 99 c49-6.cluster 100 c49-6.cluster 101 c49-6.cluster 102 c49-6.cluster 103 c49-6.cluster 104 c49-6.cluster 105 c49-6.cluster 106 c49-6.cluster 107 c49-6.cluster 108 c49-6.cluster 109 c49-6.cluster 110 c49-6.cluster 111 c49-6.cluster 112 c49-6.cluster 113 c49-6.cluster 114 c49-6.cluster 115 c49-6.cluster 116 c49-6.cluster 117 c49-6.cluster 118 c49-6.cluster 119 c49-6.cluster 120 c49-6.cluster 121 c49-6.cluster 122 c49-6.cluster 123 c49-6.cluster 124 c49-6.cluster 125 c49-6.cluster 126 c49-6.cluster 127 c49-6.cluster 128 c49-7.cluster 129 c49-7.cluster 130 c49-7.cluster 131 c49-7.cluster 132 c49-7.cluster 133 c49-7.cluster 134 c49-7.cluster 135 c49-7.cluster 136 c49-7.cluster 137 c49-7.cluster 138 c49-7.cluster 139 c49-7.cluster 140 c49-7.cluster 141 c49-7.cluster 142 c49-7.cluster 143 c49-7.cluster 144 c49-7.cluster 145 c49-7.cluster 146 c49-7.cluster 147 c49-7.cluster 148 c49-7.cluster 149 c49-7.cluster 150 c49-7.cluster 151 c49-7.cluster 152 c49-7.cluster 153 c49-7.cluster 154 c49-7.cluster 155 c49-7.cluster 156 c49-7.cluster 157 c49-7.cluster 158 c49-7.cluster 159 c49-7.cluster 160 c49-8.cluster 161 c49-8.cluster 162 c49-8.cluster 163 c49-8.cluster 164 c49-8.cluster 165 c49-8.cluster 166 c49-8.cluster 167 c49-8.cluster 168 c49-8.cluster 169 c49-8.cluster 170 c49-8.cluster 171 c49-8.cluster 172 c49-8.cluster 173 c49-8.cluster 174 c49-8.cluster 175 c49-8.cluster 176 c49-8.cluster 177 c49-8.cluster 178 c49-8.cluster 179 c49-8.cluster 180 c49-8.cluster 181 c49-8.cluster 182 c49-8.cluster 183 c49-8.cluster 184 c49-8.cluster 185 c49-8.cluster 186 c49-8.cluster 187 c49-8.cluster 188 c49-8.cluster 189 c49-8.cluster 190 c49-8.cluster 191 c49-8.cluster 192 c49-9.cluster 193 c49-9.cluster 194 c49-9.cluster 195 c49-9.cluster 196 c49-9.cluster 197 c49-9.cluster 198 c49-9.cluster 199 c49-9.cluster 200 c49-9.cluster 201 c49-9.cluster 202 c49-9.cluster 203 c49-9.cluster 204 c49-9.cluster 205 c49-9.cluster 206 c49-9.cluster 207 c49-9.cluster 208 c49-9.cluster 209 c49-9.cluster 210 c49-9.cluster 211 c49-9.cluster 212 c49-9.cluster 213 c49-9.cluster 214 c49-9.cluster 215 c49-9.cluster 216 c49-9.cluster 217 c49-9.cluster 218 c49-9.cluster 219 c49-9.cluster 220 c49-9.cluster 221 c49-9.cluster 222 c49-9.cluster 223 c49-9.cluster 224 c49-10.cluster 225 c49-10.cluster 226 c49-10.cluster 227 c49-10.cluster 228 c49-10.cluster 229 c49-10.cluster 230 c49-10.cluster 231 c49-10.cluster 232 c49-10.cluster 233 c49-10.cluster 234 c49-10.cluster 235 c49-10.cluster 236 c49-10.cluster 237 c49-10.cluster 238 c49-10.cluster 239 c49-10.cluster 240 c49-10.cluster 241 c49-10.cluster 242 c49-10.cluster 243 c49-10.cluster 244 c49-10.cluster 245 c49-10.cluster 246 c49-10.cluster 247 c49-10.cluster 248 c49-10.cluster 249 c49-10.cluster 250 c49-10.cluster 251 c49-10.cluster 252 c49-10.cluster 253 c49-10.cluster 254 c49-10.cluster 255 c49-10.cluster 256 c49-11.cluster ... list truncated at 256 Opened existing file bwma1850.f09.test1.cam.r.0023-01-01-00000.nc 0 Opened existing file /cluster/work/users/hanbre/inputdata/atm/cam/topo/fv_0.9x1.25_nc3000_Nsw042_Nrs 008_Co060_Fi001_ZR_sgh30_24km_GRNL_c170103.nc 1 srun: Job step aborted: Waiting up to 32 seconds for job step to finish. slurmstepd: error: *** STEP 235101.0 ON c49-3 CANCELLED AT 2018-12-27T17:45:13 ***