[cluster1:64027] mca: base: components_open: Looking for btl components
[cluster1:64027] mca: base: components_open: opening btl components
[cluster1:64027] mca: base: components_open: found loaded component ofud
[cluster1:64027] mca: base: components_open: component ofud has no register function
[cluster1:64027] mca: base: components_open: component ofud open function successful
[cluster1:64027] mca: base: components_open: found loaded component openib
[cluster1:64027] mca: base: components_open: component openib has no register function
[cluster1:64027] mca: base: components_open: component openib open function successful
[cluster1:64027] mca: base: components_open: found loaded component self
[cluster1:64027] mca: base: components_open: component self has no register function
[cluster1:64027] mca: base: components_open: component self open function successful
[cluster1:64027] mca: base: components_open: found loaded component sm
[cluster1:64027] mca: base: components_open: component sm has no register function
[cluster1:64027] mca: base: components_open: component sm open function successful
[cluster1:64027] mca: base: components_open: found loaded component tcp
[cluster1:64027] mca: base: components_open: component tcp has no register function
[cluster1:64027] mca: base: components_open: component tcp open function successful
[cluster1:64027] select: initializing btl component ofud
[cluster1:64027] select: init of component ofud returned failure
[cluster1:64027] select: module ofud unloaded
[cluster1:64027] select: initializing btl component openib
[cluster1][[5934,1],0][btl_openib_ini.c:166:ompi_btl_openib_ini_query] Querying INI files for vendor 0x03ba, part ID 25418
[cluster1][[5934,1],0][btl_openib_ini.c:185:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon
[cluster1][[5934,1],0][btl_openib_ini.c:166:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0
[cluster1][[5934,1],0][btl_openib_ini.c:185:ompi_btl_openib_ini_query] Found corresponding INI values: default
[cluster1:64027] openib BTL: oob CPC available for use on mlx4_0:1
[cluster1:64027] openib BTL: xoob CPC only supported with XRC receive queues; skipped on mlx4_0:1
[cluster1:64027] openib BTL: rdmacm IP address not found on port
[cluster1:64027] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cluster1:64027] select: init of component openib returned success
[cluster1:64027] select: initializing btl component self
[cluster1:64027] select: init of component self returned success
[cluster1:64027] select: initializing btl component sm
[cluster1:64027] select: init of component sm returned success
[cluster1:64027] select: initializing btl component tcp
[cluster1:64027] select: init of component tcp returned success
[cluster2:02759] mca: base: components_open: Looking for btl components
[cluster2:02759] mca: base: components_open: opening btl components
[cluster2:02759] mca: base: components_open: found loaded component ofud
[cluster2:02759] mca: base: components_open: component ofud has no register function
[cluster2:02759] mca: base: components_open: component ofud open function successful
[cluster2:02759] mca: base: components_open: found loaded component openib
[cluster2:02759] mca: base: components_open: component openib has no register function
[cluster2:02759] mca: base: components_open: component openib open function successful
[cluster2:02759] mca: base: components_open: found loaded component self
[cluster2:02759] mca: base: components_open: component self has no register function
[cluster2:02759] mca: base: components_open: component self open function successful
[cluster2:02759] mca: base: components_open: found loaded component sm
[cluster2:02759] mca: base: components_open: component sm has no register function
[cluster2:02759] mca: base: components_open: component sm open function successful
[cluster2:02759] mca: base: components_open: found loaded component tcp
[cluster2:02759] mca: base: components_open: component tcp has no register function
[cluster2:02759] mca: base: components_open: component tcp open function successful
[cluster2:02759] select: initializing btl component ofud
[cluster2:02759] select: init of component ofud returned failure
[cluster2:02759] select: module ofud unloaded
[cluster2:02759] select: initializing btl component openib
[cluster2][[5934,1],1][btl_openib_ini.c:166:ompi_btl_openib_ini_query] Querying INI files for vendor 0x03ba, part ID 25418
[cluster2][[5934,1],1][btl_openib_ini.c:185:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon
[cluster2][[5934,1],1][btl_openib_ini.c:166:ompi_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0
[cluster2][[5934,1],1][btl_openib_ini.c:185:ompi_btl_openib_ini_query] Found corresponding INI values: default
[cluster2:02759] openib BTL: oob CPC available for use on mlx4_0:1
[cluster2:02759] openib BTL: xoob CPC only supported with XRC receive queues; skipped on mlx4_0:1
[cluster2:02759] openib BTL: rdmacm IP address not found on port
[cluster2:02759] openib BTL: rdmacm CPC unavailable for use on mlx4_0:1; skipped
[cluster2:02759] select: init of component openib returned success
[cluster2:02759] select: initializing btl component self
[cluster2:02759] select: init of component self returned success
[cluster2:02759] select: initializing btl component sm
[cluster2:02759] select: init of component sm returned success
[cluster2:02759] select: initializing btl component tcp
[cluster2:02759] select: init of component tcp returned success
[cluster1][[5934,1],0][btl_openib_ini.c:166:ompi_btl_openib_ini_query] Querying INI files for vendor 0x03ba, part ID 25418
[cluster1][[5934,1],0][btl_openib_ini.c:185:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon
[cluster2][[5934,1],1][btl_openib_ini.c:166:ompi_btl_openib_ini_query] Querying INI files for vendor 0x03ba, part ID 25418
[cluster2][[5934,1],1][btl_openib_ini.c:185:ompi_btl_openib_ini_query] Found corresponding INI values: Mellanox Hermon
# OSU MPI Latency Test v3.1.1
# Size            Latency (us)
[cluster1:64027] *** Process received signal ***
[cluster1:64027] Signal: Bus error (10)
[cluster1:64027] Signal code: Invalid address alignment (1)
[cluster1:64027] Failing at address: 0xaa9053
[cluster1:64027] [ 0] /usr/mpi/gcc/openmpi-1.4.3/lib64/openmpi/mca_pml_ob1.so(+0x62f0) [0xfffff8010209e2f0]
[cluster1:64027] [ 1] /usr/mpi/gcc/openmpi-1.4.3/lib64/openmpi/mca_coll_tuned.so(+0x2904) [0xfffff801031ce904]
[cluster1:64027] [ 2] /usr/mpi/gcc/openmpi-1.4.3/lib64/openmpi/mca_coll_tuned.so(+0xb498) [0xfffff801031d7498]
[cluster1:64027] [ 3] /usr/mpi/gcc/openmpi-1.4.3/lib64/libmpi.so.0(MPI_Barrier+0xbc) [0xfffff8010005a97c]
[cluster1:64027] [ 4] /usr/mpi/gcc/openmpi-1.4.3/tests/osu_benchmarks-3.1.1/osu_latency(main+0x2b0) [0x100f34]
[cluster1:64027] [ 5] /lib64/libc.so.6(__libc_start_main+0x100) [0xfffff80100ac1240]
[cluster1:64027] [ 6] /usr/mpi/gcc/openmpi-1.4.3/tests/osu_benchmarks-3.1.1/osu_latency(_start+0x2c) [0x100bac]
[cluster1:64027] *** End of error message ***
[cluster2:02759] *** Process received signal ***
[cluster2:02759] Signal: Bus error (10)
[cluster2:02759] Signal code: Invalid address alignment (1)
[cluster2:02759] Failing at address: 0xaa9053
[cluster2:02759] [ 0] /usr/mpi/gcc/openmpi-1.4.3/lib64/openmpi/mca_pml_ob1.so(+0x62f0) [0xfffff8010209e2f0]
[cluster2:02759] [ 1] /usr/mpi/gcc/openmpi-1.4.3/lib64/openmpi/mca_coll_tuned.so(+0x2904) [0xfffff801031ce904]
[cluster2:02759] [ 2] /usr/mpi/gcc/openmpi-1.4.3/lib64/openmpi/mca_coll_tuned.so(+0xb498) [0xfffff801031d7498]
[cluster2:02759] [ 3] /usr/mpi/gcc/openmpi-1.4.3/lib64/libmpi.so.0(MPI_Barrier+0xbc) [0xfffff8010005a97c]
[cluster2:02759] [ 4] /usr/mpi/gcc/openmpi-1.4.3/tests/osu_benchmarks-3.1.1/osu_latency(main+0x2b0) [0x100f34]
[cluster2:02759] [ 5] /lib64/libc.so.6(__libc_start_main+0x100) [0xfffff80100ac1240]
[cluster2:02759] [ 6] /usr/mpi/gcc/openmpi-1.4.3/tests/osu_benchmarks-3.1.1/osu_latency(_start+0x2c) [0x100bac]
[cluster2:02759] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 64027 on node cluster1 exited on signal 10 (Bus error).
--------------------------------------------------------------------------