1. The source codes

"xthi-omp.c" is a pure OpenMP code to show thread affinity.
"xthi-nested-omp.c" is a nested OpenMP code to show thread affinity.
"xthi.c" is a hybrid MPI/OpenMP code to show process and thread affinity.

This README will show how to compile and run the pure OpenMP codes
(xthi-omp.c, and xthi-nested-omp.c) and hybird MPI/OpenMP code (xthi.c)
on NERSC Cori XC40 system Haswell and KNL compute nodes, NCSA BlueWaters 
XC30 system AMD compute nodes, and TACC Stampede2 KNL compute nodes.

Users are encouraged to explore and understand the impact of different 
runtime options to the process and thread affinity. For example:
Choices for OMP_PROC_BIND are close, master, true, false.
Choices for OMP_PLACES are : cores, sockets, and various ways to specify 
explicit lists, etc.

2. Use Cori Intel Haswell compute nodes

2.1 Compile 
To compile, from a Haswell login node. The default is Intel compiler.
% cc -qopenmp -o xthi-omp xthi-omp.c
% cc -qopenmp -o xthi-nested-omp xthi-nested-omp.c
% cc -qopenmp -o xthi xthi.c

To compile with GNU compiler:
% module swap PrgEnv-intel PrgEnv-gnu
% cc -fopenmp -o xthi-omp xthi-omp.c
% cc -fopenmp -o xthi-nested-omp xthi-nested-omp.c
% cc -fopenmp -o xthi xthi.c


To compile with Cray compiler:
% module swap PrgEnv-intel PrgEnv-cray
% cc -o xthi-omp xthi-omp.c
% cc -o xthi-nested-omp xthi-nested-omp.c
% cc -o xthi xthi.c

2.2 Request a compute node via interactive batch:

% salloc -N 1 -C haswell -p debug -t 30:00
or, if there is a reservation, such as "petascale": 
% salloc -N 1 -C haswell -p regular --reservation=petascale -t 1:00:00
...
You will then get on a compute node.

2.3 Check NUMA and hardware configuration

nid00011% numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
node 0 size: 64430 MB
node 0 free: 63002 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 1 size: 64635 MB
node 1 free: 63395 MB
node distances:
node   0   1 
  0:  10  21 
  1:  21  10 

2.4 Run 

nid00011% export OMP_NUM_THREADS=8

No default binding:
nid00011% ./xthi-omp |sort -k6n
Hello from thread 0, on nid00011. (core affinity = 0-63)
Hello from thread 1, on nid00011. (core affinity = 0-63)
Hello from thread 2, on nid00011. (core affinity = 0-63)
Hello from thread 3, on nid00011. (core affinity = 0-63)
Hello from thread 4, on nid00011. (core affinity = 0-63)
Hello from thread 5, on nid00011. (core affinity = 0-63)
Hello from thread 6, on nid00011. (core affinity = 0-63)
Hello from thread 7, on nid00011. (core affinity = 0-63)

OMP_PROC_BIND being "spread" is useful. Let's also set OMP_PLACES=cores:
(threads can migrate within the core):
nid00011% export OMP_PROC_BIND=spread
nid00011% export OMP_PLACES=cores  
nid00011% ./xthi-omp |sort -k6n
Hello from thread 0, on nid00011. (core affinity = 0,32)
Hello from thread 1, on nid00011. (core affinity = 4,36)
Hello from thread 2, on nid00011. (core affinity = 8,40)
Hello from thread 3, on nid00011. (core affinity = 12,44)
Hello from thread 4, on nid00011. (core affinity = 16,48)
Hello from thread 5, on nid00011. (core affinity = 20,52)
Hello from thread 6, on nid00011. (core affinity = 24,56)
Hello from thread 7, on nid00011. (core affinity = 28,60)

Now try to set the OMP_PLACES=threads:
(each thread is bind to a logical core, one of the two hyperthreads of a single core)
nid00011% export OMP_PLACES=threads
nid00011% ./xthi-omp |sort -k6n
Hello from thread 0, on nid00011. (core affinity = 0)
Hello from thread 1, on nid00011. (core affinity = 4)
Hello from thread 2, on nid00011. (core affinity = 8)
Hello from thread 3, on nid00011. (core affinity = 12)
Hello from thread 4, on nid00011. (core affinity = 16)
Hello from thread 5, on nid00011. (core affinity = 20)
Hello from thread 6, on nid00011. (core affinity = 24)
Hello from thread 7, on nid00011. (core affinity = 28)

Let's also try OMP_PLACES=sockets:
(each thread can freely move within a socket, 4 threads will bind to first 
socket and 4 threads will bind to the second socket).
nid00011% export OMP_PLACES=sockets
nid00011%./xthi-omp |sort -k6n
Hello from thread 0, on nid00011. (core affinity = 0-15,32-47)
Hello from thread 1, on nid00011. (core affinity = 0-15,32-47)
Hello from thread 2, on nid00011. (core affinity = 0-15,32-47)
Hello from thread 3, on nid00011. (core affinity = 0-15,32-47)
Hello from thread 4, on nid00011. (core affinity = 16-31,48-63)
Hello from thread 5, on nid00011. (core affinity = 16-31,48-63)
Hello from thread 6, on nid00011. (core affinity = 16-31,48-63)
Hello from thread 7, on nid00011. (core affinity = 16-31,48-63)

If you give an explicit list of OMP_PLACES, it will bind to these places only, 
such as:
nid00011% export OMP_PLACES=0,1,2,3
nid00011% ./xthi-omp |sort -k6n
Hello from thread 0, on nid00011. (core affinity = 0)
Hello from thread 1, on nid00011. (core affinity = 0)
Hello from thread 2, on nid00011. (core affinity = 1)
Hello from thread 3, on nid00011. (core affinity = 1)
Hello from thread 4, on nid00011. (core affinity = 2)
Hello from thread 5, on nid00011. (core affinity = 2)
Hello from thread 6, on nid00011. (core affinity = 3)
Hello from thread 7, on nid00011. (core affinity = 3)


Tests with "xthi-nested":
nid00011% export OMP_NUM_THREADS=4,3

OMP_PROC_BIND set to "spread,close" is recommended: outer layer "spread", 
inner layer "close":
nid00011% export OMP_PROC_BIND=spread,close
nid00011% export OMP_PLACES=threads

But it only have one OpenMP level!
nid00011% ./xthi-nested-omp |sort -k4,6
Hello from level 1: thread level 1= 0, on nid00011. (core affinity = 0)
Hello from level 1: thread level 1= 1, on nid00011. (core affinity = 8)
Hello from level 1: thread level 1= 2, on nid00011. (core affinity = 16)
Hello from level 1: thread level 1= 3, on nid00011. (core affinity = 24)
Hello from level 2: thread level 1=  0, thread level 2= 0, on nid00011. (core affinity = 0)
Hello from level 2: thread level 1=  1, thread level 2= 0, on nid00011. (core affinity = 8)
Hello from level 2: thread level 1=  2, thread level 2= 0, on nid00011. (core affinity = 16)
Hello from level 2: thread level 1=  3, thread level 2= 0, on nid00011. (core affinity = 24)

Now set the following two run time environment variables, it gets 2 levels 
with desired thread affinity:
nid00011% export OMP_NESTED=true
nid00011% export OMP_MAX_ACTIVE_LEVELS=2

nid00011% ./xthi-nested-omp |sort -k4,6
Hello from level 1: thread level 1= 0, on nid00011. (core affinity = 0)
Hello from level 1: thread level 1= 1, on nid00011. (core affinity = 8)
Hello from level 1: thread level 1= 2, on nid00011. (core affinity = 16)
Hello from level 1: thread level 1= 3, on nid00011. (core affinity = 24)
Hello from level 2: thread level 1=  0, thread level 2= 0, on nid00011. (core affinity = 0)
Hello from level 2: thread level 1=  0, thread level 2= 1, on nid00011. (core affinity = 32)
Hello from level 2: thread level 1=  0, thread level 2= 2, on nid00011. (core affinity = 1)
Hello from level 2: thread level 1=  1, thread level 2= 0, on nid00011. (core affinity = 8)
Hello from level 2: thread level 1=  1, thread level 2= 1, on nid00011. (core affinity = 40)
Hello from level 2: thread level 1=  1, thread level 2= 2, on nid00011. (core affinity = 9)
Hello from level 2: thread level 1=  2, thread level 2= 0, on nid00011. (core affinity = 16)
Hello from level 2: thread level 1=  2, thread level 2= 1, on nid00011. (core affinity = 48)
Hello from level 2: thread level 1=  2, thread level 2= 2, on nid00011. (core affinity = 17)
Hello from level 2: thread level 1=  3, thread level 2= 0, on nid00011. (core affinity = 24)
Hello from level 2: thread level 1=  3, thread level 2= 1, on nid00011. (core affinity = 56)
Hello from level 2: thread level 1=  3, thread level 2= 2, on nid00011. (core affinity = 25)

See non-optimal thread affinity with OMP_PROC_BIND=close,close below:
nid00011% export OMP_PROC_BIND=close,close 
nid00011% ./xthi-nested-omp |sort -k4,6
Hello from level 1: thread level 1= 0, on nid00011. (core affinity = 0)
Hello from level 1: thread level 1= 1, on nid00011. (core affinity = 32)
Hello from level 1: thread level 1= 2, on nid00011. (core affinity = 1)
Hello from level 1: thread level 1= 3, on nid00011. (core affinity = 33)
Hello from level 2: thread level 1=  0, thread level 2= 0, on nid00011. (core affinity = 0)
Hello from level 2: thread level 1=  0, thread level 2= 1, on nid00011. (core affinity = 32)
Hello from level 2: thread level 1=  0, thread level 2= 2, on nid00011. (core affinity = 1)
Hello from level 2: thread level 1=  1, thread level 2= 0, on nid00011. (core affinity = 32)
Hello from level 2: thread level 1=  1, thread level 2= 1, on nid00011. (core affinity = 1)
Hello from level 2: thread level 1=  1, thread level 2= 2, on nid00011. (core affinity = 33)
Hello from level 2: thread level 1=  2, thread level 2= 0, on nid00011. (core affinity = 1)
Hello from level 2: thread level 1=  2, thread level 2= 1, on nid00011. (core affinity = 33)
Hello from level 2: thread level 1=  2, thread level 2= 2, on nid00011. (core affinity = 2)
Hello from level 2: thread level 1=  3, thread level 2= 0, on nid00011. (core affinity = 33)
Hello from level 2: thread level 1=  3, thread level 2= 1, on nid00011. (core affinity = 2)
Hello from level 2: thread level 1=  3, thread level 2= 2, on nid00011. (core affinity = 34)

% export OMP_NUM_THREADS=4
% export OMP_PROC_BIND=spread
% export OMP_PLACES=threads
% srun -n 4 ./xthi |sort -k4n,6n
Hello from rank 0, thread 0, on nid00597. (core affinity = 0)
Hello from rank 0, thread 1, on nid00597. (core affinity = 8)
Hello from rank 0, thread 2, on nid00597. (core affinity = 16)
Hello from rank 0, thread 3, on nid00597. (core affinity = 24)
Hello from rank 1, thread 0, on nid00597. (core affinity = 0)
Hello from rank 1, thread 1, on nid00597. (core affinity = 8)
Hello from rank 1, thread 2, on nid00597. (core affinity = 16)
Hello from rank 1, thread 3, on nid00597. (core affinity = 24)
Hello from rank 2, thread 0, on nid00597. (core affinity = 0)
Hello from rank 2, thread 1, on nid00597. (core affinity = 8)
Hello from rank 2, thread 2, on nid00597. (core affinity = 16)
Hello from rank 2, thread 3, on nid00597. (core affinity = 24)
Hello from rank 3, thread 0, on nid00597. (core affinity = 0)
Hello from rank 3, thread 1, on nid00597. (core affinity = 8)
Hello from rank 3, thread 2, on nid00597. (core affinity = 16)
Hello from rank 3, thread 3, on nid00597. (core affinity = 24)
% srun -n 4 -c 16 --cpu_bind=cores ./xthi |sort -k4n,6n
Hello from rank 0, thread 0, on nid00597. (core affinity = 0)
Hello from rank 0, thread 1, on nid00597. (core affinity = 2)
Hello from rank 0, thread 2, on nid00597. (core affinity = 4)
Hello from rank 0, thread 3, on nid00597. (core affinity = 6)
Hello from rank 1, thread 0, on nid00597. (core affinity = 16)
Hello from rank 1, thread 1, on nid00597. (core affinity = 18)
Hello from rank 1, thread 2, on nid00597. (core affinity = 20)
Hello from rank 1, thread 3, on nid00597. (core affinity = 22)
Hello from rank 2, thread 0, on nid00597. (core affinity = 8)
Hello from rank 2, thread 1, on nid00597. (core affinity = 10)
Hello from rank 2, thread 2, on nid00597. (core affinity = 12)
Hello from rank 2, thread 3, on nid00597. (core affinity = 14)
Hello from rank 3, thread 0, on nid00597. (core affinity = 24)
Hello from rank 3, thread 1, on nid00597. (core affinity = 26)
Hello from rank 3, thread 2, on nid00597. (core affinity = 28)
Hello from rank 3, thread 3, on nid00597. (core affinity = 30)


nid00011% exit

3. Use BlueWaters AMD compute nodes

3.1 Compile

To compile, from a login node. The default is Cray compiler.
% cc -o xthi-omp xthi-omp.c
% cc -o xthi-nested-omp xthi-nested-omp.c
% cc -o xthi xthi.c

To compile with PGI compiler:
% module swap PrgEnv-cray PrgEnv-pgi
% cc -mp=nonuma -o xthi-omp xthi-omp.c
% cc -mp=nonuma -o xthi-nested-omp xthi-nested-omp.c
% cc -mp=nonuma -o xthi xthi.c

To compile with GNU compiler:
% module swap PrgEnv-cray PrgEnv-gnu
% cc -fopenmp -o xthi-omp xthi-omp.c
% cc -fopenmp -o xthi-nested-omp xthi-nested-omp.c
% cc -fopenmp -o xthi xthi.c

To compile with Intel compiler:
% module swap PrgEnv-cray PrgEnv-intel
% cc -qopenmp o xthi-omp xthi-omp.c
% cc -qopenmp o xthi-nested-omp xthi-nested-omp.c
% cc -qopenmp o xthi xthi.c

3.2 Request a compute node via interactive batch:

% qsub -I -l nodes=1:ppn=32 -q high -l walltime=1:00:00
Job submitted to account: bako
qsub: waiting for job 6963723.bw to start
<you will get on a service node, or so called a MOM node>
% 


3.3 Check NUMA and hardware configuration
Continue in the above interactive batch session, from the service node.

% aprun -n 1 numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 16383 MB
node 0 free: 15736 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 16384 MB
node 1 free: 14844 MB
node 2 cpus: 16 17 18 19 20 21 22 23
node 2 size: 16384 MB
node 2 free: 15391 MB
node 3 cpus: 24 25 26 27 28 29 30 31
node 3 size: 16384 MB
node 3 free: 15550 MB
node distances:
node   0   1   2   3 
  0:  10  13  13  13 
  1:  13  10  13  13 
  2:  13  13  10  13 
  3:  13  13  13  10 
Application 57100600 resources: utime ~0s, stime ~1s, Rss ~21624, inblocks ~51, outblocks ~56

Notice: you did not do "numactl -H" directly, since you will get the info 
on the service node. Instead, use "aprun -n 1 numactl -H" to run the numactl 
on a compute node.

3.4 Run

% cd $PBS_O_WORKDIR

Notice, aprun -n 1 (where "-n 1" means 1 MPI task) is needed to launch even a serial 
or a pure OpenMP job, otherwise the job will run on a service node. 

Also for pure OpenMP jobs, need to use -d xxx and -cc none options, where xxx is the 
number of threads. 

% export OMP_NUM_THREADS=8
% aprun -n 1 -d 8 -cc none ./xthi-omp |sort -k4
Hello from thread 0, on nid05859. (core affinity = 0)
Hello from thread 1, on nid05859. (core affinity = 1)
Hello from thread 2, on nid05859. (core affinity = 2)
Hello from thread 3, on nid05859. (core affinity = 3)
Hello from thread 4, on nid05859. (core affinity = 4)
Hello from thread 5, on nid05859. (core affinity = 5)
Hello from thread 6, on nid05859. (core affinity = 6)
Hello from thread 7, on nid05859. (core affinity = 7)

% export OMP_PROC_BIND=spread
% export OMP_PLACES=cores    
% aprun -N 1 -n 1 -d 8 -cc none ./xthi-omp |sort -k4
Hello from thread 0, on nid05859. (core affinity = 0)
Hello from thread 1, on nid05859. (core affinity = 4)
Hello from thread 2, on nid05859. (core affinity = 8)
Hello from thread 3, on nid05859. (core affinity = 12)
Hello from thread 4, on nid05859. (core affinity = 16)
Hello from thread 5, on nid05859. (core affinity = 20)
Hello from thread 6, on nid05859. (core affinity = 24)
Hello from thread 7, on nid05859. (core affinity = 28)

% export OMP_PROC_BIND=close 
% aprun -N 1 -n 1 -d 8 -cc none ./xthi-omp |sort -k4
Hello from thread 0, on nid05859. (core affinity = 0)
Hello from thread 1, on nid05859. (core affinity = 1)
Hello from thread 2, on nid05859. (core affinity = 2)
Hello from thread 3, on nid05859. (core affinity = 3)
Hello from thread 4, on nid05859. (core affinity = 4)
Hello from thread 5, on nid05859. (core affinity = 5)
Hello from thread 6, on nid05859. (core affinity = 6)
Hello from thread 7, on nid05859. (core affinity = 7)

% export OMP_NUM_THREADS=4,3
% export OMP_NESTED=true
% export OMP_MAX_ACTIVE_LEVELS=2

% export OMP_PROC_BIND=spread,close
% aprun -N 1 -n 1 -d 12 -cc none ./xthi-nested-omp |sort -k4
Hello from level 1: thread level 1= 0, on nid05859. (core affinity = 0,1)
Hello from level 1: thread level 1= 1, on nid05859. (core affinity = 8,9)
Hello from level 1: thread level 1= 2, on nid05859. (core affinity = 16,17)
Hello from level 1: thread level 1= 3, on nid05859. (core affinity = 24,25)
Hello from level 2: thread level 1=  0, thread level 2= 0, on nid05859. (core affinity = 0,1)
Hello from level 2: thread level 1=  0, thread level 2= 1, on nid05859. (core affinity = 2,3)
Hello from level 2: thread level 1=  0, thread level 2= 2, on nid05859. (core affinity = 4,5)
Hello from level 2: thread level 1=  1, thread level 2= 0, on nid05859. (core affinity = 8,9)
Hello from level 2: thread level 1=  1, thread level 2= 1, on nid05859. (core affinity = 10,11)
Hello from level 2: thread level 1=  1, thread level 2= 2, on nid05859. (core affinity = 12,13)
Hello from level 2: thread level 1=  2, thread level 2= 0, on nid05859. (core affinity = 16,17)
Hello from level 2: thread level 1=  2, thread level 2= 1, on nid05859. (core affinity = 18,19)
Hello from level 2: thread level 1=  2, thread level 2= 2, on nid05859. (core affinity = 20,21)
Hello from level 2: thread level 1=  3, thread level 2= 0, on nid05859. (core affinity = 24,25)
Hello from level 2: thread level 1=  3, thread level 2= 1, on nid05859. (core affinity = 26,27)
Hello from level 2: thread level 1=  3, thread level 2= 2, on nid05859. (core affinity = 28,29)

% export OMP_PROC_BIND=close,close 
% aprun -N 1 -n 1 -d 12 -cc none ./xthi-nested-omp |sort -k4
Hello from level 1: thread level 1= 0, on nid05859. (core affinity = 0,1)
Hello from level 1: thread level 1= 1, on nid05859. (core affinity = 2,3)
Hello from level 1: thread level 1= 2, on nid05859. (core affinity = 4,5)
Hello from level 1: thread level 1= 3, on nid05859. (core affinity = 6,7)
Hello from level 2: thread level 1=  0, thread level 2= 0, on nid05859. (core affinity = 0,1)
Hello from level 2: thread level 1=  0, thread level 2= 1, on nid05859. (core affinity = 2,3)
Hello from level 2: thread level 1=  0, thread level 2= 2, on nid05859. (core affinity = 4,5)
Hello from level 2: thread level 1=  1, thread level 2= 0, on nid05859. (core affinity = 2,3)
Hello from level 2: thread level 1=  1, thread level 2= 1, on nid05859. (core affinity = 4,5)
Hello from level 2: thread level 1=  1, thread level 2= 2, on nid05859. (core affinity = 6,7)
Hello from level 2: thread level 1=  2, thread level 2= 0, on nid05859. (core affinity = 4,5)
Hello from level 2: thread level 1=  2, thread level 2= 1, on nid05859. (core affinity = 6,7)
Hello from level 2: thread level 1=  2, thread level 2= 2, on nid05859. (core affinity = 8,9)
Hello from level 2: thread level 1=  3, thread level 2= 0, on nid05859. (core affinity = 6,7)
Hello from level 2: thread level 1=  3, thread level 2= 1, on nid05859. (core affinity = 8,9)
Hello from level 2: thread level 1=  3, thread level 2= 2, on nid05859. (core affinity = 10,11)

% export OMP_NUM_THREADS=4
% aprun -n 4  -d 4 ./xthi |sort -k4n,6n
Application 57248556 resources: utime ~0s, stime ~1s, Rss ~9312, inblocks ~7687, outblocks ~18240
Hello from rank 0, thread 0, on nid19708. (core affinity = 0)
Hello from rank 0, thread 1, on nid19708. (core affinity = 1)
Hello from rank 0, thread 2, on nid19708. (core affinity = 2)
Hello from rank 0, thread 3, on nid19708. (core affinity = 3)
Hello from rank 1, thread 0, on nid19708. (core affinity = 4)
Hello from rank 1, thread 1, on nid19708. (core affinity = 5)
Hello from rank 1, thread 2, on nid19708. (core affinity = 6)
Hello from rank 1, thread 3, on nid19708. (core affinity = 7)
Hello from rank 2, thread 0, on nid19708. (core affinity = 8)
Hello from rank 2, thread 1, on nid19708. (core affinity = 9)
Hello from rank 2, thread 2, on nid19708. (core affinity = 10)
Hello from rank 2, thread 3, on nid19708. (core affinity = 11)
Hello from rank 3, thread 0, on nid19708. (core affinity = 12)
Hello from rank 3, thread 1, on nid19708. (core affinity = 13)
Hello from rank 3, thread 2, on nid19708. (core affinity = 14)
Hello from rank 3, thread 3, on nid19708. (core affinity = 15)

% aprun -n 4 -S 1 -d 4 ./xthi |sort -k4n,6n
Application 57248560 resources: utime ~0s, stime ~1s, Rss ~9312, inblocks ~7687, outblocks ~18240
Hello from rank 0, thread 0, on nid19708. (core affinity = 0)
Hello from rank 0, thread 1, on nid19708. (core affinity = 1)
Hello from rank 0, thread 2, on nid19708. (core affinity = 2)
Hello from rank 0, thread 3, on nid19708. (core affinity = 3)
Hello from rank 1, thread 0, on nid19708. (core affinity = 8)
Hello from rank 1, thread 1, on nid19708. (core affinity = 9)
Hello from rank 1, thread 2, on nid19708. (core affinity = 10)
Hello from rank 1, thread 3, on nid19708. (core affinity = 11)
Hello from rank 2, thread 0, on nid19708. (core affinity = 16)
Hello from rank 2, thread 1, on nid19708. (core affinity = 17)
Hello from rank 2, thread 2, on nid19708. (core affinity = 18)
Hello from rank 2, thread 3, on nid19708. (core affinity = 19)
Hello from rank 3, thread 0, on nid19708. (core affinity = 24)
Hello from rank 3, thread 1, on nid19708. (core affinity = 25)
Hello from rank 3, thread 2, on nid19708. (core affinity = 26)
Hello from rank 3, thread 3, on nid19708. (core affinity = 27)

% aprun -n 4 -S 1 -d 4 -cc depth ./xthi |sort -k4n,6n
Application 57248562 resources: utime ~0s, stime ~1s, Rss ~9312, inblocks ~7687, outblocks ~18240
Hello from rank 0, thread 0, on nid19708. (core affinity = 0-3)
Hello from rank 0, thread 1, on nid19708. (core affinity = 0-3)
Hello from rank 0, thread 2, on nid19708. (core affinity = 0-3)
Hello from rank 0, thread 3, on nid19708. (core affinity = 0-3)
Hello from rank 1, thread 0, on nid19708. (core affinity = 8-11)
Hello from rank 1, thread 1, on nid19708. (core affinity = 8-11)
Hello from rank 1, thread 2, on nid19708. (core affinity = 8-11)
Hello from rank 1, thread 3, on nid19708. (core affinity = 8-11)
Hello from rank 2, thread 0, on nid19708. (core affinity = 16-19)
Hello from rank 2, thread 1, on nid19708. (core affinity = 16-19)
Hello from rank 2, thread 2, on nid19708. (core affinity = 16-19)
Hello from rank 2, thread 3, on nid19708. (core affinity = 16-19)
Hello from rank 3, thread 0, on nid19708. (core affinity = 24-27)
Hello from rank 3, thread 1, on nid19708. (core affinity = 24-27)
Hello from rank 3, thread 2, on nid19708. (core affinity = 24-27)
Hello from rank 3, thread 3, on nid19708. (core affinity = 24-27)

% exit

4. Use Cori KNL compute nodes

4.1 Compile
You could only compile on a Haswell login node for KNL compute nodes.

To compile, from a Haswell login node:
Make sure to do the following step first:
% module swap craype-haswell craype-mic-knl
Then
% cc -qopenmp -o xthi-omp xthi-omp.c
% cc -qopenmp -o xthi-nested-omp xthi-nested-omp.c
% cc -qopenmp -o xthi xthi.c

4.2 Request a KNL compute node
% salloc -N 1 -C knl,quad,cache -p debug -t 30:00
(or % salloc -N 1 -C knl,quad,flat -p debug -t 30:00
or % salloc -N 1 -C knl,snc2,cache -p debug -t 30:00
or % salloc -N 1 -C knl,snc2,flat -p debug -t 30:00)
...
You will then get on a KNL compute node.

4.3 Check NUMA and hardware configuration

4.3.1 68-core Quad Cache node:
NUMA Domain 0: all 68 cores (272 logic cores)
% numactl -H
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271
node 0 size: 96762 MB
node 0 free: 93067 MB
node distances:
node 0
0: 10

quad,cache node has 1 NUMA domain1.  All CPUs on NUMA domains 0 (DDR memory), 
MCDRAM appears as cache only.

4.3.2 68-core Quad Flat node:
NUMA Domain 0: all 68 cores (272 logic cores)
NUMA Domain 1: HBM with no CPUs
% numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271
node 0 size: 96723 MB
node 0 free: 93924 MB
node 1 cpus:
node 1 size: 16157 MB
node 1 free: 16088 MB
node distances:
node 0 1
0: 10 31
1: 31 10

quad,flat node has 2 NUMA domains.  All CPUs on NUMA domains 0 (DDR memory). 
NUMA node 1 has MCDRAM only.

4.3.3 68-core SNC2 Flat node:
NUMA Domain 0: all 68 cores (272 logic cores)
NUMA Domain 1: HBM with no CPUs
% numactl -H
numactl -H
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237
node 0 size: 48293 MB
node 0 free: 46047 MB
node 1 cpus: 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271
node 1 size: 48466 MB
node 1 free: 44949 MB
node 2 cpus:
node 2 size: 8079 MB
node2free:7983MB
node 3 cpus:
node 3 size: 8077 MB
node 3 free: 7980 MB
node distances:
node   0   1   2   3 
  0:  10  21  31  41 
  1:  21  10  41  31 
  2:  31  41  10  41 
  3:  41  31  41  10 

snc2,flat node has 4 NUMA domains.  All CPUs on NUMA domains 0 and 1 (DDR memory), 
each has half of the CPUs. NUMA nodes 2 and 3 have MCDRAM only.


4.4 Run 
% export OMP_NUM_THREADS=8
% export OMP_PROC_BIND=spread
% export OMP_PLACES=cores
% ./xthi-omp |sort -k6n
Hello from thread 0, on nid02304. (core affinity = 0,68,136,204)
Hello from thread 1, on nid02304. (core affinity = 9,77,145,213)
Hello from thread 2, on nid02304. (core affinity = 17,85,153,221)
Hello from thread 3, on nid02304. (core affinity = 26,94,162,230)
Hello from thread 4, on nid02304. (core affinity = 34,102,170,238)
Hello from thread 5, on nid02304. (core affinity = 43,111,179,247)
Hello from thread 6, on nid02304. (core affinity = 51,119,187,255)
Hello from thread 7, on nid02304. (core affinity = 60,128,196,264)

% export OMP_PROC_BIND=close 
% ./xthi-omp |sort -k6n
Hello from thread 0, on nid02304. (core affinity = 0,68,136,204)
Hello from thread 1, on nid02304. (core affinity = 1,69,137,205)
Hello from thread 2, on nid02304. (core affinity = 2,70,138,206)
Hello from thread 3, on nid02304. (core affinity = 3,71,139,207)
Hello from thread 4, on nid02304. (core affinity = 4,72,140,208)
Hello from thread 5, on nid02304. (core affinity = 5,73,141,209)
Hello from thread 6, on nid02304. (core affinity = 6,74,142,210)
Hello from thread 7, on nid02304. (core affinity = 7,75,143,211)
% export OMP_PLACES=threads
% ./xthi-omp |sort -k6n
Hello from thread 0, on nid02304. (core affinity = 0)
Hello from thread 1, on nid02304. (core affinity = 68)
Hello from thread 2, on nid02304. (core affinity = 136)
Hello from thread 3, on nid02304. (core affinity = 204)
Hello from thread 4, on nid02304. (core affinity = 1)
Hello from thread 5, on nid02304. (core affinity = 69)
Hello from thread 6, on nid02304. (core affinity = 137)
Hello from thread 7, on nid02304. (core affinity = 205)
% export OMP_PROC_BIND=spread
% ./xthi-omp |sort -k6n
Hello from thread 0, on nid02304. (core affinity = 0)
Hello from thread 1, on nid02304. (core affinity = 144)
Hello from thread 2, on nid02304. (core affinity = 17)
Hello from thread 3, on nid02304. (core affinity = 161)
Hello from thread 4, on nid02304. (core affinity = 34)
Hello from thread 5, on nid02304. (core affinity = 178)
Hello from thread 6, on nid02304. (core affinity = 51)
Hello from thread 7, on nid02304. (core affinity = 195)

Now with the hybrid MPI/OpenMP "xthi.c" code:
% export OMP_NUM_THREADS=4
% export OMP_PROC_BIND=spread
% export OMP_PLACES=threads
% srun -n 4  ./xthi |sort -k4n,6n
Hello from rank 0, thread 0, on nid06509. (core affinity = 0)
Hello from rank 0, thread 1, on nid06509. (core affinity = 17)
Hello from rank 0, thread 2, on nid06509. (core affinity = 34)
Hello from rank 0, thread 3, on nid06509. (core affinity = 51)
Hello from rank 1, thread 0, on nid06509. (core affinity = 0)
Hello from rank 1, thread 1, on nid06509. (core affinity = 17)
Hello from rank 1, thread 2, on nid06509. (core affinity = 34)
Hello from rank 1, thread 3, on nid06509. (core affinity = 51)
Hello from rank 2, thread 0, on nid06509. (core affinity = 0)
Hello from rank 2, thread 1, on nid06509. (core affinity = 17)
Hello from rank 2, thread 2, on nid06509. (core affinity = 34)
Hello from rank 2, thread 3, on nid06509. (core affinity = 51)
Hello from rank 3, thread 0, on nid06509. (core affinity = 0)
Hello from rank 3, thread 1, on nid06509. (core affinity = 17)
Hello from rank 3, thread 2, on nid06509. (core affinity = 34)
Hello from rank 3, thread 3, on nid06509. (core affinity = 51)

The "-c xxx --cpu_bind=cores" flags are essential:
% srun -n 4 -c 64  --cpu_bind=cores ./xthi |sort -k4n,6n
Hello from rank 0, thread 0, on nid06509. (core affinity = 0)
Hello from rank 0, thread 1, on nid06509. (core affinity = 4)
Hello from rank 0, thread 2, on nid06509. (core affinity = 8)
Hello from rank 0, thread 3, on nid06509. (core affinity = 12)
Hello from rank 1, thread 0, on nid06509. (core affinity = 16)
Hello from rank 1, thread 1, on nid06509. (core affinity = 20)
Hello from rank 1, thread 2, on nid06509. (core affinity = 24)
Hello from rank 1, thread 3, on nid06509. (core affinity = 28)
Hello from rank 2, thread 0, on nid06509. (core affinity = 32)
Hello from rank 2, thread 1, on nid06509. (core affinity = 36)
Hello from rank 2, thread 2, on nid06509. (core affinity = 40)
Hello from rank 2, thread 3, on nid06509. (core affinity = 44)
Hello from rank 3, thread 0, on nid06509. (core affinity = 48)
Hello from rank 3, thread 1, on nid06509. (core affinity = 52)
Hello from rank 3, thread 2, on nid06509. (core affinity = 56)
Hello from rank 3, thread 3, on nid06509. (core affinity = 60)
% exit

Running on a KNL Quad Flat node is similar to running on a KNL Quad Cache node
on Cori. Except you will request a compute node with
% salloc -N 1 -C knl,quad,flat -p debug -t 30:00
And you will run your executable with numactl to use MCDRAM
% numactl -m 1 ./xthi-omp   (if use memory < 16 GB)
or
% numactl -p 1 ./xthi-omp    (if use memory > 16 GB)


And for SNC2 Flat mode, you will request a node with
% salloc -N 1 -C snc2,quad,flat -p debug -t 30:00
And you will run your executable with numactl to use MCDRAM:
% numactl -m 2,3 ./xthi-omp   (if use memory < 16 GB)
or
% numactl -p 2,3 ./xthi-omp   (if use memory < 16 GB)


5. Use TACC KNL compute nodes
% cd $WORK
% module list

Currently Loaded Modules:
  1) intel/17.0.4   2) impi/17.0.3   3) git/2.9.0   4) autotools/1.1   
  5) python/2.7.13   6) xalt/1.7   7) TACC

5.1 Compile

To compile, from a login node. 

$ icc -qopenmp -xCORE-AVX2 -o xthi-omp xthi-omp.c
$ icc -qopenmp -xCORE-AVX2 -o xthi-nested-omp xthi-nested-omp.c
$ mpiicc -qopenmp -xCORE-AVX2 -o xthi xthi.c

5.2 Request a compute node via interactive batch 
(it includes 5.3 Check NUMA and hardware configuration) 

login4.stampede2(47)$ idev

 -> Compute nodes in the development queue of the stampede2 system have HyperThreading turned on.
 -> Normal HPC Applications will use no more than 68 tasks per node (=cores/node).
 -> Few Apps use all HyperThreads [4 (HWthreads/Core) x 68 (Cores/Node) = 272 ).
 -> FOR IDEV USE: specify the default tasks/per node for this queue.(68 suggested)
Please type the tasks per node and hit return: 1
 -> 1 is the number of tasks per node selected for the development queue.


We need a project to charge for interactive use.
for a Single Project : We store "use_default"              in the $HOME/.idevrc file.
for Multiple Projects: We will store your selected project in the $HOME/.idevrc file.

 You have a single project which we will use as the default.
 -> Defaults file    : ~/.idevrc   
 -> System           : stampede2   
 -> Nodes            : 1             (idev  default       )
 -> Tasks per Node   : 1             (~/.idevrc           )
 -> Project          : (use_default  (SLURM default       )

-----------------------------------------------------------------
          Welcome to the Stampede 2 Supercomputer          
-----------------------------------------------------------------

No reservation for this job
--> Verifying valid submit host (login4)...OK
--> Verifying valid jobname...OK
--> Enforcing max jobs per user...OK
--> Verifying availability of your home dir (/home1/04174/yunhe)...OK
--> Verifying availability of your work dir (/work/04174/yunhe/stampede2)...OK
--> Verifying availability of your scratch dir (/scratch/04174/yunhe)...OK
--> Verifying valid ssh keys...OK
--> Verifying access to desired queue (development)...OK
--> Verifying job request is within current queue limits...OK
--> Checking available allocation (StampedeNERSC)...OK
Submitted batch job 52992

 -> After your idev job begins to run, a command prompt will appear,
 -> and you can begin your interactive development session. 
 -> We will report the job status every 4 seconds: (PD=pending, R=running).

 ->job status:  PD
 ->job status:  PD
 ->job status:  PD
 ->job status:  PD
 ->job status:  PD
 ->job status:  PD
 ->job status:  PD
 ->job status:  PD
 ->job status:  R

 -> Job is now running on masternode= c401-094...OK
 -> Sleeping for 7 seconds...OK
 -> Checking to make sure your job has initialized an env for you....OK
 -> Creating interactive terminal session (login) on master node c401-094.

TACC Stampede2 System
Provisioned on 08-May-2017 at 13:27
 
c401-094.stampede2(1)$ numactl -H
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271
node 0 size: 98199 MB
node 0 free: 90861 MB
node distances:
node   0 
  0:  10 

c401-094.stampede2(2)$ 
c401-094.stampede2(2)$ export OMP_NUM_THREADS=8
c401-094.stampede2(3)$ ./xthi-omp |sort -k6n
Hello
Hello from thread 0, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 0)
Hello from thread 1, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 1)
Hello from thread 2, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 2)
Hello from thread 3, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 3)
Hello from thread 4, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 4)
Hello from thread 5, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 5)
Hello from thread 6, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 6)
Hello from thread 7, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 7)
c401-094.stampede2(4)$ 
c401-094.stampede2(4)$ export OMP_PROC_BIND=spread
c401-094.stampede2(5)$ export OMP_PLACES=cores
c401-094.stampede2(6)$ ./xthi-omp |sort -k6n
Hello from thread 0, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 0,68,136,204)
Hello from thread 1, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 9,77,145,213)
Hello from thread 2, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 17,85,153,221)
Hello from thread 3, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 26,94,162,230)
Hello from thread 4, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 34,102,170,238)
Hello from thread 5, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 43,111,179,247)
Hello from thread 6, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 51,119,187,255)
Hello from thread 7, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 60,128,196,264)
c401-094.stampede2(7)$ 
c401-094.stampede2(7)$ export OMP_PLACES=threads
c401-094.stampede2(8)$ ./xthi-omp |sort -k6n
Hello from thread 0, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 0)
Hello from thread 1, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 144)
Hello from thread 2, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 17)
Hello from thread 3, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 161)
Hello from thread 4, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 34)
Hello from thread 5, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 178)
Hello from thread 6, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 51)
Hello from thread 7, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 195)
c401-094.stampede2(9)$ 
c401-094.stampede2(9)$ export OMP_PROC_BIND=close
c401-094.stampede2(10)$ ./xthi-omp |sort -k6n
Hello from thread 0, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 0)
Hello from thread 1, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 68)
Hello from thread 2, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 136)
Hello from thread 3, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 204)
Hello from thread 4, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 1)
Hello from thread 5, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 69)
Hello from thread 6, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 137)
Hello from thread 7, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 205)
c401-094.stampede2(11)$ 
c401-094.stampede2(13)$ export OMP_PLACES=cores
c401-094.stampede2(14)$ ./xthi-omp |sort -k6n
Hello from thread 0, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 0,68,136,204)
Hello from thread 1, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 1,69,137,205)
Hello from thread 2, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 2,70,138,206)
Hello from thread 3, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 3,71,139,207)
Hello from thread 4, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 4,72,140,208)
Hello from thread 5, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 5,73,141,209)
Hello from thread 6, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 6,74,142,210)
Hello from thread 7, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 7,75,143,211)
c401-094.stampede2(15)$ 


Nested OpenMP:
c401-094.stampede2(15)$ 
c401-094.stampede2(15)$ export OMP_NUM_THREADS=4,3
c401-094.stampede2(16)$ export OMP_NESTED=true
c401-094.stampede2(17)$ export OMP_MAX_ACTIVE_LEVELS=2
c401-094.stampede2(18)$ 
c401-094.stampede2(18)$ 
c401-094.stampede2(18)$ export OMP_PROC_BIND=spread,close
c401-094.stampede2(19)$ export OMP_PLACES=cores
c401-094.stampede2(20)$ ./xthi-nested-omp |sort -k6n
Hello from level 1: thread level 1= 0, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 0,68,136,204)
Hello from level 1: thread level 1= 1, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 17,85,153,221)
Hello from level 1: thread level 1= 2, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 34,102,170,238)
Hello from level 1: thread level 1= 3, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 51,119,187,255)
Hello from level 2: thread level 1=  0, thread level 2= 0, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 0,68,136,204)
Hello from level 2: thread level 1=  0, thread level 2= 1, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 1,69,137,205)
Hello from level 2: thread level 1=  0, thread level 2= 2, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 2,70,138,206)
Hello from level 2: thread level 1=  1, thread level 2= 0, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 17,85,153,221)
Hello from level 2: thread level 1=  1, thread level 2= 1, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 18,86,154,222)
Hello from level 2: thread level 1=  1, thread level 2= 2, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 19,87,155,223)
Hello from level 2: thread level 1=  2, thread level 2= 0, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 34,102,170,238)
Hello from level 2: thread level 1=  2, thread level 2= 1, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 35,103,171,239)
Hello from level 2: thread level 1=  2, thread level 2= 2, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 36,104,172,240)
Hello from level 2: thread level 1=  3, thread level 2= 0, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 51,119,187,255)
Hello from level 2: thread level 1=  3, thread level 2= 1, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 52,120,188,256)
Hello from level 2: thread level 1=  3, thread level 2= 2, on c401-094.stampede2.tacc.utexas.edu. (core affinity = 53,121,189,257)
c401-094.stampede2(21)$ 


To request a quad,flat node on Stampede2, do
% idev -p flat-quadrant
And at run time, use numactl to use MCDRAM
% numactl -m 1 ./xthi-omp   (if use memory < 16 GB)
or
% numactl -p 1 ./xthi-omp    (if use memory > 16 GB)

To request a xnc4,flat node on Stampede2, do
% idev -p flat-snc4
And at run time, use numactl to use MCDRAM
% numactl -m 4,5,6,7 ./xthi-omp   (if use memory < 16 GB)
or
% numactl -p 4,5,6,7 ./xthi-omp    (if use memory > 16 GB)


Hybrid MPI/OpenMP:
Sample outputs with 16 MPI tasks x 8 OpenMP threads per task 
on a single 68-core stampede2 KNL quad,cache node using mpirun 
(binary built with Intel MPI).

% export OMP_NUM_THREADS=8
% export OMP_PROC_BIND=spread    
% export OMP_PLACES=threads      

% mpirun -n 16  ./xthi |sort -k4n,6n
Hello from rank 0, thread 0, on cc10. (core affinity = 0)
Hello from rank 0, thread 1, on cc10. (core affinity = 204)
Hello from rank 0, thread 2, on cc10. (core affinity = 69)
Hello from rank 0, thread 3, on cc10. (core affinity = 205)
Hello from rank 0, thread 4, on cc10. (core affinity = 70)
Hello from rank 0, thread 5, on cc10. (core affinity = 206)
Hello from rank 0, thread 6, on cc10. (core affinity = 71)
Hello from rank 0, thread 7, on cc10. (core affinity = 207)
Hello from rank 1, thread 0, on cc10. (core affinity = 72)
Hello from rank 1, thread 1, on cc10. (core affinity = 5)
...

% export I_MPI_PIN_DOMAIN=16
% mpirun -n 16 ./xthi |sort -k4n,6n
Hello from rank 0, thread 0, on cc10. (core affinity = 0)
Hello from rank 0, thread 1, on cc10. (core affinity = 136)
Hello from rank 0, thread 2, on cc10. (core affinity = 1)
Hello from rank 0, thread 3, on cc10. (core affinity = 137)
Hello from rank 0, thread 4, on cc10. (core affinity = 2)
Hello from rank 0, thread 5, on cc10. (core affinity = 138)
Hello from rank 0, thread 6, on cc10. (core affinity = 3)
Hello from rank 0, thread 7, on cc10. (core affinity = 139)
Hello from rank 1, thread 0, on cc10. (core affinity = 4)
Hello from rank 1, thread 1, on cc10. (core affinity = 140)
...





