cluster configuration

[kanna]

We are thinking of buying a new cluster and extra licenses. Is there something special which we need to look for ?

Should we invest more on
1. CPU frequency
2. memory per node
3. frequency of operation of the memory
4. SSD drive or SATA drive
5. any cards for mpi communication?
6. Choice of operating system

Please let us know if there is some tradeoff within the features which we can follow to optimize our cost.

Typically we run optimization and transport with electron-phonon model for 1000 TMD's.

[Anders Blom]

1. The higher the better
2. The more the better
3. The higher the better
4.  Doesn't matter for ATK
5. If you can afford  Infiniband you can buy that, but it seems for ATK the benefit is very low, generally
6. Linux

I know these are a bit "silly" answers, but they are true and correct

[kanna]

Dear Anders,

Is there a limitation on the bus speed which connects main memory and CPU cluster ?
Is it advisable to have two CPU clusters connecting to the same memory or single CPU cluster connecting to the main memory ?
Also do we need to have different bus for each cluster ?

[Anders Blom]

The memory (RAM) would never be split over "clusters"... Please check your notation.

[kanna]

Apologies for my ignorance.
If we buy a server with 2 processors (each with 16 threads, connected by QPI as shown in the attached chipset diagram), so that in total we have 32 threads in the server
Then will mpiexec -n 16 , it parallelize into the 2 processors perfectly (where each processor has 16 threads) ?

Thanks
kanna

[Anders Blom]

Be careful not confuse "threads" with "cores". Some setups will use hyperthreading to assign 2 threads per core; this is ok for office applications, but not HPC work. So, using mpiexec -n X, the number X should never exceed the number of physical cores. This may only be 8 in your setup.

Also, having a single machine means your X processes now need to use the same RAM, so if a calculation takes 8 GB in serial and you run X=8, be prepared that the calculation may consume 64 GB of memory. That is, for such a machine, get 128 GB of RAM, is my recommendation.

Now, "parallelize perfectly" depends very much on your problem - not all calculations parallelize as well, and sometimes you will be limited by memory, and so on. But generally you should have a good parallel benefit of such a machine.

[kanna]

I just got a bit confused, when we use mpiexec -n X, X is the number of threads and each thread corresponds to 1 atkslave license (Is this correct ?)
Please take a look at
http://ark.intel.com/products/83361/Intel-Xeon-Processor-E5-2667-v3-20M-Cache-3_20-GHz

and let me know if we can use only mpiexec -n 8 for this processor (this means 8 atkslave licenses ?)

Thanks
kanna

[Anders Blom]

Each MPI process needs a license, yes, so with mpeixec -n 8 you use 1 master and actually 7 slaves.

The process you indicate has 8 physical cores, not 16.