Reducing memory consumption for MTP training

[pshinyeong]

I am trying to train MTP for silicon bulk with single boron dopant (as a test).
I created the script using defect training template in the workflow.
Pristine configuration was primitive silicon bulk (2 atoms), defect configuration generated has supercell repetition of 3x3x3.
I used 2x2x2 k-point density, pseudodojo medium basis set for silicon and boron, reduced the basis size to 400, used all available processes per task.
Most of the parameters were left as default settings.
I ran the script using intel MPI & SLURM with 3 nodes 18 cores each. Each node has 64GB memory.
Calculation file, log file and screenshot of the error is attatched below.

I also tried increasing the number of nodes, reducing the number of cores, increasing the number of processes per task, reducing the basis size etc. and same error appeared (I assume it is memory error)

My main goal is to do active learning MTP but if the above calculation faces memory error, I am pretty sure active learning MTP would also result in memory error.
Could anyone suggest any further ways to reduce memory consumption during MTP training?

Thank you

[Anders Blom]

It's not necessarily out of memory. Can you check the other log files for errors?
In particular the last one, mtp_training_defectconfiguration_0_bundle_2.log

[pshinyeong]

Hello, I do not see any error message in the mtp_training_defectconfiguration_0_bundle_2.log,
I have attached the lmtp_training_defectconfiguration_0_bundle_2.log for reference

[Julian Schneider]

I tried running your script and it seems to run ok in principle. So it could be that it just runs out of memory on your computer and it looks like this is happening during the DFT calculations of the training data.
It looks like you have already tried most of the options to reduce memory by distributing across more nodes.
Apart from that it might be possible to reduce the memory a bit by deleting the calculator state of the optimized configurations, as shown in the attached script.
Also, it seems that the ParallelCG solver you use for the poisson equation uses a bit more memory than the default FFT solver, so changing that could also help a bit, unless you have a specific reason for using that solver.