DFTB optimization freezes for large system (~1100 atoms)

[foster362]

Hi, I'm trying to optimization the geometry of a system containing ~1100 atoms at the DFTB level (SK=3-ob-2-1(C,O,N,H)). I get the energy, within seconds using 36 CPUs (1 node), but the calculation freezes right before generating the forces for the first time:

Code

+------------------------------------------------------------------------------+
|  15 E = -3329.86 dE =  1.941906e-06 dM =  4.086617e-06 dH =  2.739183e-06    |
+------------------------------------------------------------------------------+
| Calculation Converged in 15 steps                                            |
|                                                                              |
| Fermi Level  = -4.457349 eV                                                  |
+------------------------------------------------------------------------------+
+------------------------------------------------------------------------------+
|                                                                              |
| Slater-Koster Calculation  [Finished Fri Feb 10 15:32:14 2017]               |
|                                                                              |
+------------------------------------------------------------------------------+
+------------------------------------------------------------------------------+
|                                                                              |
| Size of dense matrices =  3760 x  3760 [216 MB per matrix]                   |
|                                                                              |
+------------------------------------------------------------------------------+
+------------------------------------------------------------------------------+
| DiagonalizationSolver parallelization report.                                |
+------------------------------------------------------------------------------+
| Total number of processes: 36                                                |
| Total number of k-points: 1                                                  |
| Processes per k-point: 36                                                    |
| Number of process groups: 1                                                  |
+------------------------------------------------------------------------------+
+------------------------------------------------------------------------------+
|                                                                              |
| Size of dense matrices =  3760 x  3760 [216 MB per matrix]                   |
|                                                                              |
+------------------------------------------------------------------------------+
+------------------------------------------------------------------------------+
|                                                                              |
| Slater-Koster Calculation  [Started Fri Feb 10 15:32:17 2017]                |
|                                                                              |
+------------------------------------------------------------------------------+
+------------------------------------------------------------------------------+
|                                                                              |
| CPU Information                                                              |
|                                                                              |
+------------------------------------------------------------------------------+
|  Process ID 0 at ser7                                                        |
|  Process ID 1 at ser7                                                        |
.
.
.
|  Process ID 34 at ser7                                                       |
|  Process ID 35 at ser7                                                       |
+------------------------------------------------------------------------------+
+------------------------------------------------------------------------------+
|                                                                              |
| Size of dense matrices =  3760 x  3760 [216 MB per matrix]                   |
|                                                                              |
+------------------------------------------------------------------------------+
 
                            |--------------------------------------------------|
Calculating Kinetic Matrix : ==================================================
+------------------------------------------------------------------------------+
|                                                                              |
| Real space grid sampling is (68, 95, 116) in a, b, and c directions.         |
|                                                                              |
+------------------------------------------------------------------------------+
+------------------------------------------------------------------------------+
|                                                                              |
| Size of dense matrices =  3760 x  3760 [216 MB per matrix]                   |
|                                                                              |
+------------------------------------------------------------------------------+
+------------------------------------------------------------------------------+
| DiagonalizationSolver parallelization report.                                |
+------------------------------------------------------------------------------+
| Total number of processes: 36                                                |
| Total number of k-points: 1                                                  |
| Processes per k-point: 36                                                    |
| Number of process groups: 1                                                  |
+------------------------------------------------------------------------------+
 
+------------------------------------------------------------------------------+
| Checkpoint Handler                                                           |
| Filename : /tmp/1634514/checkpoint72536830.nc                                |
| Interval : 0.5 h                                                             |
+------------------------------------------------------------------------------+
 
                            |--------------------------------------------------|
Calculating Eigenvalues    : ==================================================
Calculating Density Matrix :

    However, the calculation remains running, it just doesn't appear to be doing anything. I ran a test on a smaller system (~200) and the calculation completes successfully (geometry optimization). The larger system is just a supercell of the smaller one. Note, ATK input file is below. Also, can you add an empirical dispersion correction to DFTB in  ATK-SE? Thanks ATK-SE input:

Code

# -*- coding: utf-8 -*-
# -------------------------------------------------------------
# Bulk Configuration
# -------------------------------------------------------------
 
# Set up lattice
vector_a = [25.0, 0.0, 0.0]*Angstrom
vector_b = [0.0, 35.0, 0.0]*Angstrom
vector_c = [0.0, 0.0, 42.8547]*Angstrom
lattice = UnitCell(vector_a, vector_b, vector_c)
 
# Define elements
elements = [Carbon, Carbon, Carbon, Carbon, Carbon, Carbon, Carbon, Carbon,
            Carbon, Carbon, Carbon, Carbon, Carbon, Carbon, Carbon, Carbon, 
.
.
.
                       [  6.31005   ,  23.683765  ,  41.56293078],
                         [  7.211625  ,  24.46654   ,  41.76113377],
                         [  5.4128    ,  23.838465  ,  40.53407514]]*Angstrom
 
# Set up configuration
bulk_configuration = BulkConfiguration(
    bravais_lattice=lattice,
    elements=elements,
    fractional_coordinates=fractional_coordinates
    )
 
# -------------------------------------------------------------
# Calculator
# -------------------------------------------------------------
#----------------------------------------
# Basis Set
#----------------------------------------
basis_set = DFTBDirectory("/home/mefoste/DFTB+/DFTB_parameters/3ob-2-1")
 
#----------------------------------------
# Pair Potentials
#----------------------------------------
pair_potentials = DFTBDirectory("/home/mefoste/DFTB+/DFTB_parameters/3ob-2-1")
 
iteration_control_parameters = IterationControlParameters()
 
calculator = SlaterKosterCalculator(
    basis_set=basis_set,
    pair_potentials=pair_potentials,
    iteration_control_parameters=iteration_control_parameters,
    )
 
bulk_configuration.setCalculator(calculator)
nlprint(bulk_configuration)
bulk_configuration.update()
nlsave('DR-CT_trans.nc', bulk_configuration)
 
# -------------------------------------------------------------
# Optimize Geometry
# -------------------------------------------------------------
bulk_configuration = OptimizeGeometry(
    bulk_configuration,
    max_forces=0.05*eV/Ang,
    max_steps=5000,
    max_step_length=0.2*Ang,
    trajectory_filename=None,
    disable_stress=True,
    optimizer_method=LBFGS(),
)
nlsave('DR-CT_trans.nc', bulk_configuration)
nlprint(bulk_configuration)
 
# -------------------------------------------------------------
# Electron Density
# -------------------------------------------------------------
electron_density = ElectronDensity(
    configuration=bulk_configuration,
    )
nlsave('DR-CT_trans.nc', electron_density)
 
# -------------------------------------------------------------
# Electrostatic Potential
# -------------------------------------------------------------
electrostatic_potential = ElectrostaticPotential(bulk_configuration)
nlsave('DR-CT_trans.nc', electrostatic_potential)

[Anders Blom]

Might be a memory issue. May I suggest you try to run the same large system using just 1 MPI process (or 4, or 2). Since there is only 1 k-point, there isn't much to parallelize over anyway, but the MPI brings in some memory overhead. I can't test since I don't have your DFTB parameters (and not a complete input file), but you could share that via email support@quantumwise.com if the problems persist.