No WAVECAR output using MLFF?

[jdrewitt]

Hello,
I am using vasp.6.3.2 for on-the-fly training of the MLFF. As the calculations remain highly computationally intensive, they must run over several days and require restarting mid-training. However, it appears that MLFF mode does not produce a WAVECAR required for a ab initio continuation job. I am using spin-polarization, so it is important to provide good starting wavefunctions for the ab initio part, so it it possible to enable WAVECAR output during the MLFF-training run?

My INCAR is below:

SYSTEM = Hedenbergite
# electronic degrees
ENCUT = 550
LREAL = A # real space projection
PREC = Normal # chose Low only after tests
EDIFF = 1E-5 # do not use default (too large drift)
ISMEAR = -1 ; SIGMA = 0.21 # ISMEAR = -1 Fermi smearing # SIGMA = 0.21 at 2500 K
ALGO = All
TIME = 0.05
MAXMIX = 40 # reuse mixer from one MD step to next
ISYM = 0 # no symmetry
NELMIN = 4 # minimum 4 steps per time step, avoid breaking after 2 steps
NPAR = 16

NELM = 200 # max number of electronic self consistency steps

# MAGNETIC STUFF
ISPIN = 2 # spin-polarized calculations (collinear) are performed
LORBIT = 11 # write PROCAR
MAGMOM = 30*2.93 30*0 60*0 180*0 # set initial magnetic moment of Fe

LMAXMIX = 4

LDAU = .TRUE.
LDAUTYPE = 2
LDAUL = 2 -1 -1 -1
LDAUU = 4.00 0.00 0.00 0.00
LDAUJ = 1.00 0.00 0.00 0.00
LDAUPRINT = 2

# MD
IBRION = 0 #0 = MD
MDALGO = 2 #NVT ensemble with Nose-Hoover thermostat
NSW = 500 #number of ionic steps
NWRITE = 1 #write volume at each ionic step
LCHARG = .FALSE.
TEBEG = 2500 ; TEEND = 2500 #temperature
NBLOCK = 1 ; POTIM = 1
ISIF = 2 #calculate stress tensor
SMASS = 2
ISTART = 1 #Start from scratch (0) / wavecar (1)

#Machine learning parameters
RANDOM_SEED = 88951986 0 0
ML_LMLFF = .TRUE.
ML_ISTART = 0
ML_MB = 3000
ML_LBASIS_DISCARD = .TRUE.

[marie-therese.huebsch]

Hi,

I am not sure I understand what you need.

Of course, the Kohn-Sham orbitals are only computed when an electronic minimization is performed, i.e., during an ab-initio step. If the forces for an ionic step are computed using the force field, the electronic problem is not solved. That is why MLFF is faster than AIMD.

Yet, during training, many steps are ab-initio steps, and the WAVECAR is written. Generally, the ionic positions are extrapolated such that the WAVECAR of a previous ionic step can be used as initial KS orbitals for the next step.

Maybe there was another issue in your calculation, such that part of the output was not written. I suggest you try to reproduce the issue with some small system performing only a few ionic steps so that the computational cost is small. Please post the input and main output files so I can try to reproduce the issue.

Marie-Therese

[jdrewitt]

Hello, thanks for you reply.

The WAVECAR is only written on the last ionic step. If the last ionic step is an ab initio step, then the WAVECAR is written correctly. However, if the last ionic step is an MLFF step, the WAVECAR is written with 0 bytes.

Is it possible to write a WAVECAR from the last ab initio step to restart the run? Or is this simply not possible due to the ionic motions in the MLFF step meaning the old WAVECAR is no longer a good start point to extrapolate from?

I have tested with a smaller system as requested, the input and output files are attached. MLFF_1 is 10 steps and ends on an ab initio run, so a WAVECAR was written correctly (not included in the zip as it is too large). MLFF_2 is 20 steps and ends on a MLFF run, so writes a WAVECAR of 0 bytes (included).

Best wishes,

James

[marie-therese.huebsch]

Ok, now I see. We will investigate the possibility of writing the WAVECAR corresponding to a previous ab-initio step. Currently, there is no way to change the behavior in your calculation. Sorry.

Because you have mentioned that your system requires good starting wavefunctions for the ab-initio calculation: Please also carefully test if RMM-DIIS (ALGO=Very Fast, i.e., IALGO=48) is a good choice for your system ft magnetism and on-site U. You can read the comments and linked references on the Wiki page of IALGO. I am not saying using RMM-DIIS in your calculation is wrong, but that it requires caution (as always).

Marie-Therese

[jdrewitt]

Thanks, that would be a really good update. Please let me know if/when it gets implemented?

For the magnetic calculations I am using ALGO = All (as in the INCAR pasted into my OP), and start from a well-converged configuration. The example calcs I attached were for a completely different system, just to illustrate the issue.

Best wishes, James

[marie-therese.huebsch]

Ah, yes! I mixed the two examples.

I will let you know

Marie-Therese

[jdrewitt]

Hi Marie-Therese,

Since our conversation, there have been some new releases, has this fix now been implemented in VASP.6.4.2?

Best wishes,

James

[ferenc_karsai]

Unfortunately we will not write a WAVECAR file during on-the-fly traing for the following reasons:

1) We don't know in advance which ab-initio step will be the last step. So we would have to write every step. But writing every step is computationally inefficient and would strongly slow down the on-the-fly training.

2) If the last few hundred/thousand steps would not contain an ab-initio step. Then the WAVECAR could not be ideal for the current CONTCAR requiring a restart without a WAVECAR file (that is why we anyway encourage the users to restart often) or one would have to rewind in the continuation calculation to the POSCAR file of the last ab-initio step (this could be taken from the XDATCAR file), losing all the progress in the MD made until the end. This is possibly not a big problem at the beginning of the training but can become highly inefficient in later parts of the training.

3) If the volume increases (for example in NpT runs) the current plane-wave basis (and also WAVECAR coming from that basis) can be too small. In that case frequent restarts are anyway advisable to adjust the basis to the increased volume. The adjustment is automatically done by restarting without a WAVECAR file.

[jdrewitt]

OK, thank you for your reply and I understand your rationale.

For context, I have experienced large spikes in energy/pressure/temperature when restarting long production runs without a WAVECAR. Using a WAVECAR eliminates these spikes enabling me to obtain a continuous equilibration across several runs. I'm not sure how detrimental these spikes are, but for calculating transport properties I need to average the trajectories over a long timescale and so always restart with WAVECAR.

I assume that during on the fly training, each new ab initio step will calculate new wavefunctions from scratch? So regarding your answer 1), would there be any benefit to ensure the very last step of each simulation (as specified in the INCAR) is always an ab initio step? Would this be identical to restarting after n-1 steps without a WAVECAR?

Best wishes,

James

[ferenc_karsai]

The main source of problem is that many magnetic systems like yours can excibit a lot of local minima in the electronic minimization that are energetically not strongly separated from each other. The minimization struggles to find the real minimum and is easily trapped in a local minimum signaling convergence. In other words the electronic minimization is not robust. This is evident from the observation that when you start from scratch or from a WAVECAR you get different results. When startiing from a WAVECAR the calculation is biased and steered towards a local minimum that is very likely not a global minimum. The first thing I would have advised you is to use ALGO=ALL because that is usually more robust for magnetic systems, but you already did that.


I have discussed the problem with Georg Kresse and he reported that another group tried to learn a strongly magnetic system with on the fly and they failed. Currently magnetic systems are not very likely to work well with on-the-fly learning and we don't have a clear strategy yet.


Another thing I saw is that you wrote that you are doing production calculations, so I assume you want to use them in combination with on-the-fly sampling. It is generally not the best strategy to do that because the Hamiltonian changes when the force field is updated. That means that the energy predicted at step X with force field A would be differently predicted than with the updated force field A'. This introduces statistical inconsistencies. So it's better to finish sampling and training first and use the "fixed" force field in the application runs.  


What you could try is to:

1) On-the-fly sample your training data without careing much about magnetization.

2) Redo all the ab-initio calculations for all structures starting each from scratch and build a new ML_AB file (this you will have to do by hand).

3) Reselect local reference using ML_MODE=select.

4) Refit using ML_MODE=refit.

5) Run applications.

[ferenc_karsai]

I would like to look at your problem also with our chief scientific officer.
Could you please provide the following files:
INCAR, POSCAR, POTCAR, KPOINTS, ML_LOGFILE, OUTCAR and ML_AB

The OUTCAR should be from the on-the-fly calculation.

Also another thing is that you have set 2500 K as target temperature. Magnetism breaks down at high temperature temperatures.

[jdrewitt]

Apologies for the delayed response. The temperature is high because these are molten Fe-bearing silicates which I intend to run to high pressure, see https://www.nature.com/articles/s41467-020-14773-4 where spin-polarisation is used to simulate Earth's magnetic dynamo.

Attached are the MLFF files you requested.

The OUTCAR is 166 Mb so can be downloaded here: https://1drv.ms/u/s!AkfjfW1yabD4gdtFyw5 ... g?e=JcHffE

Thank you.