It is in fact possible to use your own pseudopotentials. QuantumATK supports three pseudopotential formats:
For using a pseudopotential you also need to provide a compatible LCAO basis set. The LCAO basis set consists of a set of confined orbitals that are generated on the fly from the pseudopotential: you have to specify which orbitals and tweak things like confinement potential. For LCAO-DFT finding a good and at the same time fast basis set is by it-self not an easy task. For PW-DFT only the occupied orbitals of the basis set is used to initialize the density before the self-consistent iterations, so the quality of the basis set is unimportant for the results.
OpenMX pseudopotentials are special and need to use a matching LCAO basis set pregenereated and numerically tabulated in special .pao files.
To use a custom pseudopotential I suggest you create a structure in the Builder with the elements you are missing, send it to the scripter, select either PseudoDojo (ONCVPSP) or OMX (OpenMX) pseudopotential (with spin-orbit I presume) in CalculatorSettings. Then in the Scripter you select "Show Defaults" in the "Script Details" drop down and then send to Editor. It will then generate a script from which you can see how the basis set orbitals and pseudopotentials are specified. You can then modify it to suit your custom pseudopotential.
Feel free to ask any questions if you get stuck.
