Geometry optimization is really hard and time consuming for all but the simplest systems due to the very high number of degrees of freedom. You need very high numerical parameters (basis set, k-point sampling, density mesh cutoff) in order for it to converge smoothly, as you otherwise get random numerical errors that are on the order of magnitude as the energy differences and forces involved in the optimization steps.
The first step before doing geometry optimization is always to first determine the numerical parameters to use by doing a convergence study. This means doing repeatet DFT calculations of energy, forces and stress for the initial structure at increasing basis set, k-point sampling and density mesh cutoff (vary each independently) until the energy/forces/stress change less than at least an order of magnitude lower than the tolerances you want to use for the geometry optimization. Having found the optimal numerical parameters you can then proceed to use those for the geometry optimization.
However, in many cases, especially for larger composite system, geometry optimization is simply not possible in a single "hit the button and run" approach. Either it simply takes too long time or it gets stuck jumping between the many local minima. In these situations it is better to break the optimization into steps.
For instance in your case you can break the optimization into 4 steps, each done as separate calculations/jobs:
- Optimize the Stanene nanoribbon geometry alone by running a geometry optimization (atoms + unit cell along periodic direction) of a single unit cell
- Optimize the molecule alone without the nanoribbin in a separate calculation
- Find the optimimal nanoribbon <-> molecule distance by simply calculating the total energy of the nanoribbon + molecule system at different distances, fitting some polynomial to the results and finding the minimum
- Relax the atoms in the molecule and in the nanoribbon close to the molecule with a regular geometry optimization where you fix the unit cell and most atoms in the nanoribbon. The more atoms and thus degrees of freedom you fix, the faster and more stable the geometry optimization will be
In this way you trade computational load with more manual work, but the above approach should work in most cases. Also you can find the optimum numerical parameters from doing calculations on the sub-parts: nanoribbon unit cell and molecule separately - this will be much, much faster than trying to do so for the entire system.