The mesh cut-off is given as input by the user. It's an energy (default 150 Ry) which corresponds to the fineness of the real-space grid on which the Poisson equation is solved. (Or, grids, there are a couple of different grids, including the real-space electron density and the effective potential.) A higher value of the mesh cut-off gives a finer real-space grid and hence better accuracy.
The fineness or grid point spacing dx is calculated according to the formulas (hmm, isn't there a way to type math prettier...?)
E = hbar2 k2 / 2m
dx = pi/k (no factor 2 here!)
which if E is in Rydberg and dx is in Bohr becomes
dx = pi/sqrt(E)
k is the equivalent plane-wave cut-off component, but one can not really compare the values used in a plane-wave code and ATK, for several reasons.
This is perhaps obvious to you, but it's best to have a full explanation at hand.
Now, regarding how to choose the mesh cut-off, the only really proper way is to try with one value, then change it and see if the results (say, the current, or band structure) changes. If it does, you keep increasing it until the results converge.
The default (150 Ry) is often sufficient in most cases, in particular for electronic structure and transport calculations. In fact, for carbon nanotubes and other systems you can often decrease it and still get quality results (but not necessarily very fine plots of the density...). For geometry optimizations, a higher value is sometimes needed, in particular for elements with complex d-shells like iron.
So, the number of atoms is not really a determining factor a priori, and in most cases the real-space grids are not a limiting factor in the sense that the calculation uses too much memory. The main impact of increasing the mesh-cut off is rather on calculation time.