Using the rhombohedral cell is quite difficult. I can try to give you some tips later on, if really needed. However, if you look into the more modern literature about topological insulators, you will find that most use the hexagonal supercell for calculations, and then it's a lot easier because ATK has all the relevant symmetry points and the notation is standardized.