Generally yes, and most often the optimization will converge faster if you have sufficient accuracy, because the forces will be more accurate, and the forces are actually more sensitive to settings like k-points than the total energy is. However, sometimes it can save some time to use slightly lower accuracy settings for the first part of the optimization, maybe converge to higher force tolerance like 0.05 eV/A, and then use finer settings to converge the last bit to 0.01 eV/A or so. But the safest option is to ensure good accuracy first, and then run the optimization with that.