Before the form builder service invokes the form object's submitForm method, it invokes the object's validateForm method. In the validation method, the form state can be used to check values and perform logic checks. In the event that an item is deemed invalid and an error is set on an element, the form cannot be submitted and will show errors to the user.
When an error is added to an element, an overall counter for the number of errors on the form is incremented. If the form has any errors, the form builder service will not execute the submit method.
This process is executed through the \Drupal\Core\Form\FormValidator class, which is run through the form builder service.