Drupal core defines a plugin.manager.field.formatter service. By default, this is handled through the \Drupal\Core\Field\FormatterPluginManager class. This plugin manager defines the field formatter plugins that should be in the Plugin/Field/FieldFormatter namespace, and all the classes in this namespace will be loaded and assumed to be field formatter plugins.
The manager's definition also sets \Drupal\Core\Field\FormatterInterface as the expected interface that all field formatter plugins will implement. This is why most field formatters extend \Drupal\Core\Field\FormatterBase to meet these method requirements.
As field formatters are annotated plugins, the manager provides \Drupal\Core\Field\Annotation\FieldFormatter as the class that fulfills the annotation definition.
The entity view display system uses the plugin.manager.field.formatter service to load field definitions and add the field's render array, returned from the viewElements method, to the entity view render array.