The Responsive Image module attaches the picturefill library to the responsive image element definition. The element's template also provides HTML to implement the polyfill. The polyfill can be removed by overriding the element's template and overriding the picturefill library to be disabled.

The following snippet, when added to a theme's info.yml, will disable the picturefill library:

libraries-override: 
  core/picturefill: false 

Then, the responsive-image.html.twig must be overridden by the theme to remove the extra HTML generated in the template for the polyfill:

1. Copy responsive-image.html.twig from core/modules/responsive_image/templates to the theme templates folder.
2. Edit responsive-image.html.twig and delete the Twig comment and IE conditional to output the initial video tag.
3. Remove the last conditional, that provides the closing video tag.