The images would not 404 as they exist. So even if an older browser can't render the image it will not affect the pages ability to rank.
I would say if this concerns you then to use image-set so that you can give the browser a choice of images but that tends to be for the browser to pick the most appropriate image based on PPI and not image format. Plus of cause image-set is also not supported by all browsers. Firefox up to v72, for example, does not support this.
Speed of a page, however, has been confirmed as a ranking signal by Google. So using these formats to reduce the page load speed can only be a good thing.
Browser support for webp has been growing, the only exceptions to this are IE, Safari. So you have to balance out do you go now with webp to speed up your site and hopefully better ranking or go for wider support for users.
You would think the answer should be simple and go for widest support however the majority of internet users don't use these browsers so personally, I think as webp is supported by all the big browsers I would go for speed where possible.
As a footnote, I would say that reducing your image sizes will not be your magic bullet to reaching the top slots in search. It's only 1 element that a webmaster should be doing housekeeping on.