How to get "font-display" attribute to respect "unicode range"

Question

I have a website that has a mix of Hebrew and English characters. I load a custom Hebrew font in the css like so:

@font-face {
  font-family: 'Taamey David CLM';
  src: url('TaameyDavidCLM/TaameyDavidCLM-Medium.woff2') format('woff2'),
       url('TaameyDavidCLM/TaameyDavidCLM-Medium.woff') format('woff');
  font-weight: 500;
  font-style: normal;
  font-display: block;
  unicode-range:  U+0590-05FF, U+FB1D-FB4F;
}

I want to use the font-display: block attribute so that until the Hebrew font loads, Hebrew characters are not rendered visibly, so that the user doesn't see a switch from the system default font to the custom font.

However, I also don't want the English text to wait for the Hebrew font to load, which it seems to be doing. I have the following css:

#container p {
  font-family: 'Taamey David CLM', Times, Serif;
}

Based on that, I was expecting the browser to first try the custom font for all text, but then to recognize that there is a unicode-range that limits the custom font to Hebrew characters, and therefore for English characters to fall back to the to the "Times" family which is listed next in the font-family declaration. What seems to be happening is that the font has to download before the browser recognizes that the English text is outside the unicode range and falls back to "Times."

Is there any way to specify in the css that I want the English text to be rendered in the system Times font so that it doesn't have to wait for the Hebrew font to download? (I know I could put different HTML classes on the Hebrew and English text, but the text is very mixed and generated from user input so it isn't so simple to follow that approach.)

Edit: Interestingly, if the text has an <i> or <b> tag that contains only English characters, and if I have an additional font-face declaration for an italics or bold version of this font respectively, the text of the <i> or <b> renders before the font is downloaded, unlike the rest of the text. For some reason in that case, it seems that it is doing a check to see if any of the characters in the element are in the unicode-range, and if not, rendering immediately with Times. However, the rest of my text has many span elements with only English characters, and these do wait for the font to download before defaulting to Times. Not sure what the difference is.