Multi Lingual Bot with Audio input and output using React

Question

I am working on a requirement where I have to embed azure chat bot into React. I need to enable audio input and output functionality and also detection of user language and translation. I have seen Microsoft documentations where this is done from server side ( C# ) using speech and translator services. I am a beginner in react and would like to know if this can be achieved purely from React. Below is the way how I call Directline echo bot from React.

BotChat.App({
      bot: bot,
      locale: 'en-us',
      user: user,
      directLine: {
        secret: 'XXXXXXXXXXXXXXXXXXXXXX',
        webSocket: true,
          },
    },
    document.getElementById('bot')
  );

Since I already have an echo bot and speech, translator services created in Azure, I would like to know if these cognitive services be triggered from React Web chat bot.

Answer 1

I have acheived enabling speech to text and text to speech using the below code where I have used Speech services pony fill factory. For translator services, I have used Azure Translator service in echo bot .Net application.

import React, { useMemo } from 'react';
import ReactWebChat, { createCognitiveServicesSpeechServicesPonyfillFactory, createDirectLine, createStore } from 'botframework-webchat';
import './App.css';

function App() {

  const store = createStore();

  const directLine = useMemo(() => createDirectLine({ secret: 'YOUR_SECRET_HERE' }), []);

  const styleOptions = {
    bubbleBackground: 'rgba(0, 0, 255, .1)',
    bubbleFromUserBackground: 'rgba(0, 255, 0, .1)',
    botAvatarInitials: 'Bot',
    userAvatarInitials: 'Me',
    hideUploadButton: true,
    hideSendBox: false,
    sendTypingIndicator: false
  };

  const webSpeechPonyfillFactory = createCognitiveServicesSpeechServicesPonyfillFactory({
    credentials: {
      region: 'REGION',
      subscriptionKey: 'YOUR_KEY'
    },
    speechRecognitionEndpointId: 'YOUR_ENDPOINT',
  });

  const selectVoice = (voices, activity) => {
    if (activity.locale === 'US') {
      return voices.find(({ name }) => /AlvaroNeural/.test(name));
    } else {
      return voices.find(({ name }) => /NeerjaNeural/.test(name));
    }
  };

  return (
    <div>
      <h1>Custom </h1>
      <ReactWebChat
        store={store}
        directLine={directLine}
        selectVoice={selectVoice}
        webSpeechPonyfillFactory={webSpeechPonyfillFactory}
        styleOptions={styleOptions}
      />
    </div>
  );
}

export default App;