Android unable to switch from camera to screen sharing webrtc in middle of communication

Question

I'm working on android app and using webrtc with openvidu. I got screen sharing and camera sharing working. But only either of those works in one peerconnection. I'm unable to switch between them. By default when connection is established, the camera will be shared and on click of a button I should be able to share screen without disconnecting existing connection. However that is not happening in my case. It doesn't publish the screen on click of button. It goes blank.

If anyone has worked on switching between camera to sharescreen and vice versa, please help me with this. Thanks in advance.

    public VideoTrack MakeCameraSettingsReady() {
            final EglBase.Context eglBaseContext = EglBase.create().getEglBaseContext();
            PeerConnectionFactory peerConnectionFactory = this.sessionManager.getPeerConnectionFactory();
    
            // create AudioSource
            AudioSource audioSource = peerConnectionFactory.createAudioSource(new MediaConstraints());
            this.audioTrack = peerConnectionFactory.createAudioTrack("101", audioSource);
    
            surfaceTextureHelper = SurfaceTextureHelper.create("CaptureThread", eglBaseContext);
            // create VideoCapturer
            videoCapturer = createVideoCapturer();
    
            //MediaConstraints constraints = new MediaConstraints();
    
            VideoSource videoSource = peerConnectionFactory.createVideoSource(videoCapturer.isScreencast());
            localVideoTrack = peerConnectionFactory.createVideoTrack("100", videoSource);
            videoCapturer.initialize(surfaceTextureHelper, context, videoSource.getCapturerObserver());
            videoCapturer.startCapture(720, 1280, 30);
    
            return localVideoTrack;
        }
    
        public VideoTrack MakeScreenCaptureReady() {
            final EglBase.Context eglBaseContext = EglBase.create().getEglBaseContext();
            PeerConnectionFactory peerConnectionFactory = this.sessionManager.getPeerConnectionFactory();
    
            // create AudioSource
            AudioSource audioSource = peerConnectionFactory.createAudioSource(new MediaConstraints());
            this.audioTrack = peerConnectionFactory.createAudioTrack("101", audioSource);
    
            surfaceTextureHelper = SurfaceTextureHelper.create("CaptureThread", eglBaseContext);
            // create VideoCapturer
            screenCapturerAndroid = (ScreenCapturerAndroid) createScreenCapturer();
    
            //MediaConstraints constraints = new MediaConstraints();
    
            VideoSource videoSource = peerConnectionFactory.createVideoSource(screenCapturerAndroid.isScreencast());
            localVideoTrack = peerConnectionFactory.createVideoTrack("100", videoSource);
            screenCapturerAndroid.initialize(surfaceTextureHelper, context, videoSource.getCapturerObserver());
            screenCapturerAndroid.startCapture(720, 1280, 30);

            return localVideoTrack;
        }

When I click button to share screen, I use below code:

  private fun stopCameraShare(){
            videoCapturerAndroid?.stopCapture()
            localRenderer.dispose()
            localVideoView.release()
            localVideoView.clearImage()
            stream?.removeTrack(localVideoTrack)
            localVideoTrack.dispose()
        }

    private fun shareScreen(){
            stopCameraShare()
            val mediaProjectionManager = activity!!.getSystemService(Context.MEDIA_PROJECTION_SERVICE) as MediaProjectionManager
            startActivityForResult(mediaProjectionManager.createScreenCaptureIntent(), 29)
        }

    override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
        super.onActivityResult(requestCode, resultCode, data)
        if (requestCode == SCREEN_RECORD_REQUEST_CODE) {
            if (resultCode == RESULT_OK) {
                //Start screen recording
                mResultCode = resultCode
                mResultData = data

                runOnUiThread {
                    screenCaptureVideoTrack = mLiveSessionViewModel!!.localParticipant?.MakeScreenCaptureReady()
                    mLiveSessionViewModel!!.localParticipant?.screenCaptureVideoTrack = screenCaptureVideoTrack
                }
            }
        }
    }

References that I tried: How to share screen remotely in a video/audio call?

https://github.com/Jeffiano/ScreenShareRTC

https://chromium.googlesource.com/external/webrtc/+/b75f2541c942e2f35c3b7d7003ed17504176ced1/webrtc/api/android/java/src/org/webrtc/ScreenCapturerAndroid.java

WebRTC - change video stream in the middle of communication

Ultimately I'm trying to find a way where I can addTrack/removeTrack in middle of communication. Thank you!

Answer 1

I think you are using two video tracks. Why don't you try with only one video track.

Like for camera sharing, assign video track as

mLiveSessionViewModel!!.localParticipant?.mediaStream.videotraks[0] = cameraVideoTrack

To switch to screen share, remove existing video track and add the respective videotrack

mLiveSessionViewModel!!.localParticipant?.mediaStream.videotrakcs[0].removeTrack(cameraVideoTrack)
mLiveSessionViewModel!!.localParticipant?.mediaStream.videotracks[0].addTrack(screenShareVideoTrack)