how to send video stream using webrtc

Question

I'm new to webrtc,i liked to make simple application where client send video and audio stream to server using webrtc and from server i will use the video frame for detecting object in the video using opencv, i have implemented simple server side code using docs of aiortc package,but i was stuck,because the on_track never called i dont know whats wrong with my code

Client code

   const { RTCPeerConnection, RTCSessionDescription } = window;
socket = io()
const peerConnection = new RTCPeerConnection()
const offerAsync = peerConnection.createOffer();
offerAsync.then(async offer=>{
    await peerConnection.setLocalDescription(new RTCSessionDescription(offer));
    socket.emit("connect_to_server", {
      offer
    });
})

    
socket.on("connect_to_server_response",async function(data){
     navigator.mediaDevices.getUserMedia({
         video:true,
         audio:true
     }).then(async stream=>{
         const video = document.getElementById("video")
         await peerConnection.setRemoteDescription(
             data
          );
         console.log(data) 
         stream.getTracks().forEach(track => {
             console.log("this is track")
            peerConnection.addTrack(track, stream)
         });  
         console.log(video)
         video.srcObject = stream
         video.play()
     }).catch(err=>{
         console.log(err)
         console.log("something went wrong please connect administrator, error code = [100]")
     })
})    

console.log("working");

Server code

async def res(data):
    offer = RTCSessionDescription(sdp=data["offer"]["sdp"], type=data["offer"]["type"])
    pc = RTCPeerConnection()
    @pc.on("track")
    def on_track(track):
        print("on Track")
        print(track.kind)     
    await pc.setRemoteDescription(offer)  
    answer = await pc.createAnswer()
    await pc.setLocalDescription(answer)  
    emit("connect_to_server_response", {"sdp": pc.localDescription.sdp, "type": pc.localDescription.type})

@socketio.on("connect_to_server")
def connect_to_server(data):
    asyncio.set_event_loop(asyncio.SelectorEventLoop())
    asyncio.get_event_loop().run_until_complete(res(data))
    print(data)
    print(request.sid)
    print("new user joint")  

Answer 1

It looks like you are exchanging the SDP messages fine and in the right order.

Your issue is because you are never exchanging ICE candidates. In order to establish a connection, you will need to emit ice candidate messages from the js client to the python client as well as from the python client to the js client.

You need to define an event handler for RTCPeerConnection.onicecandidate (this will start getting fired once you call createOffer) on both sides and emit these candidates that you're receiving. Then you have to RTCPeerConnection.addIceCandidate once you receive the candidates.

Once you successfully do that, you should be able to get the on_track event.

Check out these links for reference:

What are ICE Candidates and how do the peer connection choose between them?

RTCIceCandidate

Simple webrtc workflow sample code