I am trying to make a Video Calling app using Flutter and Webrtc. I am using the webrtc plugin for flutter from https://pub.dev/packages/flutter_webrtc.
I have made two different projects, one does the calling, the other receives the call. I made a makeshift signalling server using rails and actioncable.
When the caller(doctor) and callee(patient) both try to connect, The offer from the doctor is sent to the patient and the answer is sent back. ICE Candidates are also exchanged. But neither the doctor nor patient can see their respective "remote" streams, ie, they cannot see each other. The connection states finally end up as RTCPeerConnectionStateFailed.
I thought there was some race condition, so I tried to redo the code, so that the ice candidates are buffered into a list and then shared once the offer and answers are exchanged. That did not solve the issue either.
START OF EDIT:
I forgot to mention, I ran the doctor version on my phone and the patient version on the android emulator in my laptop.
I even tried with both being emulated by my laptop. Both are giving me same results.
END OF EDIT
I am pasting the excerpts from the code below.
Caller (Doctor)
import 'dart:convert';
import 'package:flutter/cupertino.dart';
import 'package:shared_preferences/shared_preferences.dart';
import 'package:web_socket_channel/io.dart';
import 'package:web_socket_channel/web_socket_channel.dart';
import 'package:flutter_webrtc/flutter_webrtc.dart';
typedef void StreamStateCallback(MediaStream stream);
class Signaling {
Map<String, dynamic> configuration = {
'iceServers': [
{
'urls': [
'stun:stun1.l.google.com:19302',
'stun:stun2.l.google.com:19302'
]
}
]
};
List<Map> iceCandidates = [];
List<Map> remoteIceCandidates = [];
RTCSessionDescription offer;
RTCPeerConnection peerConnection;
MediaStream localStream;
MediaStream remoteStream;
String roomId;
String uniqueId;
String patientId;
String currentRoomText;
StreamStateCallback onAddRemoteStream;
WebSocketChannel channel;
bool sentOffer = false;
bool gotAnswer = false;
void processWSMessage(dynamic event) async {
var payload = jsonDecode(event);
if (payload["message"] != null && payload["type"] == null) {
String type = payload["message"]["type"];
String msg = payload["message"]["message"];
// listen for hello message and send ice candidates and offer details
if(msg.indexOf("HELLO_DOCTOR") >= 0){
// got a hello from the doctor. doctor joined after patient.. so they havent gotten any details from us. send them everything
//send offer
sendPatientOffer();
}
// when you get ICE candidates from doctor
if(msg.indexOf("ICE_CANDIDATE")>=0){
String iceCandidate = msg.split("ICE_CANDIDATE:")[1];
Map data = jsonDecode(iceCandidate);
print('got ICE from patient');
remoteIceCandidates.add(data);
if(gotAnswer == true){
peerConnection.addCandidate(
RTCIceCandidate(
data['candidate'],
data['sdpMid'],
data['sdpMLineIndex'],
),
);
}
}
// when you get an answer
if(msg.indexOf("ANSWER")>=0){
String answerFromPatient = msg.split("ANSWER:")[1];
print("got answer from patient");
Map ans = jsonDecode(answerFromPatient);
var answer = RTCSessionDescription(
ans['sdp'],
ans['type'],
);
await peerConnection.setRemoteDescription(answer);
gotAnswer = true;
//send all our ice candidates
sendPatientAllIceCandidates();
// process old remote ICE candidates
print(remoteIceCandidates.length);
print(remoteIceCandidates);
remoteIceCandidates.map((data){
print("adding old candidates");
peerConnection.addCandidate(
RTCIceCandidate(
data['candidate'],
data['sdpMid'],
data['sdpMLineIndex'],
),
);
});
}
}
}
void sendPatientOffer(){
String json = jsonEncode(offer.toMap());
String message="OFFER:$json";
tellPatient(message);
sentOffer = true;
gotAnswer = false;
}
void sendPatientIceCandidate(Map iceCandidate){
String json = jsonEncode(iceCandidate);
String message="ICE_CANDIDATE:$json";
tellPatient(message);
}
void sendPatientAllIceCandidates(){
print("GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGOING TO SEND ICCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCE CANDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDIDATES");
for(var i = 0; i< iceCandidates.length; i++){
print("HOI HOI HOI HOI");
sendPatientIceCandidate(iceCandidates[i]);
}
}
void tellPatient(String message) {
Map<String, String> identifier = {
'channel': 'NotificationChannel',
'type': 'doctor',
'id': uniqueId
};
Map<String, String> data = {
'action': 'tell_patient',
'doctor_id': uniqueId,
'patient_id': patientId,
'chat_room_id': '999',
'message': message,
};
Map<String, String> request = {
'command' : 'message',
'identifier': jsonEncode(identifier),
'data':jsonEncode(data),
};
channel.sink.add(jsonEncode(request));
}
void registerAsDoctor() {
Map<String, String> identifier = {
'channel': 'NotificationChannel',
'type': 'doctor',
'id': uniqueId
};
Map<String, String> request = {
'command': 'subscribe',
'identifier': jsonEncode(identifier),
};
channel.sink.add(jsonEncode(request));
}
Future<void> createOffer(bool restart) async {
if(restart == true){
offer = await peerConnection.createOffer({"iceRestart" : true});
} else {
offer = await peerConnection.createOffer();
}
await peerConnection.setLocalDescription(offer);
}
Future<void> joinRoom(WebSocketChannel channel, String roomId, String patientId, RTCVideoRenderer remoteVideo) async {
iceCandidates = [];
SharedPreferences sp = await SharedPreferences.getInstance();
uniqueId = sp.getString("unique_id") ;
this.patientId = patientId;
this.channel = channel;
//register as a doctor
registerAsDoctor();
// setup listener for inbound messages
channel.stream.listen((event) {
// process inbound messages from the chat room
processWSMessage(event);
});
//create a peer connection
peerConnection = await createPeerConnection(configuration);
registerPeerConnectionListeners();
peerConnection.onSignalingState = (state) {
print("PEER CONNECTION STATE CHANGE: $state");
};
// on getting ice candidate tell the doctor that and store it
peerConnection.onIceCandidate = (RTCIceCandidate candidate) {
if (candidate == null) {
print('onIceCandidate: complete!');
return;
}
iceCandidates.add(candidate.toMap());
if (sentOffer==true && gotAnswer==true) {
sendPatientIceCandidate(candidate.toMap());
}
};
// create offer and start gathering ice candidates
await createOffer(null);
// add local stream to the peer connection
if(localStream != null){
localStream.getTracks().forEach((track) {
peerConnection.addTrack(track, localStream);
});
}
peerConnection.onTrack = (RTCTrackEvent event) {
print('@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@');
print('Got remote track: ${event.streams[0]}');
event.streams[0].getTracks().forEach((track) async {
await Future.delayed(Duration(microseconds: 200));
print('@@^^^^^^^^^^^^^^@@@@@@@@@@@@@@@@@@@@@');
print('Add a track to the remoteStream: $track');
remoteStream.addTrack(track);
});
};
// say hello to the patient
tellPatient("HELLO_PATIENT");
}
Future<void> openUserMedia(
RTCVideoRenderer localVideo,
RTCVideoRenderer remoteVideo,
) async {
var stream = await navigator.mediaDevices
.getUserMedia({'video': true, 'audio': false});
localVideo.srcObject = stream;
localStream = stream;
print("((((((((((((((((((((((");
print(localStream);
remoteVideo.srcObject = await createLocalMediaStream('key');
}
Future<void> hangUp(RTCVideoRenderer localVideo) async {
tellPatient("I_AM_LEAVING");
// List<MediaStreamTrack> tracks = localVideo.srcObject.getTracks();
// tracks.forEach((track) {
// track.stop();
// });
if (localStream != null) {
localStream.getTracks().forEach((track) => track.stop());
}
if (remoteStream != null) {
remoteStream.getTracks().forEach((track) => track.stop());
}
if (peerConnection != null) peerConnection.close();
channel.sink.close();
if(localStream != null) localStream.dispose();
if(remoteStream != null) remoteStream.dispose();
}
void registerPeerConnectionListeners() {
peerConnection.onIceGatheringState = (RTCIceGatheringState state) {
print('ICE gathering state changed: $state');
};
peerConnection.onConnectionState = (RTCPeerConnectionState state) {
print('Connection state change: $state');
// if(state == RTCPeerConnectionState.RTCPeerConnectionStateFailed){
// print("re negotiating");
// createOffer(true);
// sendPatientOffer();
// }
};
peerConnection.onSignalingState = (RTCSignalingState state) {
print('Signaling state change: $state');
};
peerConnection.onIceGatheringState = (RTCIceGatheringState state) {
print('ICE connection state change: $state');
};
peerConnection.onAddStream = (MediaStream stream) {
print("Add remote stream");
onAddRemoteStream.call(stream);
remoteStream = stream;
};
}
}
Callee (Patient)
import 'package:shared_preferences/shared_preferences.dart';
import 'package:web_socket_channel/io.dart';
import 'package:web_socket_channel/web_socket_channel.dart';
import 'package:flutter_webrtc/flutter_webrtc.dart';
typedef void StreamStateCallback(MediaStream stream);
class Signaling {
Map<String, dynamic> configuration = {
'iceServers': [
{
'urls': [
'stun:stun1.l.google.com:19302',
'stun:stun2.l.google.com:19302'
]
}
]
};
List<Map> iceCandidates = [];
List<Map> remoteIceCandidates = [];
RTCSessionDescription offer;
RTCPeerConnection peerConnection;
MediaStream localStream;
MediaStream remoteStream;
String roomId;
String uniqueId;
String doctorId;
String currentRoomText;
StreamStateCallback onAddRemoteStream;
WebSocketChannel channel;
bool gotOffer = false;
bool sentAnswer = false;
void processWSMessage(dynamic event) async {
var payload = jsonDecode(event);
if (payload["message"] != null && payload["type"] == null) {
String type = payload["message"]["type"];
String msg = payload["message"]["message"];
// listen for hello message and send ice candidates and offer details
if(msg.indexOf("HELLO_PATIENT") >= 0){
tellDoctor("HELLO_DOCTOR");
}
// listen for offers and send answer
if(msg.indexOf("OFFER") >= 0 && peerConnection.signalingState != RTCSignalingState.RTCSignalingStateHaveLocalOffer){
String offerFromDoctor = msg.split("OFFER:")[1];
// create answer
Map offer = jsonDecode(offerFromDoctor);
await peerConnection.setRemoteDescription(
RTCSessionDescription(offer['sdp'], offer['type']),
);
gotOffer = true;
var answer = await peerConnection.createAnswer();
await peerConnection.setLocalDescription(answer);
//send the answer
sendDoctorAnswer({'sdp': answer.sdp, 'type': answer.type});
//remember that we sent the answer
sentAnswer = true;
// send doctor all the ice candidates
sendDoctorAllIceCandidates();
//process any ice candidates
remoteIceCandidates.map((data){
peerConnection.addCandidate(
RTCIceCandidate(
data['candidate'],
data['sdpMid'],
data['sdpMLineIndex'],
),
);
});
}
// when you get ICE candidates from doctor
if(msg.indexOf("ICE_CANDIDATE")>=0){
String iceCandidate = msg.split("ICE_CANDIDATE:")[1];
Map data = jsonDecode(iceCandidate);
print('got ICE from doctor');
if(sentAnswer == true) {
peerConnection.addCandidate(
RTCIceCandidate(
data['candidate'],
data['sdpMid'],
data['sdpMLineIndex'],
),
);
} else {
//save ice candidates to be sent later
remoteIceCandidates.add(data);
}
}
}
}
void sendDoctorIceCandidate(Map iceCandidate){
String json = jsonEncode(iceCandidate);
String message="ICE_CANDIDATE:$json";
print("sending doctor ice candidate");
tellDoctor(message);
}
void sendDoctorAnswer(answer){
String json = jsonEncode(answer);
String message="ANSWER:$json";
tellDoctor(message);
}
void sendDoctorAllIceCandidates(){
print("going to send doctor all ice candidates");
for(var i = 0; i< iceCandidates.length; i++){
print("HOI");
sendDoctorIceCandidate(iceCandidates[i]);
}
}
void tellDoctor(String message) {
Map<String, String> identifier = {
'channel': 'NotificationChannel',
'type': 'patient',
'id': uniqueId
};
Map<String, String> data = {
'action': 'tell_doctor',
'doctor_id': doctorId,
'patient_id': uniqueId,
'chat_room_id': '999',
'message': message,
};
Map<String, String> request = {
'command' : 'message',
'identifier': jsonEncode(identifier),
'data':jsonEncode(data),
};
channel.sink.add(jsonEncode(request));
}
void registerAsPatient() {
Map<String, String> identifier = {
'channel': 'NotificationChannel',
'type': 'patient',
'id': uniqueId
};
Map<String, String> request = {
'command': 'subscribe',
'identifier': jsonEncode(identifier),
};
channel.sink.add(jsonEncode(request));
}
Future<void> joinRoom(WebSocketChannel channel, String roomId, String doctorId, RTCVideoRenderer remoteVideo) async {
iceCandidates = [];
SharedPreferences sp = await SharedPreferences.getInstance();
uniqueId = sp.getString("unique_id") ;
this.doctorId = doctorId;
this.channel = channel;
//register as a patient
registerAsPatient();
// setup listener for inbound messages
channel.stream.listen((event) {
// process inbound messages from the chat room
processWSMessage(event);
});
//create a peer connection
peerConnection = await createPeerConnection(configuration);
registerPeerConnectionListeners();
peerConnection.onSignalingState = (state) {
print("PEER CONNECTION STATE CHANGE: $state");
};
// on getting ice candidate tell the doctor that and store it
peerConnection.onIceCandidate = (RTCIceCandidate candidate) {
if (candidate == null) {
print('onIceCandidate: complete!');
return;
}
iceCandidates.add(candidate.toMap());
if (sentAnswer == true) {
sendDoctorIceCandidate(candidate.toMap());
}
};
// add local stream to the peer connection
if(localStream != null){
localStream.getTracks().forEach((track) {
peerConnection.addTrack(track, localStream);
});
}
peerConnection.onTrack = (RTCTrackEvent event) {
print('@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@');
print('Got remote track: ${event.streams[0]}');
event.streams[0].getTracks().forEach((track) async {
await Future.delayed(Duration(microseconds: 200));
print('@@^^^^^^^^^^^^^^@@@@@@@@@@@@@@@@@@@@@');
print('Add a track to the remoteStream: $track');
remoteStream.addTrack(track);
});
};
// say hello to the doctor
tellDoctor("HELLO_DOCTOR");
}
Future<void> openUserMedia(
RTCVideoRenderer localVideo,
RTCVideoRenderer remoteVideo,
) async {
var stream = await navigator.mediaDevices
.getUserMedia({'video': true, 'audio': false});
localVideo.srcObject = stream;
localStream = stream;
remoteVideo.srcObject = await createLocalMediaStream('key');
}
Future<void> hangUp(RTCVideoRenderer localVideo) async {
tellDoctor("I_AM_LEAVING");
// List<MediaStreamTrack> tracks = localVideo.srcObject.getTracks();
// tracks.forEach((track) {
// track.stop();
// });
if (localStream != null) {
localStream.getTracks().forEach((track) => track.stop());
}
if (remoteStream != null) {
remoteStream.getTracks().forEach((track) => track.stop());
}
if (peerConnection != null) peerConnection.close();
channel.sink.close();
if(localStream != null) localStream.dispose();
if(remoteStream != null) remoteStream.dispose();
}
void registerPeerConnectionListeners() {
peerConnection.onIceGatheringState = (RTCIceGatheringState state) {
print('ICE gathering state changed: $state');
};
peerConnection.onConnectionState = (RTCPeerConnectionState state) {
print('Connection state change: $state');
// if(state == RTCPeerConnectionState.RTCPeerConnectionStateFailed){
// hangUp(null);
// }
};
peerConnection.onSignalingState = (RTCSignalingState state) {
print('Signaling state change: $state');
};
peerConnection.onIceGatheringState = (RTCIceGatheringState state) {
print('ICE connection state change: $state');
};
peerConnection.onAddStream = (MediaStream stream) {
print("Add remote stream");
onAddRemoteStream.call(stream);
remoteStream = stream;
};
}
}
In one of the iterations of code where the ICE Candidates were not buffered, but simply shared as and when they were created, the doctor side was able to see the patient's video. The patient still couldnt see the dctor's video. Moreover, this only happened, if the patient was already connected to the signalling server (sort of like waiting for the doctor to call). If the doctor connected to the signalling server first and then the patient connected, either remote views were blank.
To give a brief outline of code, the caller (Doctor) connects to the signalling server and says "Hello patient" and waits for a hello from the patient. The patient on receiving a hello from the doctor, replies with "Hello Doctor".
Once the doctor gets the hello from the patient, he send the offer to the patient. Patient, on receiving the offer sends an answer back to the doctor. ICE Candidates gathered between these messages are buffered and shared once the doctor has received the answer.
I have read that the webrtcpeerconnection object is kind of like a state machine and hence there is some order in which things should happen. If that's the case, is my order of exchange of messages wrong? or Is my understanding flawed?
I am a novice in both Flutter and WebRTC. This is more like a hobby project, so please excuse any coding stupidity from my end.
I did some tinkering and further digging. Various sources helped. The following factors lead me to the solution.
All the above seemed to have helped me in making the video chat work. I will list the resources that I feel helped me a lot.
Remote VideoStream not working with WebRTC
Youtube Crash course on webrtc by Hussain Nasser