Using streamlit-webrtc with STUN/TURN Azure AI avatar service that requires avatar config, SDP exchange etc.

[corticalstack]

Hi,

I'm just wondering if it's possible to use streamlit-webrtc to stream one-way the video/audio tracks from the Azure AI Avatar service.

Example code provided by Microsoft, for example chat.js which is used in a .html page, makes a request to the AI avatar service with a generated SDP offer and additional config, which responds with an SDP answer, which in turn is used to create the RTC peer connection. A call to the AI avatar service with the ICE and sdp configuration looks like below:

`try:
speech_config = speechsdk.SpeechConfig(subscription=self.speech_key, endpoint=f'wss://{self.speech_region}.tts.speech.microsoft.com/cognitiveservices/websocket/v1?enableTalkingAvatar=true')

            speech_synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config, audio_config=None)

            avatar_config = {
                'synthesis': {
                    'video': {
                        'protocol': {
                            'name': "WebRTC",
                            'webrtcConfig': {
                                'clientDescription': sdp,
                                'iceServers': [{
                                    'urls': [self.ice_token['Urls'][0]],
                                    'username': self.ice_token['Username'],
                                    'credential': self.ice_token['Password']
                                }]
                            },
                        },
                        'format':{
                            'crop':{
                                'topLeft':{
                                    'x': 0,
                                    'y': 0
                                },
                                'bottomRight':{
                                    'x': 1920,
                                    'y': 1080
                                }
                            },
                            'bitrate': 1000000
                        },
                        'talkingAvatar': {
                            'customized': False,
                            'character': self.avatar_character,
                            'style': self.avatar_style,
                            'background': {
                                'color': self.avatar_background_color,
                            }
                        }
                    }
                }
            }
            
            connection = speechsdk.Connection.from_speech_synthesizer(speech_synthesizer)
            connection.set_message_property('speech.config', 'context', json.dumps(avatar_config))

            speech_sythesis_result = speech_synthesizer.speak_text_async('').get()
            print(f'Result id for avatar connection: {speech_sythesis_result.result_id}')
            if speech_sythesis_result.reason == speechsdk.ResultReason.Canceled:
                cancellation_details = speech_sythesis_result.cancellation_details
                print(f"Speech synthesis canceled: {cancellation_details.reason}")
                if cancellation_details.reason == speechsdk.CancellationReason.Error:
                    print(f"Error details: {cancellation_details.error_details}")
                    raise Exception(cancellation_details.error_details)
            turn_start_message = speech_synthesizer.properties.get_property_by_name('SpeechSDKInternal-ExtraTurnStartMessage')
            remoteSdp = json.loads(turn_start_message)['webrtc']['connectionString']
            print(f"Remote SDP: {remoteSdp}")

            return Response(content=remoteSdp, status_code=200)
        except Exception as e:
            return Response(content=f"Error message: {str(e)}", status_code=400)` 

As streamlit-webrtc seems to abstract away much of the low-level WebRTC setup, so we don't directly work with SDP objects, with the library handling the SDP exchange, ICE candidate gathering, and peer connection setup internally, I'm wondering if it's possible to use the component in this way with a third-party STUN/TURN server that requires additional config (i.e. the sdp, avatar config).

Thanks for any hints.

[corticalstack]

Took the test app a bit further. Steps are:

1. Create streamlit-webrtc client
2. Create an Azure speech synthesizer connection and update with the avatar config.

Instantiating the webrtc client is successful and shows as playing. The logging outputs the SDP offer generated by streamlit-webrtc.

When the streamlit-webrtc.WebRtcMode is RECVONLY, I don't get any streaming content in the video player element. Both video and audio tracks are set to "inactive" in the answer SDP, e.g.:
m=video 9 UDP/TLS/RTP/SAVPF 96 97 102 103 106 107
c=IN IP4 0.0.0.0
a=inactive

When the WebRtcMode is SENDRECV, it shows my webcam, not the avatar video streaming service:
m=video 9 UDP/TLS/RTP/SAVPF 96 97 102 103 106 107
c=IN IP4 0.0.0.0
a=sendrecv

import streamlit as st
import streamlit_webrtc
import json
import av
import streamlit_webrtc.component
from streamlit_webrtc.session_info import SessionInfo
import azure.cognitiveservices.speech as speechsdk

import logging
logging.basicConfig(level=logging.DEBUG)

def video_frame_callback(frame):
    img = frame.to_ndarray(format="bgr24")
    logging.info(f"Received video frame: shape={img.shape}, dtype={img.dtype}")
    return av.VideoFrame.from_ndarray(img, format="bgr24")

def main():
    st.title("WebRTC Avatar Connection Example")

    webrtc_ctx = streamlit_webrtc.webrtc_streamer(
        key="example",
        mode=streamlit_webrtc.WebRtcMode.SENDRECV,
        rtc_configuration={
            "iceServers": [{
                "urls": ["turn:relay.communication.microsoft.com:3478"],
                "username": "<username>",
                "credential": "<password>"
            }]
        },
     media_stream_constraints={
        "video": {"deviceId": "default"},
        "audio": True
    },
    video_receiver_size=1,  # Expect one video track
    async_processing=True,
    )

    speech_key = "<speech key>"
    speech_region = "westeurope"
    avatar_character="lisa"
    avatar_style="casual-sitting"
    avatar_background_color ='#FFFFFFFF'

    if webrtc_ctx.state.playing:
        st.write("WebRTC connection established!")
        st.write(f"WebRTC state: {webrtc_ctx.state}")
        st.write(f"Video receiver: {webrtc_ctx.video_receiver}")
        st.write(f"Audio receiver: {webrtc_ctx.audio_receiver}")
        speech_config = speechsdk.SpeechConfig(subscription=speech_key, endpoint=f'wss://{speech_region}.tts.speech.microsoft.com/cognitiveservices/websocket/v1?enableTalkingAvatar=true')
        speech_synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config, audio_config=None)

        # Configure the avatar
        avatar_config = {
            'synthesis': {
                'video': {
                    'protocol': {
                        'name': "WebRTC",
                        'iceServers': [{
                            "urls": ["turn:relay.communication.microsoft.com:3478"],
                            "username": "<user name>",
                            "credential": "<password>"
                        }],
                    },
                    'format':{
                        'crop':{
                            'topLeft':{
                                'x': 0,
                                'y': 0
                            },
                            'bottomRight':{
                                'x': 1920,
                                'y': 1080
                            }
                        },
                        'bitrate': 1000000
                    },
                    'talkingAvatar': {
                        'customized': False,
                        'character': avatar_character,
                        'style': avatar_style,
                        'background': {
                            'color': avatar_background_color,
                        }
                    }
                }
            }
        }

        st.write(f"Avatar config: {avatar_config}")

        # Connect to the avatar service
        connection = speechsdk.Connection.from_speech_synthesizer(speech_synthesizer)
        connection.set_message_property('speech.config', 'context', json.dumps(avatar_config))
        speech_sythesis_result = speech_synthesizer.speak_text_async('').get()
        print(f'Result id for avatar connection: {speech_sythesis_result.result_id}')
        if speech_sythesis_result.reason == speechsdk.ResultReason.Canceled:
            cancellation_details = speech_sythesis_result.cancellation_details
            print(f"Speech synthesis canceled: {cancellation_details.reason}")
            if cancellation_details.reason == speechsdk.CancellationReason.Error:
                print(f"Error details: {cancellation_details.error_details}")
                raise Exception(cancellation_details.error_details)

        st.write("Avatar service configured and connected!")
    else:
        st.write("WebRTC connection not yet established.")

if __name__ == "__main__":
    main()