SmoothStreamingMediaElement for Live

Mar 8, 2012 at 11:31 PM

Hi All,

I'm using SmoothStreamingMediaElement for my client side application so that video + data (caption) can be streaming. This work perfect for archive video + data but not for live.

Here is how I did, I follow this article http://msdn.microsoft.com/en-us/library/ee958034%28v=vs.90%29.aspx to handler caption (data) as Timeline Markers.

When the media open I use this event hander

 SmoothPlayer = new SmoothStreamingMediaElement
            {
                AutoPlay = false,
                SmoothStreamingSource = new Uri(_streamingUrl),
            };

SmoothPlayer.MediaOpened +=
                     new RoutedEventHandler(SmoothPlayer_MediaOpened);

The StreamIndex element in the manifest that contains markers is identified in code as a Script stream

void SmoothPlayer_MediaOpened(object sender, RoutedEventArgs e)
    {
        foreach (SegmentInfo segmentInfo in SmoothPlayer.ManifestInfo.Segments)
        {
            List<StreamInfo> selectStreams = segmentInfo.SelectedStreams.ToList<StreamInfo>();
            foreach (StreamInfo streamInfo in segmentInfo.AvailableStreams)
            {
                if (streamInfo.Type == System.Windows.Media.MediaStreamType.Script)
                {
                    if (streamInfo.Attributes["Name"] == "ClosedCaptions" ||
                        streamInfo.Attributes["Name"] == "MARKERS")
                    {
                        selectStreams.Add(streamInfo);
                        segmentInfo.SelectStreamsAsync(selectStreams);

                        foreach (TrackInfo trackInfo in streamInfo.SelectedTracks)
                        {
                            foreach (ChunkInfo chunk in streamInfo.ChunkList.ToList<ChunkInfo>())
                            {
                                IAsyncResult ar =
                                    trackInfo.BeginGetChunk(
                                    chunk.TimeStamp, new AsyncCallback(AddMarkers), streamInfo.UniqueId );
                            }
                        }
                    }
                }
            }
        }
    }

The previous code example calls the BeginGetChunk method and specifies the AddMarkers callback method to handle the results. The AddMarkers method is shown below. It calls the EndGetChunk method and initializes the markers.


private void AddMarkers(IAsyncResult argAR)
    {
        if (!Deployment.Current.Dispatcher.CheckAccess())
        {
            Deployment.Current.Dispatcher.BeginInvoke(() => AddMarkers(argAR));
        }

        foreach (SegmentInfo segmentInfo in SmoothPlayer.ManifestInfo.Segments)
        {
            foreach (StreamInfo streamInfo in segmentInfo.SelectedStreams)
            {
                if (streamInfo.UniqueId == ((string)argAR.AsyncState))
                {

                    List<ChunkInfo> markerChunks = streamInfo.ChunkList.ToList<ChunkInfo>();

                    foreach (TrackInfo trackInfo in streamInfo.SelectedTracks)
                    {
                        ChunkResult chunkResult = trackInfo.EndGetChunk(argAR);

                        if (chunkResult.Result == ChunkResult.ChunkResultState.Succeeded)
                        {
                            System.Text.Encoding enc = System.Text.Encoding.UTF8;
                            int length = (int)chunkResult.ChunkData.Length;
                            byte[] rawData = new byte[length];
                            chunkResult.ChunkData.Read(rawData, 0, length);
                            String text = enc.GetString(rawData, 0, rawData.Length);
                            TimelineMarker newMarker = new TimelineMarker();
                            newMarker.Text = text;
                            newMarker.Time = chunkResult.Timestamp;

                            SmoothPlayer.Markers.Add(newMarker);
                        }
                    }
                }
            }
        }
    }

So here the captions (data) are added to SmoothPlayer as markers collection. Then I need to handle MarkerReached event

After the TimeLineMarker objects are initialized in the Markers collection, application code can catch and respond to timeline events identified by markers

To do this, assign a delegate to handle the MarkerReached event as shown in the following:

SmoothPlayer.MarkerReached +=
      new TimelineMarkerRoutedEventHandler(SmoothPlayer_MarkerReached);

void SmoothPlayer_MarkerReached(object sender, TimelineMarkerRoutedEventArgs e)
  {
      OutputText.Text = "  Marker Text: " + e.Marker.Text ;

  }

Like I said this works perfect for the case of archive video + data. The client get all the caption (data) upfront and then just add the caption to smoothplayer as marker collection.

And this is not the case for live, since during live video + data (caption) are not known upfront.

My question is: how can I hand live video + data using SmoothStreamingMediaElement or is there another control that can help to handle live video + data?

Please help to share.

Thanks

Truyenle

 

Mar 9, 2012 at 8:20 AM

Hi

I am wondering why you havent implemented this logic as a IMarkerProvider, is there any particular reason for that?

- Loke

Mar 9, 2012 at 3:42 PM

Thank for your fast response.

Still a newbie in smoothstreaming so I'm not sure that I can get your point. Can you elaborate a little bit more of what is IMarkerProvider? Is it an interface that help to get the live data? Any example code would be very much appreciated.

Sorry for my newbie question?

Thanks

Truyen

Mar 13, 2012 at 4:42 PM

Hi Truyen

IMarkerProvider is the plugintype that can supply captions. Thats the way we handle .SMI captions for our player.

You should try and download the samples and look at the MarkerProviderExample.

Also check out the Plugin developer guide at http://smf.codeplex.com/releases/view/51263#DownloadId=150269

- Loke

Mar 13, 2012 at 4:46 PM

That's super fast response. Thank, I'll look that up now. I will update back the progress.

Mar 14, 2012 at 6:24 PM

Hi lokexstream,

I have read the document and it seem that it is apply for SMF not for SmoothStreamingMediaElement (SSME)? right?

Thanks

Truyen

Mar 15, 2012 at 9:58 AM

I am not entirely sure what you mean. But this applies to using SMF with Smooth Streaming yes.

- Loke

Mar 15, 2012 at 10:32 AM

If you want to use the naked SSME, you can re-implement what's in SMF, especially as the source is available, but it seems a waste.

Else, you have the MarkerReached event to handle:

http://msdn.microsoft.com/en-us/library/microsoft.web.media.smoothstreaming.smoothstreamingmediaelement.markerreached%28v=vs.90%29.aspx

Mar 15, 2012 at 3:31 PM

Thank you lokexstream and Julienv42 for the responses.

There is some unknown reasons that the project lead decide to go with SSME in prism 4.0 framework and he already left. So I'm trying to continue using naked SSME to finish it.

I got a little bit progress for the video yesterday as:

 SmoothPlayer = new SmoothStreamingMediaElement
            {
                AutoPlay = false,
                SmoothStreamingSource = new Uri(http://server/live.isml/Manifest),
            };

The live video is working now but the caption is the issue. In this live stream there is caption and I don't know how to get these caption live.

@Julienv42: I am using that MarkerReched Event but before the event fired when time coming, we have to retrieve the markers and add it to the SmoothPlayer. Something like:

     TimelineMarker newMarker = new TimelineMarker();
      newMarker.Text = stringValue;
      newMarker.Time = timeSpanValue;
      SmoothPlayer.Markers.Add(newMarker);

So the issue here is how to get the markers in case of live?

The article in my first post http://msdn.microsoft.com/en-us/library/ee958034%28v=vs.90%29.aspx only show how to get these Markers in case of OnDemand that mean all the markers are known upfront. I try the same logic with live and it didn't work.

Any suggestion on how to achieve this, please help.

Thank you

Mar 15, 2012 at 3:54 PM

Hi

I misunderstood that you were using raw SSME.

You could probably look at what the ClearTextCaptions plugin does for SMF. Basically it listens for the players DataRecieved event and then does:

        private void Player_DataReceived(object sender, DataReceivedInfo e)
        {
            if (IsCaptionStream(e.StreamAttributes))
            {
                try
                {
                    var stream = new MemoryStream(e.Data);
                    var text = new StreamReader(stream).ReadToEnd();
                    var duration = e.DataChunk.Duration != TimeSpan.Zero
                                        ? e.DataChunk.Duration
                                        : TimeSpan.FromMilliseconds(2002);

                    if (_captionRegion.Begin == TimeSpan.MinValue || _captionRegion.Begin > e.DataChunk.Timestamp)
                    {
                        _captionRegion.Begin = e.DataChunk.Timestamp;
                    }

#if SILVERLIGHT3
                    if (!SystemExtensions.IsNullOrWhiteSpace(text))
#else
                    if (!string.IsNullOrWhiteSpace(text))
#endif
                    {
                        var caption = new CaptionElement
                        {
                            Content = text,
                            Begin = e.DataChunk.Timestamp,
                            End = e.DataChunk.Timestamp.Add(duration)
                        };

                        _captionRegion.Children.Add(caption);
                    }
                }
                catch (Exception err)
                {
                    Debug.WriteLine(err.Message);
                }
            }
        }

Instead of adding to a captionRegion you could then add it as a marker i think.

- Loke

Mar 15, 2012 at 4:47 PM

I'm not sure what you want is easily feasible. Can't you inject the caption into the stream rather than have them apart?

Mar 15, 2012 at 5:28 PM

Hi Loke,

Sorry to confusing you, I'm still continueing using raw SSME since we don't have enough time to switch it to SMF. The project code is so much now. But down the road, if I can't find a way to get caption in case of Live with raw SSME, then I'll follow your very helpful instruction.

@Julienv42: Can't you inject the caption into the stream rather than have them apart? Not sure if I understand this correctly

The .isml/manifest is as below which does has StreamIndex of type text, does it mean that the stream has caption already?

<SmoothStreamingMedia MajorVersion="2" MinorVersion="0" Duration="0" TimeScale="10000000" IsLive="TRUE" LookAheadFragmentCount="2" DVRWindowLength="0" CanSeek="TRUE" CanPause="TRUE">

<StreamIndex Type="audio" Name="audio" Subtype="AACL" Chunks="0" TimeScale="10000000" Url="QualityLevels({bitrate})/Fragments(audio={start time})">

<QualityLevel Index="0" Bitrate="64000" CodecPrivateData="1210" FourCC="AACL" AudioTag="255" Channels="2" SamplingRate="44100" BitsPerSample="16" PacketSize="4"/>

<c t="0" d="20201360"/>

<c d="19969161"/>

<c d="19969161"/>

...

</StreamIndex>

<StreamIndex Type="video" Name="video" Subtype="H264" Chunks="0" TimeScale="10000000" Url="QualityLevels({bitrate})/Fragments(video={start time})">

<QualityLevel Index="0" Bitrate="603000" CodecPrivateData="000000016764001EAC2CAC1E0BFE5FFC100010014830303200000300020000030078C040012668000499BFF8C718080024CD00009337FF18E1DA1225380000000168CA535250" FourCC="H264" MaxWidth="480" MaxHeight="360"/>

<c t="0" d="20293226"/>

<c d="20028527"/>

<c d="20018103"/>

<c d="20018307"/>

...

</StreamIndex>

<StreamIndex Type="text" Name="textstream" Subtype="CAPT" Chunks="0" TimeScale="10000000" Url="QualityLevels({bitrate})/Fragments(textstream={start time})">

<QualityLevel Index="0" Bitrate="32000" CodecPrivateData="" FourCC="TTML"/><c t="0" d="20293226"/>

<c d="20028527"/>

<c d="20018103"/>

<c d="20018307"/>

...

</StreamIndex>

</SmoothStreamingMedia>

Thank guys for bare with me.

 

Mar 16, 2012 at 9:19 AM

Yes, those streamindex means it has captions instream. Try and look at the code i pasted, you should be able to use that with raw SSME. Just hook it up to the Player.DataRecieved event and use that to get it as markers if needed.

- Loke

Mar 16, 2012 at 3:32 PM

Hi lokexstream,

SSME doesn't have DataReceived event? http://msdn.microsoft.com/en-us/library/microsoft.web.media.smoothstreaming.smoothstreamingmediaelement%28v=vs.90%29.aspx

 

Truyen

Mar 19, 2012 at 10:31 AM

Hi truyenie

You are absolutely right, that is implemented by SMF, my fault, sorry.

- Loke

Mar 19, 2012 at 2:10 PM

Again, I would suggest to look at the sources of SMF for the functionality you want. SMF is a quite big piece, but it's structured well enough that you should be able to find the relevant parts and take inspiration from them.

Mar 19, 2012 at 3:37 PM

Thank guys, I'll look at it a bit further and If I can't figure out, will switch to smf then.

Sep 3, 2012 at 7:40 PM

I guess there's a separate forum too for SmoothStreaming at MS (or maybe at answers.microsoft.com or something)