Custom MediaStreamSources

Jul 8, 2010 at 10:29 AM
Hi everybody, I'm wondering if the team has any plans to implement custom MediaStreamSources to provide support for additional media formats. The reason I'm asking is that my media library consists of quite a few different media containers and I'd rather not transcode my files if I don't have to. I've seen an few .NET projects that already parse the containers and given the fact Silverlight plays WMV, VC-1 and h.264 it could be a real opportunity to widen the platform. Did you know that Silverlight doens't play mp4 files with VC-1 streams? Also, the highly popular MKV container isn't supported at all .. :( Let me know what you guys think :)
Jul 8, 2010 at 6:03 PM

Hi Fegelein, we don't have any immediate plans for implementing custom MediaStreamSources.  However, this sounds like a good use case for a new type of Media Plugin.  Have you considered writing a plugin to meet these requirements?

Jul 8, 2010 at 8:52 PM

Hi Krohling,

I'm working on a personal project that manages and plays all my media. Fortunately I'm using WPF so I don't really need to worry about this problem because it uses Windows codecs. The project does have some code to extract subtitles from Matroska containers and I can tell you that format wasn't easy to parse :(. If I have something to share in the future I'll most definitely will, but that isn't the case right now.

Sorry :(

Jul 11, 2010 at 9:58 PM

This post sparked me to spend most of my weekend delving deep into the SMF code. I wanted to make a custom media plugin to support SilverSynth . I ran into many issues that make it rather difficult to add any new media plugins to SMF, particularly when trying to include multiple plugins in one app.

First issue I ran into is SMF relies on Uri for the mediasource. No support for Streams or a CustomMediaSource implementation. This completely stopped me from even moving further with that idea. SilverSynth has no defined file formats.

After that I investigated how SMF figures out what plugin to use and was surprised with what I found. It begins by querying the plugins of what DeliveryMethods they support. I found that if the PlayListItem is DeliveryMethod="NotSpecified" it will just pick the first plugin instead of any plugins that actually declared NotSpecified as a supported type. I tried to create a plugin that just supported NotSpecified and it was never selected. Later on I found that even when the plugin has metadata that matched the playlist item it still wasn't selected.

The documentation made it a little confusing on how to add metadata to your plugin. "You add an Attribute to your class as metadata that indicates the plugin capabilities (using the ExportMedia Attribute)." After looking further I figured that this actually meant to create a new attribute that inherited from ExportMediaPluginAttribute. Any public properties on your custom attribute will be come metadata. After doing this and adding the matching metadata entries to my PlayListItem it finally selected my plugin. I found it strange as well that the PluginMediaRequiredMetaData values have to match both key and value, I had wrongly assumed that this MetaData would require the keys and use the values as information.

Even if a plugin is part of the initial list of plugins for the playlistitem SMF then gives weight to plugins that have already been created so it's a crapshoot if you want your plugin to be used.

Some problems I see with the process

  1. It is difficult for someone to know what MetaData a plugin needs and how it will be used
  2. the JS api doesn't expose the metadata property at all
  3. DeliveryMethod is a poor indicator for what a plugin supports. (same delivery methods, different file formats). Having to declare metadata and then having the playlistitem match key and value is complex and confusing.

I personally think the way this is done is backwards. I think the plugin should have a say in what media it can play with as little effort on the consumers end. It should be given some evidence and allowed to tell SMF if it can support the item or not. There is little way for me to say create a plugin that technically supports "Progressive" delivery but only supports the .ogg file format. While still supporting wmv via the Progressive plugin. On both the plugin and consumer end I'd need to say "hey this is an ogg file" even though the file extension already does. Instead why couldn't the plugin be given the Uri and asked "do you support this format?". This would simplify things quite a bit on the consumer end and when it comes down to it those are the people that should have it easy.

With relying on DeliveryMethods it makes it so the consumer has to determine per PlayListItem how it should be downloaded. What if they don't know that? Like in the project I was just on my client's API returned urls that mixed between MMS, HTTP, IIS Smooth Streaming and IIS Live Smooth Streaming. The client's system had no data on what they were other than the url. SMF V1.1 handled being fed these urls just fine. Now I'm going to have to write a system to determine from the url the DeliveryMethod, thus complicating my end.

I'm curious as to why the "Data" of SMF is no longer in it's own library? It would be nice if the plugin could have access to more than just the Uri/Stream/MediaSource . So custom information for the particular media can be passed to the plugin: Alternative download sources, tweaks for rendering.

Some things I've been thinking that are possible for Silverlight to gain support of is MKV, F4V, FLV, Adobe Adaptive Streaming from 3rd parties. Support for MKV and F4V wouldn't be too far fetched since both can be merely containers for h.264. There is someone already claiming to have an FLV implementation for Silverlight. With the current state of SMF MediaPlugin selection it is difficult to mix multiple types.

Here are some of my thoughts on what could be done in SMF

  1. Add 2 new properties to the PlayListItem for MediaStreamSource and MediaStreamSource . I think it's fine that the JS api wouldn't be able to see these
  2. Add a way of passing "Evidence" to a plugin allowing it to determine if it can support the media. Like if the MediaStreamSource is of a particular class type or the Uri is a certain filetype. For what I can think of the different sources, the delivery methods and the metadata collection would be sufficient.
  3. Maybe instead of passing evidence the "fun" way could be defining regex patterns for urls and declaring the CustomMediaStream types supported
  4. Repurpose the MetaData collection or add a new dictionary property to be able to pass custom information to the plugin if it is really needed.
Jul 12, 2010 at 7:35 PM

Hey darthobiwan, first thanks for the feedback.  Let me try to address your points one at a time and please, let me know if I've missed anything:

  • Media Plugin selection process 

You have described the process for selecting a Media Plugin, but I'll recap for clarity:
1. Filter available plugins on supported Delivery Method. If “NotSpecified” Delivery Method is ignored.

2. Filter on PlaylistItem.MediaPluginRequiredMetadata (custom metadata key/value pairs)

3. Filter on whether LiveDvr support is required

4. From the remaining list, select 1, favoring those that have already been instantiated


The goal of this process is not to facilitate the selection of a “specific” media plugin, but instead to simply pick one that will do the job. I wouldn’t use the word “crapshoot” but, you are correct in that you are not guaranteed that your plugin will be selected if there are others available that have equivalent capabilities. In cases where a specific plugin or more complex filtering is required (i.e. supported containers, etc) use PlaylistItem.MediaPluginRequiredMetadata.


  • Why not use the URL to determine the media type and just use that to pick the right plugin?

This is something that we thought about at one point in time but eventually decided the URL isn’t reliable. On the web there’s no requirement that the file extension be part of the string (think URL shortener), also the “content-type” HTTP header isn’t much better in this respect. Making this a viable (and reliable) solution would be very challenging and potentially error prone, so instead, we decided on DeliveryMethod. While I agree this does not cover every case, that was not the intent, we wanted simple. We could have expanded on this, made it more robust, but we believed that would make it more complicated for the average developer. Instead, as mentioned above, for more complex scenarios we support the specification of metadata on the PlaylistItem and the MediaPlugin.

Per your comment about SMF v1.1, this is not something that was supported in that version of SMF either. There was no need for “DeliveryMethod” because there were no plugins to pick from, it was coupled to the SmoothStreamingMediaElement. In v2, this property only serves to filter available plugins. If you only have 1 plugin available in your XAP, don’t bother with it, the default is “not specified” so it will be ignored and that 1 plugin will always be used. That said, there’s no reason for you to write a system to determine “DeliveryMethod” from a URL if everything worked fine in v1.1.


  • Why not pass “Evidence” to the plugin allowing it to determine if it supports the media?

This is also something we considered, perhaps adding an IMediaPlugin.SupportsPlaylistItem(PlaylistItem) method? Instead we decided to allow the plugins to expose their “evidence” using MEF attributes and allow the Player to query over the body of evidence that is available at that time. One of the reasons for this was performance. Instead of instantiating each plugin, which we would be required to “pass it evidence”, we use MEF’s built in support for Lazy<T, TMetadata> which means we don’t have to instantiate anything until it’s selected.



  • PlaylistItem support for MediaStreamSource

This is probably a good idea, and one that several people have now expressed an interested in seeing.  I'll make sure this is part of the feature discussion for our next release.


  • Using PlaylistItem.MediaPluginRequiredMetadata

This property on the PlaylistItem allows you to specify key/value pairs that the player compares to the metadata that is exposed on available plugins. The easiest way to expose metadata on a plugin is simply to add an ExportMetadataAttribute with a matching key/value pair, like this:

[ExportMetadata("MyFavoriteFruit", "MangosAreAwesome")]

If a PlaylistItem has matching metadata you can be sure this plugin will be selected:



       <smf_util:MetadataItem Key="MyFavoriteFruit" Value="MangosAreAwesome"/>