Thursday, September 22, 2011

WP7 Marketplace share (or how we became a victim of our own success)

This post tells the story of how good faith estimates of our WP7 marketplace penetration were under-reported by 500%. This is not a “gotcha blog” – there are only good actors with the best of intentions in this story; but that’s why I think it’s a story worth telling.

You see, we don’t just obfuscate – we hide the fact that your app is obfuscated. We don’t just offer application instrumentation and monitoring, we inject that logic to simplify and streamline packaging and improve performance. …and therein lies the rub.

Something was in the air

While at Build soaking up the heady atmosphere of Windows 8 and all that’s coming with it, a few MVP’s and also the Lowdermilk brothers took a moment to ask me if I’d read this awesome blog post where a developer had downloaded and analyzed all of the XAPs in WP7 marketplace. Apparently the market insight was killer and covered everything from the most popular libraries to snagging cloned apps. No more guessing – the facts were all laid out. And they each mentioned to me how surprised and even a little worried they were by how few apps were being obfuscated. I didn’t think too much of it at the time as I am among the first to point out that not every app needs to be obfuscated – but I did make a mental note to be sure to check out the blog.

When I got back home, I finally had a chance to track this “all seeing blog” down – it was Justin Angel’s blog and the post was Windows Phone 7 Marketplace Statistics. And it really is a fascinating post with both initiative and insight; and then I got to the obfuscation section.

According to Justin’s analysis, only 3% of the apps in the market were obfuscated! And as I scanned down, there were comments to the effect of “gee, since no one else is doing it, perhaps I shouldn’t bother either.”

Even more surprising to me was the fact that our analytics (Runtime Intelligence) was not even listed in a very long list of third party tools – when I knew for a fact that we have nearly a thousand apps sending data.

This can’t be right! (and I was)

Given the nearly 6,000 downloads of Runtime Intelligence and Dotfuscator for WP7 and the activity that I had been seeing over the past year, these numbers just didn’t seem right. I wrote to Justin who was quick to share his detection logic (in fact he posted the source on his blog) and just as quick to invite any comments or refinements that I might have to offer.

To put a fine point on this, Justin was in no way defensive and was as interested in getting to the right answer as I was.

Hung by our own petard

Without going into a lot of detail, Justin’s approach was to bust open the XAP and examine the various files and manifests to separate Silverlight from XNA and to identify the presence of third party tools. This approach proved to be effective because frameworks, tools and components leave behind files and other telltale fingerprints as a matter of course. There is one limitation though; this approach cannot detect when an application is modified or extended through IL manipulation or injection. And that’s exactly what we do.

From Build to Bill’ed (new numbers and how)

As I like to say, my ideas only have to be good, they don’t have to be original (trademark and patent laws not withstanding) and so I did what I often do when confronted with a conundrum – I asked Bill Leach, our CTO, for help. He quickly (dare I say magically?) authored our own “marketplace crawler” that populated our own XAP repository. Rather than look at XAP contents at a component level, he wrote some code that examined the binaries themselves.

The first pass looked for the custom attribute DotfuscatorAttribute inside the binaries. This is a good way (but not an absolute way) to determine if a file has been processed by Dotfuscator (for either obfuscation or injection of analytics). It’s not infallible because developers can remove that attribute if they chose (to further cloak the fact that they have used Dotfuscator). Here is what we found:

We downloaded 26,159 XAP files and 14.5% to have been processed by Dotfuscator.

This is basically 5X as many apps as Justin’s analysis had found (and that does not include the developers who configured Dotfuscator to remove the attribute we were searching for – so the number is certainly a bit higher).

In fact, we were surprised that Justin had found any at all – where did his 3% come from? Upon inspection, we think it’s an unexpected side-effect of how XAP’s are assembled – there are some instances where the configuration file of Dotfuscator gets pulled into the XAP – this is unnecessary and should never happen. We will document this behavior and make sure that users know how to prevent this from happening. In short, his 3% showed the prevalence of a bug – not the use of Dotfuscator.

To determine if an application was instrumented (rather than obfuscated), we applied some heuristics that are less obvious but can be shared if someone is interested (we looked for the existence of some high-level classes).

2.6% of marketplace apps are instrumented.

From my perspective, this is a low number – but to put it in perspective (or let’s be honest – I’m looking for the silver lining) we have a larger share than Google Analytics and Admob but a slightly lower number than Flurry.

Attack of the clones

Just one more point to be made in this post. If one were to consider each family of cloned apps as essentially a single, re-skinned app – these numbers have the potential to change materially. We may take a look at that, but I think we have already gotten most of what we can from the static analysis of the marketplace.

So – is that the whole story? (Of course not)

Don’t give me (just) static

As interesting as the static analysis of the WP7 marketplace is (and it is), static analysis only gives us a backwards facing snapshot of what’s already been deployed. We get no insight into:

  • Best practices that we would want to replicate (which are different that common practices),
  • Developer motivations behind their development choices
  • Future trends especially when driven by new technology and market opportunities
In the context of Dotfuscator and Runtime Intelligence, I would want to know
  • If the developers who built the 14.5% of “Dotfuscator processed” apps leaders or laggards?
  • Do they have special requirements that set them apart?
  • In short, do they have anything to teach the rest of us?
Want to know more about the 14.5% apps and what they have to teach us?

Coming soon, WP7 Developer Survey Results. We’ve been running surveys since the WP7 launch last year (you can checkout survey 1 and survey 2). As part of this ongoing effort, we have just closed out our third survey in the series and I will be posting results in the next few days – stay tuned!

2 comments:

Anonymous said...

Dotfuscator isn't the only WP7 obfuscator. My WP7 apps all use CryptoObfuscator and wouldn't count towards the obfuscation stats because it doesn't generate the Dotfuscator attribute.

Ken Morse said...

Thanks for sharing this info, Sebastian -- very interesting.