Monday, September 26, 2011

Runtime Intelligence and Dotfuscator for WP7 developers speak (Mikey likes it!)

In my last post, I drew a correlation between apps that used Runtime Intelligence and their relative (positive) success as measured by user ratings and engagement. While it was fairly clear that developers who chose to use Runtime Intelligence built more successful apps than their counterparts, it really said nothing about a) Runtime Intelligence analytics’ contribution to their success or b) developer satisfaction with Runtime Intelligence overall.

Well, there’s really only one way to answer these questions …and that’s to ASK THEM.

I sent out an electronic survey starting on Monday of last week (September 19) and have received over 200 developer responses. Here is what they said…

Who is doing what?
  • 32% indicated that they are only using Dotfuscator to protect their application
  • 24.5% said they were only using Runtime Intelligence
  • And 43.4% indicated that they were using both.

Do these “smarter than the average bear” developers see the value?

In a word, YES.

Looking only at those developers who indicated that they already had their applications in the marketplace (representing over 100 development organizations):
  • 60% indicated that analytics set the wp7 platform apart from all other platforms or added significant value to the platform.
  • 68% indicated that protection set the wp7 platform apart from all other platforms or added significant value to the platform.

Analytics’ perceived value increases by 450% with developer experience

When looking at those developers that indicated that analytics and/or protection “set the WP7 development platform apart from all others,” analytics’ value actually increased by 450% (from 2% to 9%) as developers moved from no app, to less than four weeks to a ship date, to actually having an app in the marketplace (and getting analytics back). Interestingly, obfuscation (protection) peaked in value just prior to shipping.

So what’s the takeaway?

In my last post, we established the user of Runtime Intelligence were more successful than other WP7 developers. In this post, we see that these developers credit their success, to some material degree, to either Runtime Intelligence or Dotfuscator protection (or both).

In their own words
(selected - but unedited - responses to the open-ended question "what are you most excited about?")

I like being able to get crash reports without much additional work.

It gives the developer ability to know about usage patterns in an application. Obviously code obfuscation is a necessity, especially for paid apps.

It offers a unique way to see how users interact with the application, and with the latest release it also has error reporting. Awesome!

I'm an excitable person.

Fabulous data provided by RIS to analyze the performance, usage and app demographics.

So I can know what is happening in my app and protect my code.

Used correctly, the analytics really let me see how and by whom my application is being used. I get more insight into this information than I could if I set up a usability lab or just did extensive user testing. There is no better way to observe than to do so in production.

The concept of attaching runtime analytics after the compilation process is very useful for us (standard software development, single application in various customer-specific configurations), since we are able to attach this on a per-customer basis and don't have to manage it in code.

UI for parameterization

It gives detailed statistics about the usage of all parts of the software and helps to recognize the hot features of the software are and which parts are less used. This adds great value into the effort of making software better.

Really gives me insight into what my customers are doing with my applications. They help me to understand where I can enhance functionality and add value.

Quality of product

Analytics give me an idea on what I should work on next to improve my application

Runtime analytics is cool because there is no code to write.

I can collect the exact information i need.

It allows me to phase out or strengthen certain parts of my apps. I currently have seven apps and the instrumentation is crucial.

Because i can have a deep analysis of when and especially how my application is being used. If you add the fact that all these data are aggregated and presented in such a nice way by the portal, you end up with a great product

I produce libraries (DLL's) that are handed to third parties, hence the need for obfuscation.

Kickass obfuscator.

It helps me keep track of any bugs. And it allows feature tracking. And it gives me the cool world map that shows where some of the users are.

WP7 apps with Runtime Intelligence have higher ratings and engagement rates

In my last post, I lamented that only %2.5 of the apps in the marketplace were actively using Runtime Intelligence (RI). This begged an obvious question; were these developers leaders or laggards? Were RI-enabled apps more successful and effective and, by extension, worth emulating?

The short answer appears to be yes – to be clear, I am NOT saying that merely turning Runtime Intelligence on increases an app’s success. What I think the following data does show is that the developers who chose to include Runtime Intelligence as a component of their development process are indeed more successful than those that did not.

I took a second look at the 26,469 apps that we downloaded from the marketplace and compared those apps that were instrumented with Runtime Intelligence with those that were not.

Specifically, apps with Runtime Intelligence (analytics) are:

Ranked 25% higher: RI-enabled apps averaged roughly 4 stars (8.02 out of 10). Non-RI apps came in at 3 ½ stars (7.6 out of 10). NB: apps with no ratings whatsoever were excluded from this calculation.

Have a 50% higher rate of engagement: 73% of the apps using RI had at least one rating while only 48% of non-RI apps had any rating whatsoever.

As I've already said, simply turning on RI will not get you a 1/2 star bump in your ratings - but clearly, the kinds of developers who are achieving higher user satisfaction are the kinds of developers who are choosing to use RI.

Coming next: Users of RI and Dotfuscator for WP7 make themselves heard!

Thursday, September 22, 2011

WP7 Marketplace share (or how we became a victim of our own success)

This post tells the story of how good faith estimates of our WP7 marketplace penetration were under-reported by 500%. This is not a “gotcha blog” – there are only good actors with the best of intentions in this story; but that’s why I think it’s a story worth telling.

You see, we don’t just obfuscate – we hide the fact that your app is obfuscated. We don’t just offer application instrumentation and monitoring, we inject that logic to simplify and streamline packaging and improve performance. …and therein lies the rub.

Something was in the air

While at Build soaking up the heady atmosphere of Windows 8 and all that’s coming with it, a few MVP’s and also the Lowdermilk brothers took a moment to ask me if I’d read this awesome blog post where a developer had downloaded and analyzed all of the XAPs in WP7 marketplace. Apparently the market insight was killer and covered everything from the most popular libraries to snagging cloned apps. No more guessing – the facts were all laid out. And they each mentioned to me how surprised and even a little worried they were by how few apps were being obfuscated. I didn’t think too much of it at the time as I am among the first to point out that not every app needs to be obfuscated – but I did make a mental note to be sure to check out the blog.

When I got back home, I finally had a chance to track this “all seeing blog” down – it was Justin Angel’s blog and the post was Windows Phone 7 Marketplace Statistics. And it really is a fascinating post with both initiative and insight; and then I got to the obfuscation section.

According to Justin’s analysis, only 3% of the apps in the market were obfuscated! And as I scanned down, there were comments to the effect of “gee, since no one else is doing it, perhaps I shouldn’t bother either.”

Even more surprising to me was the fact that our analytics (Runtime Intelligence) was not even listed in a very long list of third party tools – when I knew for a fact that we have nearly a thousand apps sending data.

This can’t be right! (and I was)

Given the nearly 6,000 downloads of Runtime Intelligence and Dotfuscator for WP7 and the activity that I had been seeing over the past year, these numbers just didn’t seem right. I wrote to Justin who was quick to share his detection logic (in fact he posted the source on his blog) and just as quick to invite any comments or refinements that I might have to offer.

To put a fine point on this, Justin was in no way defensive and was as interested in getting to the right answer as I was.

Hung by our own petard

Without going into a lot of detail, Justin’s approach was to bust open the XAP and examine the various files and manifests to separate Silverlight from XNA and to identify the presence of third party tools. This approach proved to be effective because frameworks, tools and components leave behind files and other telltale fingerprints as a matter of course. There is one limitation though; this approach cannot detect when an application is modified or extended through IL manipulation or injection. And that’s exactly what we do.

From Build to Bill’ed (new numbers and how)

As I like to say, my ideas only have to be good, they don’t have to be original (trademark and patent laws not withstanding) and so I did what I often do when confronted with a conundrum – I asked Bill Leach, our CTO, for help. He quickly (dare I say magically?) authored our own “marketplace crawler” that populated our own XAP repository. Rather than look at XAP contents at a component level, he wrote some code that examined the binaries themselves.

The first pass looked for the custom attribute DotfuscatorAttribute inside the binaries. This is a good way (but not an absolute way) to determine if a file has been processed by Dotfuscator (for either obfuscation or injection of analytics). It’s not infallible because developers can remove that attribute if they chose (to further cloak the fact that they have used Dotfuscator). Here is what we found:

We downloaded 26,159 XAP files and 14.5% to have been processed by Dotfuscator.

This is basically 5X as many apps as Justin’s analysis had found (and that does not include the developers who configured Dotfuscator to remove the attribute we were searching for – so the number is certainly a bit higher).

In fact, we were surprised that Justin had found any at all – where did his 3% come from? Upon inspection, we think it’s an unexpected side-effect of how XAP’s are assembled – there are some instances where the configuration file of Dotfuscator gets pulled into the XAP – this is unnecessary and should never happen. We will document this behavior and make sure that users know how to prevent this from happening. In short, his 3% showed the prevalence of a bug – not the use of Dotfuscator.

To determine if an application was instrumented (rather than obfuscated), we applied some heuristics that are less obvious but can be shared if someone is interested (we looked for the existence of some high-level classes).

2.6% of marketplace apps are instrumented.

From my perspective, this is a low number – but to put it in perspective (or let’s be honest – I’m looking for the silver lining) we have a larger share than Google Analytics and Admob but a slightly lower number than Flurry.

Attack of the clones

Just one more point to be made in this post. If one were to consider each family of cloned apps as essentially a single, re-skinned app – these numbers have the potential to change materially. We may take a look at that, but I think we have already gotten most of what we can from the static analysis of the marketplace.

So – is that the whole story? (Of course not)

Don’t give me (just) static

As interesting as the static analysis of the WP7 marketplace is (and it is), static analysis only gives us a backwards facing snapshot of what’s already been deployed. We get no insight into:

  • Best practices that we would want to replicate (which are different that common practices),
  • Developer motivations behind their development choices
  • Future trends especially when driven by new technology and market opportunities
In the context of Dotfuscator and Runtime Intelligence, I would want to know
  • If the developers who built the 14.5% of “Dotfuscator processed” apps leaders or laggards?
  • Do they have special requirements that set them apart?
  • In short, do they have anything to teach the rest of us?
Want to know more about the 14.5% apps and what they have to teach us?

Coming soon, WP7 Developer Survey Results. We’ve been running surveys since the WP7 launch last year (you can checkout survey 1 and survey 2). As part of this ongoing effort, we have just closed out our third survey in the series and I will be posting results in the next few days – stay tuned!