November 20, 2009

Microsoft StreamInsight vs Rx Framework

Few days ago Microsoft StreamInsight was presented on Urals .NET User Group. Nikita Samgunov, the author of presentation, have made a really good overview of its features and architecture, so it was really interesting.

But I left it confused: the same day Rx Framework became available, so I had a chance to look on it much closer - and I was really impressed. It was clear StreamInsight solves almost identical problem, but there are significant differences:
  • StreamInsight is positioned as CEP framework. Rx is positioned as general purpose event processing framework. This must mean StreamInsight should solve some specific problems much better than Rx. But as it initially looks like, this is arguable.
  • StreamInsight uses LINQ-to-CepStream. CepStream is IQueryable (we checked this - e.g. it fails on translation of ToString() \ GetHashCode()), so there is a LINQ translator for it. But Rx uses custom LINQ extension methods to IObservable monad. So StreamInsight builds a program transforming CepStreams while compiling the expression stored in IQueryable, but Rx does the same "on the fly" - event crunching machine gets built while LINQ combinators are applied to IObservables one after another.
  • It seems StreamInsight adds implicit conditions in some cases, such as "event intervals must overlap" for joins. I'm not fully sure if this is really correct, since I didn't study StreamInsight so closely, but at least it was looking so. Of course, the same is possible with Rx, but you must do this explicitly.
But there are many similarities as well:
  • Both frameworks run all the calculations in memory. There is no persistent state.
  • Both frameworks can be hosted inside any application.
  • Both frameworks are ready for concurrent event processing. It seems StreamInsight executes everything in parallel by default; the same is possible in Rx, but what's more important, concurrency is fully controllable there (btw, that's really impressive advantage of Rx: events are asynchronous by their nature, so concurrency looks much more natural there, than e.g. in PLINQ).
After summarizing all this stuff for my own, I made the following conclusion: Rx is what definitely worth to be studied (it seems I wrote that earlier ;) ), but StreamInsight... Is, possibly, a dead evolutional chain. 

First of all, Rx seems much more powerful and generic. I really can't imagine why I should compile the queries. If there is Rx, an approach provided by StreamInsight looks like writing enumerable.ToQueryable().[Your query].ToEnumerable() instead of just enumerable.[Your query].

Moreover, I clearly understand how complex this problem is - to write a custom LINQ translator. Most of developers are able to write their own extension methods to IEnumerable; many are capable to rewrite the whole LINQ to enumerable. But there are just a handful of teams that wrote their own LINQ translator. So presence of this layer in StreamInsight looks as unnecessary complexity. If so, it will evolve much slower, will be more complex to extend (e.g. with Rx you can use custom methods without any additional code; but the same won't work with StreamInsight), etc.

On the other hand, there is a statement from leaders of both teams stating that both frameworks are useful.

I think Erik Meijer should simply say: "occasionally we kicked StreamInsight team's ass" - but certainly, this seems not what really possible ;) Phrases like "It (Rx) is particularly useful to reify discrete GUI events and asynchronous computations as first class values." looks especially funny, if you seen any videos with Erik Meijer on Rx (this one is a very good intro): Rx application area is much wider than just GUI event processing. In fact, Rx offers a new language (DSL inside C#, F# and so on) allowing you to describe asynchronous computations much more naturally (or distributed ones, such as Paxos). Taking into account we can increase just CPU count (or machine count) now, but not their frequency, Rx appearance seems very important. If you looked up the video, you should remember Erik Meijer said, that, likely, he can retire right now -  he's almost fully sure this is the nicest abstraction he invented.

So... Study Rx, not StreamInsight ;)

P.S. Amazing, how few persons (or may be, even just a single person) may change the way we think. And how fast a new technology they built can kill the technology it was originated from.