An old object-database company learns new tricks

I've been following the emerging Event-Stream Processing field (also known as Streaming Database or Complex Event Processing) for a few years now. The relational database movement tamed disk-bound transactional data with a set of engineering techniques built on sound theoretical principles, and streaming databases offer to do the same for data in flight.

"Data in flight" is a broad term, encompassing signals generated by sensors, deltas generated by transaction processing, patterns recognized by algorithmic trading rules and intrusion-detection systems, and records flowing within a global enterprise's service-oriented architecture. The streaming database approach - though not currently any one product - can in principle address all of these application areas.

The relational database world fractured, for a time at least, into sub-disciplines: embedded databases, object-oriented databases, data warehousing engines, rule-based databases (I'm thinking of datalog here) and transaction-processing engines, before the likes of Oracle, SQL Server and DB2 could truly call themselves general-purpose databases. So, it's interesting to see the same ecological niches being filled in the streaming database landscape.

Among the companies out there, Coral8 and Streambase are SQL-centric and inclined to applications involving analytics, Progress Apama ESP has a more rule-based approach, the open-source Esper engine is embedded and written in pure Java, and there are several more companies each with their own vision of what a streaming database (or event-stream processing engine, if you prefer) should look like.

The article "Building Real-Time Applications with Continuous Query Technology" in August's Java Developers Journal is the first time in a few years that Gemstone have drifted onto my radar. Back in the day, Gemstone pioneered object-databases, which aimed to make disk-based data more accessible to people who wrote object-oriented 3GL code, and in particular complex user-interfaces, for a living.

So it's no surprise that Gemstone bring the same mindset to what they call "continuous query". Again, they're looking at the problem from the application developer's perspective, and the article describes a compelling programming model for Java-based developers, although I can't speak for the robustness of the engine underneath it all.

The event-stream processing (ESP) field is still young, yet already the diversity of products in the arena is inspiring. ESP is finding a wide variety of applications, and vendors are adopting a correspondingly wide range of architectures. Don't hold your breath waiting for a standard to emerge, but the ESP approach is looking more and more sound.