email article printer friendly more resources

Why Our Computers Act Irrationally
Complex event processing may provide a framework for understanding our computer systems
by Peter Varhol

Posted June 7, 2004

If I may over-generalize for a minute, there are two kinds of computer professionals in an organization: those who keep the computer systems, servers, and networks running, and those who use the data those systems hold and generate to produce meaningful information for those who make decisions. Both do important tasks in the scheme of building and running the applications that make up our livelihood and our employers' business.

And there's rarely any overlap between the two. Most developers can maintain and diagnose systems and networks but prefer that they just work right to begin with, while many system/network administrators can write long and mind-numbing scripts but prefer digging into the insides of servers and routers and understanding their secrets.

In this dichotomy lies a fundamental problem. There's little apparent connection between problems found in the application and clues to that problem in the systems and networks. Say a developer builds a Web application for the enterprise that is intended for use by many small, retail customers. For the first month the application works fine, but after that many customers complain that the application is slow or completely unavailable.

The developer will invariably seek the answer in the application itself, looking for some sort of failure or bottleneck that would account for this change in behavior. Failing that, he or she will go to the server and network administrator, who may be able to note that network traffic to the application is heavy and that the application itself is running at close to its server capacity.

But while the administrator can make some guesses based on experience as to why this is occurring, there's no ability to identify causation between the server and network events and the user experience with the application. Most of the solutions at this point require trial and error, rather than a pointer to the root cause at a human-understandable level. Usually the problem gets identified and eventually solved, though surely not in the most efficient manner possible. Sometimes it never gets solved, as the parties involved settle for a work around instead.

Observing Events
This scenario is the purported realm of an emerging field called complex event processing (CEP). CEP involves obtaining a human understanding of the multitude of low-level events produced by computers and information systems. The goal is to understand why an end result occurred based on the interaction of large numbers of these low-level events. Probably the most comprehensive introduction to CEP is the book , by David Luckham (Addison Wesley, 2002).