Database Management System Market Dynamics (Continued)

Another set of challenges stemmed from incomplete standards and limited data models. ANSI SQL has made major advances since 1999, but the previous version of the standard, published in 1992, was incomplete and led to DBMS vendors implementing proprietary SQL extensions. Until recently, DBMSs have also entailed constraints in terms of data model expressiveness, with most DBMS products, for example, unable to handle multivalued columns or recursive queries. As a result, DBMSs were considered appropriate for text and number "crunching" but not for documents or other, more elaborate data types.

For most developers, today's world of persistent data is divided among three domains, as suggested in Figure 1. Databases are structured sets of data, designed to be used by applications. Documents are designed for human comprehension and include sequence, hierarchy, and narrative dimensions that aren't present in databases. Objects are programming abstractions that combine structure and behavior in a model optimized for developer productivity.

Historically, the three domains were addressed with three largely distinct tool sets. DBMSs served databases, content and document management systems were for documents, and object-oriented programming tools and application servers fit with objects naturally. At the peak of client/server wave during the early 1990s, some DBMS products expanded to address some object-related capabilities as well as traditional databases. Illustra was a leading example, building on the UC Berkeley Postgres project, which in turn followed the pioneering Ingres research also led by Michael Stonebraker at UC Berkeley.

Sybase, another DBMS pioneer during the 1980s, led with triggers and stored procedures, putting more application logic into the DBMS. Triggers and stored procedures were important innovations because they meant application logic as well as data benefited from fundamental DBMS capabilities, e.g., making it possible to define a procedure for determining customer credit rating once and then have the procedure consistently applied by all applications rather than having the procedure done in each application.

Several object-oriented database (OODB) products were also introduced during this period, and many people expected there would be another wave of "database wars," this time with relational being displaced by object-oriented DBMSs. OODB products failed to expand beyond niche status, however, and some have recently been creatively recycled, as we'll see in a moment.

The use of different tools for different data models produced what has been termed an "impedance mismatch," creating challenges for developers who need to work with multiple tools and models. It has been difficult to use SQL with object-oriented programming tools, for example.

Relegated to a Reduced Role in the Rush to the Web
A funny thing happened on the way to the Web: As the rush to commercial Web applications started during the mid-1990s, DBMSs, which had until that time been evolving to become the center of the client/server application platform, were relegated to a reduced role. As Web applications shifted developers' focus to HTML-based pages (documents), often drawing on data from disparate systems, the market shifted to the five-tiered model depicted in Figure 2.

The Web application-led transition produced some "conventional wisdom" that was perplexing for many DBMS-focused developers. It became a common practice to create super-user database identities and to optimize database connection pooling, for example, shifting identity, authentication, and authorization to the application (or application-server) level. This was often done for performance, as DBMS deployments that were designed to support tens or hundreds of concurrent users often couldn't scale to serve the exponentially larger user populations of successful Web applications.

Another best-practice shift was a movement away from DBMS-managed triggers and stored procedures, with application/business logic migrating to the middle tier, in application servers. This was a pragmatic option when applications had to work with data from disparate data sources, but it meant moving application logic from DBMSs, where it was consistently used by all applications, to the middle tier where it was easier to (inadvertently or deliberately) circumvent.

The Web application wave also resulted in extensive midtier data caching, a practice with serious implications for database integrity. In some respects, this shift was a step backward to earlier approaches, when separate transaction processing monitor and DBMS layers were widely used, except the application servers weren't as tightly integrated with DBMSs.

XML has also impacted DBMS usage patterns. Although XML is still relatively young (the W3C completed its XML 1.0 recommendation in February 1998), XML documents have rapidly grown into a pivotal role in interapplication data exchanges. Several XML database products have been introduced to address the need for more robust, secure, and scalable XML data management, but, as with earlier OODB products, the XML database products have been niche offerings and not poised to displace DBMSs. Indeed, many XML database products are actually just thin XML veneers atop older OODB products, as XML and object data models have a lot in common with a focus on hierarchical relationships and navigation.

All of these developments led to great stress for database designers and administrators who had previously grown accustomed to being masters of their database domains. The changes also led to new security challenges, such as the increased risk of SQL injections due to the use of dynamic SQL in Web applications.

Poised for Resurgence
The last decade has clearly been a trying time for DBMS loyalists. The worldwide DBMS market is still huge, with more than $10 billion in annual license revenues and far more invested in DBMS-related application development and maintenance, but the historical cost, complexity, and other constraints have limited DBMSs to somewhere between 20 percent and 30 percent of most organizations' stored data. The Web application wave has also created new challenges, including an awkward distribution of work between application server and DBMS tiers.

DBMSs are now poised for major resurgence and expansion, however, for several reasons. One important advance is the fact that ANSI SQL was updated in 1999 and again in 2003, with significant extensions for XML, object/relational, and other requirements. Although it will take the leading DBMS vendors several years to catch up with the latest standards, and although the standards will continue to evolve, the latest ANSI SQL advances have made DBMSs more powerful and more readily interoperable.