Bake in Business Intelligence
The architect of Crystal Reports speaks out on the impact of business intelligence on developers, security, and more
by Lee Thé

Tony Wind

Q. What do IT staffers need to understand about business intelligence?

Wind: It is as essential as e-mail. I know that like e-mail, it has a difficult return on investment (ROI) at times—especially with connecting people to corporate information. But as soon as it's there, if you try to take it away, people will refuse that. And because it's so essential, it requires an enterprise solution. That's what we've been focused on for five or six generations. Crystal Reports seeds that, but now in a mission-critical framework. If your BI system goes down, it really does impact the business. People will understand that more and more as they deploy these systems. That's also why it's important to go with a substantial BI vendor who understands how to provide high availability.

Q. What does it take to provide BI on the database side?

Wind: Ask yourself how many types of data do you connect to—Web logs, e-mail or other logs, different kinds of relational or OLAP/analytical databases, data from enterprise applications by SAP, Oracle, and the like? More importantly, can you bring these all together in reports? Your BI vendor should be able to provide drivers for everything.

I was the first programmer of database drivers for Crystal Reports. After a while, we started to partner, and I remember the vendors coming to me looking for database drivers. I gave them the DLL and let them build their own drivers to a standard interface. That way it didn't touch the core development team. Of course, if anyone has an obscure data source, you can map to ODBC or your BI vendor's own interfaces.

You should also be asking how strong are the engines the BI vendor provides. For example, we offload a lot of processing from the database server.

Q. How about BI requirements on the end-user side?

Wind: Ask how your users interact with the data. Is it easy to go to an Excel product or create an Oracle query and publish it, click on a chart or map, jump from object to object and from report to report with no programming? See how deeply your partner's technology lets you interact with the data.

Then look at output: How well does the BI solution support formatting output from HTML to PDF, MS formats. And can you integrate that output into Web apps seamlessly, to make the BI object kind of disappear into your Web apps? In general, how does the BI solution integrate with the Web environment?

Q. How challenging is scaling for BI?

Wind: Moving away from core database and presentation, you have to recognize you're building an enterprise system with enterprise requirements for scalability and reliability. You need a redundant middle-tier system for scheduling and servicing requests, so that one machine can fail without incurring BI solution downtime, and so you can add machines for scaling without having to take down the system.

A system that provides 80 percent of the solution needed to support 100 users may seem solid. But when you have to scale that system to 1,000 users, you may discover that the solution you bought only provided 10 to 20 percent of the solution you now need. Scaling from a workgroup system to an enterprise one is a lot more complex than many IT people realize. The workgroup system may have some components you need but lack the backbone. And see how much of the middle tier is built for you compared to how much you have to build yourself.

And you must design for peak loads—spikes at the end of a quarter or beginning of a sales week. Ideally, you want your solution to provide "page on demand only" that loads just what you need in your browser; fills in the page quickly; then adds charts and page numbering as needed, followed by drilldown as needed. The system shouldn't be shipping multimegabyte objects to users.

Q. Security?

Wind: It's important in this space. This is your corporate info. You let people see it on a need-to-know basis. It's good to grant as wide an access as possible, however, especially when you leave the corporation. Our technology lets you deploy data out to partners and customers on a more limited basis, with security layers designed in from beginning. We embrace and support LDAP, Microsoft security systems, encryption.

You need a security model that plugs into the enterprise framework, letting you configure BI objects for types of users—do they have the right to configure a complex query?

Q. Assuming that most IT shops buy rather than build whenever possible, where does building come into your vision of BI?

Wind: A lot of features are needed to provide scalability, manageability, and reliability. And they're difficult to implement. Better to buy a framework and customize it for your needs. I never want to adopt a platform where I feel my hands are tied. You need customizability. What to ask for starts with creating content. How programmable is ad hoc query? With .NET, you can program any front-end Web app you want. Which means you need API-level access to developer interfaces equivalent to anything in those Web apps' UIs. And you'll want to manage through a Web console or management console you can program through an SDK. For example, an SDK familiar to people who know ASP will give you a jumpstart on creating your Web front end. You want APIs to give you programmatic control of everything from building the OLAP cubes to the front end to the management of the system. It takes hundreds.

Q. What's the difference between BI and simply running a data cube with OLAP?

Wind: Scalability. And of course, reporting and analytics. BI is lot broader. When they profiled the BI industry 10 years ago, it was ad hoc, data mining, OLAP. You can make finer slices, but it boils down to one problem: how to let users monitor the info they do know about and explore what they don't know about.

If we take the technology viewpoint, we can have that discussion, but you have to raise your sights. Sometimes users just want to look at sales reports; other times, they're analyzing. If doing both tasks requires jumping from one tool to another, it wastes time and doesn't scale. So it's important to think about what you're delivering to users.

Q. Do the heterogeneous data sources and networked, distributed apps promised by both .NET and J2EE provide particular problems for BI?

Wind: .NET and J2EE have a lot in common. We can compare and contrast, but I think this decision is very important, and will be made for a lot of reasons, of which BI is one. We as a BI vendor feel it's important to play in both environments. We've done deep work on .NET and Java as well. Both are a change from older programming models.

This new programming model makes it easier to connect user machines with the back end in a corporation. You can build zero client apps interacting with middle and back-end tiers. That lets us deliver BI a lot more easily to users. You don't have to install a lot of software or download big pieces of code or data. The evolution in platforms will assist in delivering more data-driven apps. You'll look for a BI vendor rather than download to SQL queries yourself.

At the level that matters for our technology, they're equivalent platforms. So checklists of features shouldn't drive your standardizing decision. It should be driven by the platforms your company chooses to use, the skill level of your developers, and their particular skillsets.

BI can function equally well on either platform. We can be an integrating point. SAP chose us last year as their standard for enterprise reporting. It used to require much more custom coding, but now it's more Web based. Suddenly, no matter which data source you use, the front end is consistent—even against multiple ERP data sources. The BI system hides the complexity. This also makes it easier to handle mergers when the two companies use different platforms.

Q. Do development teams often bolt on BI after the fact rather than designing applications so they can get data out?

Wind: Ten years ago, it was more of an afterthought. I really saw a change during Y2K, because now the platforms are more standardized and IT is increasingly important. And IT itself is seeing it more strategically. So I think it's evolving. Some companies are farther ahead than others, but over the last five years, a majority of IT has come to recognize that BI is strategic.

But you can tack BI on later on. SAP has 20 years of reporting history, yet chose us as a new partner for presenting new reports. Certainly it's possible to take existing apps and data sources and use BI to get the data out. You need to create an API to access the data. SAP created one, for example. Other companies will have other ways of doing this. We've never insisted on one format.

But just as designing an ERP is a big decision, BI is too. Enterprises should study their needs and adopt a BI framework. Companies reap a lot of rewards when they do. Collaboration starts to happen. You start to publish different types of data sources into the same Web app. You can also start to allow users to explore their data more. You do canned reports to start with, but if you use advanced analysis tools, you just have to design a view of the data and give users ways to explore that data further.

Q. Are there design considerations that are overlooked, such as database schemas and security, that should be done at the design stage to help support Business Intelligence later?

Wind: It's valuable if you provide a database interface—such as a data warehouse—to all your systems. We can be the integration point, but to allow users to jump quickly to drill into data from another app requires designing the data warehouse—especially if you're integrating many apps. It's not essential, but it adds more and more value.

Q. How do data warehouses figure in BI in 2002?

Wind: Our products have been designed to go directly to the data, from scheduled running of reports to on-demand delivery, hooking up directly to the data. The good news about BI is that we're always read-only. BI can't corrupt the data. I've worked on these technologies for a long time and I'm glad that we're read-only.

Customers want a consistent view of their data. And IT wants to present one. But if people are querying minute by minute, it won't be consistent. We'll work either way. I'd say it's an important decision and not an easy one to make. You should allow people to go directly as much as possible except when user load is too heavy or where they need data consolidation to go across areas. It could be an API that runs dynamically across data sources. Then the customers will get out of your hair and they won't be bugging you about the data they need. In the end, there's plenty of work for IT however you slice it.

Q. Here's my blue-sky question. Biologists say we evolved from tree-dwelling primate ancestors. That means they had to do accurate, realtime, 3-D visual processing just to survive. Whatever you think about the origin, it's inarguable that humans have impressive 3-D realtime visual processing capabilities, from our stereo vision to massive networks of neurons in the brain backing that up. Yet BI tools still typically provide multidimensional information through a 2-D spreadsheet interface. Just as data presentation has moved from stacks of greenbar printout to nice graphs and charts, don't we need to harness the human mind's visual processing capabilities better, including presenting time series information with animation, much as TV weather reports now include moving weather formation?

Wind: I agree to a large extent. Though just getting thrown into a BI tool that lets you surf six dimensions at once is mind-boggling. You need systems to be looking for outlier data—exception conditions, both good and bad. You have to understand your business to create these levels of presentation.

It's going to be solved by the application logic people are adding to Web apps. If you get thrown into an app like Amazon.com where you're choosing which country to shop in and which kind of product, you're actually going through dimensions. If the app is kind of friendly and graphical, they're doing things like OLAP and exploration of data, yet it's not nearly as painful for them. The user moves from branch to branch, tree to tree.

Q. That kind of takes us full circle back to those primate ancestors. In the future, do you think game technology might be adapted to BI applications? In other words, when is the business sector going to catch up with the game sector?

Wind: It's possible. You could imagine something involving a VR helmet and gloves, going into data. But you can do a lot with the tools and systems available today. As I've said, people can interact with data even without a lot of fancy animation. Crystal Analysis, released mid-2001 and winner of the "Tech-Ed 2001 Best In Show Award," is an example of Crystal providing 3-D visualization of data for end users. A key benefit is the Web focus of this product, allowing powerful analysis to take place with just a Web browser.

Q. I should give you a chance to mention the rest of Crystal Decisions' product line—especially the new products Wind: We produce Crystal Reports, Crystal Analysis, and Crystal Enterprise. Reports everyone knows. Analysis is more OLAP-based. Analysis Pro is a Web-based product for SQL Server, and there's Holos, a high-end platform. Enterprise is for customers larger than, say, a small indie consulting shop. It combines the Reports and Analysis products, providing a single product you buy to develop a BI solution for your company. The Enterprise family adds value by giving you a scalable framework.

Q. Let's turn from blue sky to the everyday world of reports. Crystal Reports certainly provides an easy-to-use interface. But when developers don't understand the underlying data structures, the requests for reports can become a drain on programming resources. And the end users don't understand why, with such a WYSIWYG tool, it takes developers a long time to get a report written.

Wind: Obviously, people developing reports need to have a basic understanding of their data. Crystal Reports is powerful Web reporting software, but it can't solve the problem of developers not understanding the basics of the data they're collecting. On the other hand, Crystal Reports does not require developers to learn a proprietary language. If developers understand their data, then they're ready to transform their data into valuable information end users can understand.

Q: Do you expect to add to the number of samples/walkthroughs on your Web site? The original five walkthroughs have been upgraded to RTM. It might help to somehow integrate the help for previous versions that's applicable to the .NET version as well.

Wind: Absolutely. There will be more samples and walkthroughs available. Our new .NET zone is a site dedicated to helping developers use the Crystal Reports for Visual Studio .NET. As the content calendar moves forward, this resource site will provide white papers, walkthroughs, demos, and hot topics. It should also be noted that Crystal Reports for Visual Studio .NET has full MSDN documentation, F1 support, and dynamic support.

Q: With the amount of attention given to .NET, many developers overlooked the Crystal Reports .NET version's functionality initially. Are you moving to get the word out better?

Wind: Yes. For one example, on MSDN readers can link to http://msdn.microsoft.com/library/?url=/library/en-us/crystlmn/html/crconcrystalreports.asp.

Q: What are your plans for enhancing the "viewer" controls for Windows Forms/Web Forms?

Wind: Crystal Decisions built completely new Win Forms and Web Forms viewers for Visual Studio .NET, utilizing C#. You can access these new viewers on the Visual Studio .NET toolbar. The question we've received about the Web Forms viewer is about printing on the Web. Naturally, the Web presents printing dilemmas when you render reports in the HTML format end users demand, but there are ways to overcome these dilemmas. A technical white paper on our customer support site explains how developers can provide printing functionality.

Q: What are your plans for enhancing the .NET Engine support? Drilling from Chart to Report to SubReport is great, but what about more interactivity between, say, drilling from charts, to .NET datagrids, to the detail row that I then want to update, and so on?

Wind: These kinds of custom drill-downs can be accomplished through hyperlinks.

Q: These days, everyone wants to understand security, especially with Web services. How can developers secure the display and delivery of information?

Wind: Crystal Enterprise (available since 1996) is our answer to the secure and scalable delivery of information to users around an organization over the Web. Integrating Crystal Reports that are managed using Crystal Enterprise is a drag-and-drop scenario, because Visual Studio .NET has a built-in Crystal Enterprise SOAP connector permitting rapid integration of secure Crystal Reports. In the process, security measures carry over into the .NET application, allowing only users at the assigned security level to view or modify the report. Or developers can provide security to the applications they're developing that include Crystal Reports. This may make a lot of sense for developers, as those accessing Web applications will likely require some form of authentication/sign-in. It's not difficult to extend this authentication to control which Crystal Reports they can view.

Q: Are you planning to add support for OLAP grid objects to a future version of Crystal Reports?

Wind: Future releases of Crystal Reports will provide many new features and actually update the edition of Crystal Reports built into Visual Studio .NET.

Q: Could you discuss your licensing model? Some developers have gotten the impression that it has become vastly more expensive to use for enterprises or distribution.

Wind: This is not true. If anything, the licensing model has become more clear through dozens of focus groups and research studies conducted by Crystal Decisions. What has always been free is exactly the same in the Crystal Reports for Visual Studio .NET edition. Windows applications are free runtime. Web applications offer a free five active user license. This is different than the license provided in our past and current shipping product, in that the way you measure usage is no longer based on a 20-minute session time, but instead is now based on the processing actually happening on the server. The feedback from developers has been that this is a clearer and more generous approach. You can read full licensing information at www.crystaldecisions.com/net/licensing.

Q: How would you defend the cost of your product? What features/functionality do you feel is only offered by Crystal compared to your competition?

Wind: We should first clarify what the pricing is. The fully featured product comes free with Visual Studio .NET. The only potential cost to developers is in deployment. If you expect to support a user community larger than 50, you can purchase license packs. The first entry point is the $3,000 workgroup expansion pack; subsequent license packs are $7,500. A processor license is $25,000. Note that you're not paying for the report writer, which comes free with Visual Studio .NET. What you're paying for is the ability to perform real-time, zero-client Web distribution of formatted reports to communities larger than 50.

Q: The .NET-managed provider for Crystal Reports that shipped with .NET doesn't natively support XML. Do you plan on supporting native XML, or will developers have to work around this by loading the XML into a DataSet?

Wind: I believe that ADO.NET's support for XML is good and the need to natively read XML is not going to be very large. However, this is on our radar for future releases.