Rapid Development Speeds Recovery
Learn how Verizon managed
September 11 NYC phone service
recovery using a VS.NET
customer service tool finished
in one weekend.
by Edmund X. DeJesus

Not long after the September 11 attacks in New York City, recovery and renewal had already begun. Homeless families were taken in by friends and relatives. Schoolchildren began attending classes, safely removed from the destruction. Businesses relocated. People began to resume their lives, go back to their work, and follow their "normal" daily routine.

Technology is not so resilient. Buildings don't reconstruct themselves. Roads and subways don't repair themselves. Utilities don't restore themselves. It's often during crises such as this that we truly appreciate the services we usually take for granted: running water, electricity, a dial tone…

Indeed, the recovery of communications is one of the success stories of this disaster. Yes, there were outages and circuits in the area were jammed for days. Yet remarkably, considering the devastation, the communications so essential to our connections with other human beings went on. The phone service was restored, and recovery from the destruction proceeded day by day.

This was no small feat. In New York City, the infrastructure that carries calls, the switching circuits that route calls, the lines to individual phones, and the support databases that match logical phone numbers to physical phone lines were all destroyed, damaged, or severed from the rest of the system. Those responsible for restoring this vital communication had to decide how to reassemble this mind-boggling collection of hardware and information quickly and efficiently. They needed to evaluate where to best allocate limited resources to get the most lines working the soonest, and decide what replacement equipment would be necessary where.

By an astonishing coincidence, Verizon Communications, a major telecommunications provider in New York City, had a tool at hand that could take on this monumental task. As with many who were yanked from their usual routine to assist in the crisis, this tool was drafted from its original purpose to handle far more than was initially intended. Code-named Mustang, this software application was originally created by Ajilon (Towson, Md.) to assist Verizon managers in planning and analyzing the status of work orders. However, it soon received a far more important and urgent assignment.

Rising From Humble Beginnings
"Mustang was one of a variety of projects we worked on for Verizon over several years," says John Keefauver, senior account manager for Ajilon. Ajilon designed Mustang to handle several related tasks. Managers could use it as a day-to-day project management tool, to keep an eye on various orders and projects. The application could also provide a historical overview of completed projects and orders. This would offer analysts insight into bottlenecks and obstacles in past projects, and enable them to find better strategies for similar tasks in the future. Finally, Mustang could help analyze decisions in pending projects to predict the outcome of different approaches. For example, if Verizon replaced unit A instead of unit B, Mustang could anticipate if that decision would result in faster service to customers and how it would impact current service.

Mustang arose from a desire to provide better customer service, and to allocate resources expeditiously and to minimize inconvenience to customers—valuable tasks for any application to perform for any organization.

Figure 1 Review the Structure

However, September 11 changed all that. Suddenly, Verizon faced the enormous task of restoring telephone service to thousands of people and businesses in New York City as quickly as possible. Equipment and data had been destroyed, damaged, or lost. Verizon managers had to decide how to best allocate existing equipment, how to help the most customers fastest, and how to impact the fewest customers.

Verizon realized a software application could help plan and oversee the restoration, and Mustang sounded a lot like what they needed. Ajilon added many new features for enhanced user access and logistics planning, and an application code-named Viper was born (see Figure 1). Verizon needed it fast.

Fortunately, Ajilon was able to deliver. "We had actually been in the process of migrating Mustang to .NET technology in the weeks preceding September 11," notes Brad McCabe, senior application development specialist for Ajilon. As with previous projects for Verizon, this one involved Microsoft products, such as Visual Basic 6 (VB6) and SQL Server. "The original Mustang was written in VB6, using VB6 Web classes, and some Active Server Pages (ASP)," says McCabe. On the back end, Microsoft SQL Server 6.5 served as the database, running on Windows NT 4.0 servers. Luckily, the transition from SQL Server 6.5 to SQL Server 2000, and from Windows NT 4.0 to Windows 2000, had been completed. Ajilon developers were working with beta 2 of Visual Studio .NET (VS.NET), and they'd completed about 80 percent of the .NET version by September 11.

When Verizon approached Ajilon with an urgent need for new features, Ajilon realized that, although the VB version of Mustang was complete, it would be faster to finish the rest of the .NET version and use it as the basis for the new features. "We received the new requirements on that Friday," recalls McCabe. "Seven developers finished the base .NET version, and added all the new features over that weekend."

Viper's .NET technology handled the new features easily. For example, because Mustang had been intended as an internal application, extensive security hadn't been a major concern. Although Mustang had been secure from casual browsing and editing, it hadn't included special features such as Secure Sockets Layer (SSL) or encryption. However, Ajilon enhanced security through SSL so authorized Verizon users could access Viper from anywhere through Verizon's intranet. The new user interface, using ASP.NET, is feature rich and required less development than the original. Although the original Mustang application had been intended for just a few users, Viper can handle thousands of users. "We used SQP Ajilon (the former Software Quality Partners, acquired by Ajilon) to load-test and stress-test Viper," says Keefauver. "With over a thousand concurrent users, we were only hitting a few percent utilization level."

Viper also had to be able to scale rapidly to handle a large volume of data. The .NET technology satisfied that requirement handily. "The database grew by 6 gigabytes within a week," ob-serves McCabe. The technology copes with such unpredictable, ad hoc increases automatically. "Verizon was pleased with the per-formance, which required no special optimization," states McCabe. Surprisingly, despite the new features, the .NET code base was about 15 percent smaller than the original VB application.

Tools for Rebuilding
McCabe says he appreciates the object-oriented features of VB.NET, including inheritance, "We can write objects, and in-herit from them, which saves development time and effort." As an example, McCabe points to the menu system in Viper. All the ASP.NET pages have a menu across the top, developed using a beta version of UltraWebNavigator from Infragistics. Ajilon developers created a base class, and all other pages inherit from that class. The menus change automatically, depending on the specific page attributes and the user. This was a tremendous simplification for developers, allowing code reuse over hundreds of pages. In addition, they can add changes to the base class as Verizon requires, and those changes propagate automatically through all inheriting pages.

Inheritance also simplifies the code for database access. After writing a basic infrastructure class, developers all inherit from that class. Data manipulation functions such as ordering, sorting, and paging are built in. "You need about five lines of code per function, about twenty lines total," says McCabe. In contrast, the VB6 version required writing all that code manually.

Figure2 Get Data You Need

Ajilon also appreciated VB.NET's multithreaded capabilities, as compared with single-threaded Visual Basic, for access to mainframe systems. Viper isn't allowed to access the Verizon mainframe data directly, so it has to act like many 3270 screens and scrape information from a dozen different mainframes (see Figure 2). "With single-threaded Visual Basic, this could take a day for each information scrape. Now it's a fraction of that time," remarks McCabe. Ajilon reported satisfaction with .NET tools, even though they were in beta. "This beta was better then gold code was a few years ago," says McCabe. The tools were very stable—and had to be. "We're consultants," says Keefauver, "If the result is unstable, we expose ourselves to risk and bad reviews with our customer. We have to be confident in a tool to use it."

Recovery Continues
The rapidity with which Ajilon was able to accommodate new features has impressed Verizon, and it has expressed confidence in Viper as a basis for new features. Indeed, Ajilon receives req-uests for new features on an almost daily basis. "We deliver a new version in the morning, receive new requests in the afternoon, and can implement them before we leave for the day," comments McCabe. These developers aren't working full time on Viper either; they have other projects as well.

Ajilon has found that VB.NET is as fast to run as C++, while far easier to write, debug, and maintain. Another plus is that there are many VB developers around (as opposed to Java and C++ developers) who will have an easier time making the transition to VB.NET. Ajilon has migrated its other applications to .NET, and recently upgraded to the release candidate 1 of VS.NET. The Ajilon team is confident in .NET technology as a basis for applications they will be living with for a long time. That makes a good foundation for building a better future.

About The Author
Edmund X. DeJesus is a freelance technical writer in Norwood, Mass. Reach him at .