The Data Maze – January 2010
By Dan Sutherland, IT Architect, IBM

Over the last several years, the rapid explosion of data along with business clamoring for that data and the need to turn it into information faster and faster has driven our calm, staid data world into a new era. This new era demands the ability to capture data, rationalize it, and disseminate it in a rapid fashion along with the ability to use that information as the foundation for decision-making based on analytics.

Today, the majority of the modern corporation’s data is currently stored in a relational database such as Oracle, DB2, and SQL Server. These database management system (DBMS) vendors designed and implemented databases for transactional or online transactional processing (OLTP) systems. As data warehousing or decision support systems (DSS) grew, it made sense that since the enterprise licenses were already available and paid for, and the labor skill set already in place, that these same database platforms be used to store data for the DSS data stores; such as the operational data stores, atomic data warehouses, and departmental data marts that were and still are proliferating across the enterprise. Traditionally, these legacy database platforms were designed to update a small set of rows for small and limited boundary queries, and a “one size fits all” (OLTP and DSS) mentality. Because of the rapid change caused by competitive analytics, these legacy database platforms are no longer enough for the enterprise, and a need for database platform specialization has come to pass.

This need for database platform specialization between OLTP applications and DSS applications has been proven in the marketplace because of three main reasons:

1. Businesses are being inundated with data. Data is growing at an unprecedented pace, not only because of additional transactional data and sources, but because of the rapid growth of unstructured data and new streams of data from sensors and other types of instruments that feed data on an ongoing basis. Much of this data is a rich source of information, but it is currently just being stored and dropped because there are no resources to analyze it and convert it into competitive analytics.

2. Businesses need information faster. The driving need for real-time analytics across the enterprise is driving architecture and design changes. Not only is the data deluge getting bigger, but businesses need it faster in a consumable format.

3. Current software and hardware limitations hamper the previous two drivers. In today’s new world, the ability to scale-up, scale-out, and “parallelize” workload, is an overriding requirement.

Database platform specialization has started to address many of these concerns, through many methods, and the proliferation of companies and products not seen since the advent of the Web explosion.

As data professionals, we cannot afford to learn one or two new skills, but must constantly learn and keep up with the fast-paced changes happening in our sphere of influence. We must be prepared when our management asks about ways to modify and/or enhance the current architecture and infrastructure, to enable analytics across the enterprise, to meet the pent up demand for these capabilities. To that point, as we both learn together, hopefully we will be motivated to take the necessary steps to broaden our knowledge so that we are prepared to support these future business requirements with an efficient approach.

 

---Source: TDAN.com January 2010 newsletter (www.tdan.com). Dan Sutherland is an IT Architect at IBM specializing in business intelligence solutions and integrated data architectures.