It’s a typical scenario in an IT organization that separate sets of resources manage applications, servers, data and storage issues and the other groups emphasize on efficient management of these areas. In other words, the traditional methods of storage and storage resource management deal with storage devices, capacity and connections, whereas, data management mainly concerns with data movement, protection and recovery of this data. Such a difference in focus leads to a divergence in the development of management processes that, as such, focus on the satisfaction of the needs of either the storage administrators or the data administrators with little correlation between the activities of these groups.

The earlier form of management of data was more of a hardware-oriented strategy, with relatively few and basic level software tools being used that had the limitation of being able to support only one application and one operating environment. Over the times, the situation today is different. The exponential growth of enterprise data led to ever increasing demands for storage capacity. Parallel to this, storage environments have also become complex and have distributed storage architecture, pervasive networks and are supported by multiple operating environments. Such changing trends led to development of storage management approaches geared towards managing increased capacity and complexity of enterprise data. These new line of attacks have generally held on to a storage centric approach to the problem, with little or no linkage to application requirements.

In the traditional approaches, the usual case is that the application, database, storage and system administrators each focus on only a part of the data/storage management problem. This results in end to end data management depending upon manual communication between the data administrator and storage administrator. This is an inefficient and error prone method.

The following figure describes how in a traditional approach, the application and server administrators call upon the storage administrator to satisfy even the most basic requests for storage services which leads to a decline in the efficiency of both storage utilization and management efficiency.

In one common approach to dealing with such inefficiencies, interdisciplinary teams are formed to manage critical business application areas including ERP, business intelligence and messaging. The advantage offered by this method is that it encourages increased communications and sharing of common knowledgebase and local expertise. But it still requires involvement at a personal level from multiple from multiple administrative layers even for routine activities.

Integrated Data Management or IDM is a new approach that takes an application centric and data centric view of storage management and allows the controlled automation of vital storage management activities. The strategy followed is to integrate as many of the important functions of data management with application and to allow each administrative layer to become self sufficient. This strategy satisfies the needs of the storage administrator by reducing the frequency of outside interruptions without sacrificing management control over the storage environment. It satisfies the needs of server, application and storage administrators to control their data and, within limits, to manage their own storage requirements. This allows each of the administrative groups to function more autonomously and work together with more efficiency. Of course, there is cost reduction benefit as well, which results in a win-win situation for all.

There are other products like NetVault, 394 Genomatica and Pentaho Data Integration Tool which offer features such as:

• Rich transformation library with over 100 out-of-the-box mapping objects.
• Broad data source support including packaged applications, over 30 open source and proprietary database platforms, flat files and Excel documents.
• Advanced data warehousing support for Slowly Changing and Junk Dimensions.
• Proven enterprise-class performance and scalability.
• With Pentaho tools, integration with the Pentaho BI Suite for Enterprise Information Integration (EII), advanced scheduling, and process integration.

The following figure shows how the storage administrator is relieved of interruptions and operational workflows are improved when datasets and policies are combined in an IDM environment.

Companies like IBM have come up with successful tools and technologies for using this approach. IBM Optim Integrated Data Management allows organizations to respond to data-intensive business opportunities appropriately. It also allows companies to meet service level agreements for data-driven applications, comply with data privacy and data retention regulations, and grow the business while driving down total cost of ownership.

• Deliver increasing value across the lifecycle, from requirements to retirement
• Facilitate collaboration and efficiency across roles, via shared artifacts automation and consistent interfaces
• Increase ability to meet service level agreements, improving problem isolation, performance optimization, capacity planning, and workload and impact analysis
• Comply with data security, privacy, and retention policies leveraging shared policy, services, and reporting infrastructure

Integrated data management is proposed to provide a holistic approach to managing data by combining the functions and benefits of data, storage, and storage resource management. Two primary technology aspects that allow the IDM model to work are efficient storage virtualization and policy based automation. Without these it would be difficult or impossible to achieve the goals of integrated data management.

With IDM, data administrators can make best use of the benefit from storage investments by using policies to take full advantage of the most advanced storage system features. IDM also promotes interactions between data and storage administrators to a higher functional level, reducing the incidences of disturbances and making both groups more productive. Finally, IDM shortens the time needed to react to changing conditions, increasing the flexibility and responsiveness of the data management process.