For the past decade, I have been witness to and part of an evolution in the IT industry that has seen client-server architectures replaced with service-oriented architectures (SOA), where expensive and dedicated big-iron hardware has been replaced with shared infrastructure based on virtualization and commodity hardware. (Cloud) I have seen organizations succeed with SOA and cloud, and I have seen some epic failures.
The organizations that failed were too focused on the IT aspects of what SOA and cloud were all about. They spent huge amounts of time and money building out an infrastructure that failed to show specific benefits to the business owners. They built services for the sake of building services with little or no thought as to the utility to the business. They failed to understand that it really is all about the data and you can’t back your way into a data strategy. The services and infrastructure only provided access to about 20% of the data available across the enterprise because they focused on what they had done in the past, and not what needed to be done for the future.
As organizations look to capitalize on this new data economy they need to leverage all of the information available to them. The shear volume of structured, unstructured, and semi-structured data that is being created at unprecedented rates from a variety of systems, devices, social networks, logs, tweets and full motion video should be an integral part of an enterprise IT data strategy. The volume, variety, velocity, and complexity of this information is the definition of the ‘big data’ challenge facing organizations today.
The focus on the ‘big data’ challenge in the enterprise is what brings real relevance and focus to SOA and cloud initiatives by driving alignment of IT to the business. With that focus, these initiatives establish the foundation to quickly develop and deploy new business capabilities, facilitate change to current business process, and provide a new level of predictive analytics to help chart a future course.
Big data needs SOA and cloud in-order to be effectively leveraged across the enterprise. SOA and cloud need big data in-order to establish deeper value and relevance to the business.