We often get asked by our clients to differentiate the Kinaesis DataOps methodology to a standard Data Management Methodology. Many organisations are implementing standard methodologies and are not seeing the benefits. The key differentiator to me between the methodologies is that DataOps is based around practical actions that over time add up to deliver results greater than the sum of the parts. If delivered correctly across the 6 pillars of DataOps they help you to transform the organisation to be data driven. The key to this is the integration of people and process that deliver real business outcomes avoiding paper exercises.
On my consulting travels I find that many industry data management methodologies layout the theory around implementing a Data Dictionary for example and this is taken as a mandate to deliver a dictionary as a business outcome. Within DataOps a dictionary is not a business outcome, it is part of the deliverables that are part of the methodology and an accelerator of delivering a business outcome. This is a subtle difference, but one which leads to the effort of the Data Dictionary being part of the business process and not an additional tax on the strained budgets. Within the methodology it is produced as an asset within the project and for the future to make subsequent projects easier.
Another difference is that in standard data management approaches the methods are quite prescribed and consistent across all different use cases. The nature of the DataOps methodology is that it fits the approach to the problem being solved. For example some data management problems are highly model driven like credit scoring, customer propensity, capital calculations, etc. Other problems can be more reporting and analytics. Each of these require a different focus and a different emphasis and sometimes a different operating model. Through the iterative approach then there is freedom within the methodology to achieve this.
Many data management approaches prescribe an approach that tries to encapsulate all of the data within an organisation. This is a noble cause, but it is a large impediment to making progress. Firstly in many cases we have found that only 15-20% of the legacy data is ever required to meet existing business cases. Secondly we find that the shape of data is highly dependent on the use case being implemented and because you do not know all the use cases and future use cases it is not pragmatic to do this. By being able to measure this usage through instrumentation and driving them through use cases then the data management problems can be simplified to achievable outcomes in short periods that can form the foundations of the data management strategy for a business area to be leveraged.
There are many other differentiators within the DataOps methodology however all of them start with the principles that anything you do needs to be implemented and pervasive. The methodology builds its strength from tying business benefit to the process and builds from this. The goal is to deliver value early and often and to leverage the benefits to build momentum to deliver more benefits over time. Once integrated into the operating model then the approach builds and transforms the culture from the ground up. This delivers great benefits to the organisation where many data management methodologies start with great promise and then struggle to gain support when the size and the complexity of the task start to become apparent.
If you would like to find out more about the Differences of DataOps then please do not hesitate to contact me.
In recent meetings with clients we have come across a lot of instances of the need to improve the art of capturing requirements and building the 'Solution Contract'. Typically, these projects are large data analytics and reporting projects where the data is spread across the organisation and needs to be pulled together and analysed in particular ways. A typical data science, data engineering task in today's data centric world.
The problem that people are describing is that they are quite often asked to solve problems that the data does not support, or the requirements process does not extract the true definition of what a solution should be.
We are asked “how can you improve the process of requirements negotiation using the DataOps methodology?”
Kinaesis breaks down its DataOps methodology into the 6 IMPACT pillars. These are;
• Meta Data
• (Extensible) Platforms
• (Collaborative) Analytics
The requirements process is predominantly within the Target Pillar with leverage of the Instrumentation and Meta Data pillars. The Target pillar starts with establishing the correct questions to ask within the requirements process. These questions recognise the need to establish not only output, but people, process and data. You should ask a series of questions to capture this for the immediate requirement, but within the context of an overall Vision.
The second step is then Instrumenting the data and Meta Data. It is important to capture these efficiently and effectively using tools and techniques, but also to run profiling of the data to match to the model and check feasibility. Through the results of this process you can then work Collaboratively with the sponsor and stakeholders to solve the data and process requirements. Using data prototyping methods to illustrate the solution further assists in communicating the agreed output and the identified wrinkles which helps to build the collaboration through shared vision.
We find in our projects that following a structured approach to this part of the project yields results of building consensus, establishing gaps and building trust.
In one particular client engagement this improved the delivery velocity to the level that is the difference between success and failure. In another client engagement we are able to deliver very large and complex problems within an incredibly tight timescales.
The key point is that requirements definitions are there to build a shared contract that defines a solution that is achievable, therefore you need to include the DataOps analysis into the process to achieve the results that you want.
An ex colleague and I were talking a few days ago and he mentioned people don't want to buy DataOps. To this I have given some thought and can only come up with the conclusion that I agree, in the same way that I can agree that I don't want to buy running shoes, however I want to be fit and healthy. It is interesting to find so many people that are happy with their problems that a solution is less attractive.
The way to look at DataOps methodology and training is that it is an investment in yourself, or your organisation that enables you to tackle the needs that you struggle to make progress on. The needs that might resonate more with you are machine learning and AI, Digitisation, Legacy migration, Optimisation and cost reduction, improving your customer experience, and improving the speed to delivery and compliance of your Reporting and Analytics.
The DataOps methodology provides you with a recipe book of tools and techniques that if followed enables you to deliver the data driven organisation.
At one organisation that works with us, they found that working with the DataOps tools and techniques enabled them to deliver an important regulation in record time, rebuild the collaboration between stakeholders and form a template for future projects. For more information then please feel free to reach out to me.