News and Events

DataOps at Kinaesis

Posted by Benjamin Peterson on 10 May 2018
DataOps at Kinaesis

The core concepts behind DataOps drive everything that we do at Kinaesis.

Kinaesis DataOps is a powerful toolkit for creating data-driven business change, covering people, process and technology. We use it to manage the entire data pipeline, from requirements and vision to analytics and presentation.

We specialise in applying DataOps within the financial sector to achieve compliance, cost savings and revenue growth. Our approach to DataOps reflects the specific needs and challenges of the sector and our experience as long time practitioners and SMEs.

Where does Kinaesis DataOps come from?

Kinaesis DataOps is a new approach to delivering data, based on a methodology that has proved successful in software delivery. Our DataOps Pillars are more than a set of best practices; each of our pillars within DataOps blends lessons learnt in the software world with solutions appropriate for the more challenging world of data.

To truly maximise the value of DataOps requires more than knowledge of DevOps and tooling; it requires a deep familiarity with data management, analytics and sector specific needs. That is what drives our interpretation of DataOps into these Six Pillars:

Pic

Design: Make sure the data pipeline is driven by a business vision, not just by the data that happens to be available. Map out user journeys and visions to drive technical change.

Instrumentation: Instrument the data flow every step of the way using profiling, DQ and monitoring to create a clear view of data reliability and timeliness. Always present data quality information with actual data.

Global metadata: Maintain clear business definitions and models; keep them connected to your data and up to date.

Collaborative Analytics: Make sure that data consumers can collaborate together along with IT and data owners.

Governance: Meet quality, attestation and audit targets by applying proper version control and proper release management. Channel the output of instrumentation into strong exception handling and DQ processes.

Extensible platforms: Data needs to be leveraged and the business will always come up with new demands. Open standards, IT components that interoperate well, clear contracts and operating models are essential to ensure that new demands can always be met.

Background:

Pic

Back in 2008, a group of tried and tested software development practices combined with some new tooling that improved environment and release management were bundled together as DevOps. DevOps is the principal inspiration behind DataOps and generated the requirement for version control, release management, environment management and documentation.

DataOps takes the emphasis on collaboration, quick release cycles and iterative refinement from Agile development methodology.

The focus on instrumentation comes from Lean; Lean methodologies propose that fully instrumenting a supply chain is necessary to optimise it. Our DataOps vision takes this a little further - not just the supply chain itself, but the data, needs to be instrumented.

Finally, from User Experience (UX) development comes a toolkit for generating vision and requirements by interaction and exploration with the business. This is essential if DataOps is to be more than just optimisation!