Welcome to
our blog

Dark data is slightly less sinister than it sounds. It is neither the source of fake news propagation, nor some rehashed, conspiracy theory catchphrase. A more appropriate term would be “feral data”, because the data was in fact captured at some point in the past, but then released in its original, “untamed state”, into multiple digital and/or physical repositories. Dark data is thus simply untapped and often, unknown data already in possession of organisations.

So, if it is simply a question of knowing that you have a wealth of data and where to find it, why is the “Dark Data Analytics” topic all the rage?

The main reason why collected data slips out of focus and fade into darkness, is because approximately 90% of it is unstructured and for that very reason is found over a range of disparate sources and formats. It is thus messy and fragmented and not found in easily accessible relational databases, or in tabular form, for quick analysis.

Typical unstructured data sources are log files, IoT device notifications, documents, presentations, raw survey data, correspondence (email, text, hardcopy letters), scanned copies of old, yet relevant documents, recorded video and audio and financial statements. Until fairly recently, the technology to tap into and query these unstructured, dark data sources, at a scale and manner that was both time and cost effective, simply did not exist. 

Dark data contains a wealth of descriptive information, relevant to situational context, location, discipline, date, time and perhaps even individual user/customer behaviour. Today, the technology exists to extract, classify and aggregate dark data with existing relational databases. Together, these become invaluable and vastly enriched source material for a multitude of AI-driven platforms that deliver diagnostic, predictive and prescriptive analytics, using all available data (and not just the usual 2-5% of structured data), hence being more accurate, trustworthy and valuable.

So, why is this an opportunity for asset intensive industries and why now? 

1.      Dark data on tap

Asset intensive organisations are complex beasts, shaped and layered with bolt-on acquisitions over several decades e.g. oil & gas, mining, energy & utilities, infrastructure, real estate, manufacturing, healthcare and aviation. They carry vast stores of dark data in both digital and hardcopy formats, acquired over many decades, but because of the scale, fragmented nature and lack of holistic solutions, have not taken advantage to leverage these into insight, strategy and action. 

2.      Looming threats and challenges

The Oil & Gas industry was facing a triple threat, of price volatility, workforce challenges (ageing and shrinking) and public and legislative pressure such as Environmental, Social and Corporate Governance (ESG) before Covid-19. The repercussions of Covid-19 have however given us a glimpse of a real-world scenario where proactive, precision tracking analysis and management of capital investments and automation and remote control of operations, are becoming necessities and not “nice to haves”. Not only are these required for the improvement of operations and reporting, but critically, for plans and tangible action to safeguard against environmental impact, especially during extreme events. Further, to maintain exemplary health and safety standards that protect employees and wider communities (engaged with the organisation) from harm and injury. Such a proactive approach requires insight into past events and intelligence, to harness effective action with fewer resources. This requires contextual and rich data as a baseline. Similar situations are to be found across the Energy and Utility asset industries.

3.      Interoperability and interconnectedness

The operation and maintenance of large, expensive, or remote assets present many challenges, from optimising complex global supply chains to mitigating the impact of unpredictable weather events. Poor management of these operational processes can increase costs significantly and have a negative impact on profitability.

The flipside is however also true. By digitally mapping each asset, together with its historic and current data, it is possible to leverage the digital footprint of assets, together with AI powered Asset Performance Management systems (Energy, Utilities, Manufacturing) and Building Automation Systems (Real estate), into wider asset portfolio and supply chain management systems.

Individual assets thus become digitally visible and interconnected within the wider asset network. With the enrichment of additional volumes of contextual data to the asset network, it becomes possible to develop an interoperable asset platform that responds automatically to signals from the surrounding environment and supply chain and thereby exploit efficiencies and or smooth any cyclical downturns (e.g. power generation and distribution assets; real estate occupancy and energy consumption patterns, individual oil & gas wells within a wider reservoir).   

4.      User and customer experience 

Asset management strategies and operations are gradually changing focus from a purely functional orientation to more value added, user and/or customer experience (e.g. transport infrastructure, commercial real estate). New data driven operational models are required that feed off contextual nuances, embedded in dark data sources. These are then used to interpret and diagnose user/customer behaviour, but then used further to improve user experiences, via recommender systems and wider predictive platforms. 

What is required to turn dark data sources from opportunity to real benefits? 

• Sound leadership within asset intensive organisations that understands the underlying business objectives and anticipated outcomes.
• A holistic approach to data management solutions and not a patchwork of self-interest, product solutions. The transition to full digitalisation requires the creation of an efficient, comprehensive and robust digital nervous system. A datascape that contains all organisational assets, knowledge and wisdom, that is both quick to access and identify, from millions of records and objects. This requires intelligent multi-faceted pre-classification of the data, together with autonomous management solutions and AI.

Kyanite360 is a digital platform developer and together with our partners and wider ecosystem, we are building platform opportunities and supporting customers in building future-fit organisations.