The New Data-Plumbing that Gets Organisations Going in Double-Quick Time

The New Data-Plumbing that Gets Organisations Going in Double-Quick Time

Elaine Fletcher

Partner at Teradata and Business Intelligence Strategy Specialist

Views 692

The New Data-Plumbing that Gets Organisations Going in Double-Quick Time

04.11.2016 06:15 am

Banks now have to squeeze the maximum value from all of their data faster and more accurately than ever.

Not only must they compete with the initiatives of rivals, new demands are being placed on them from regulators and industry watchdogs eager for accurate and easily digestible reports and audits.

However, as data volumes balloon, severe problems are emerging with regard to the quality, consistency, provenance, duplication and accessibility of data in large enterprises such as banks, insurance companies and telecommunications providers.

Resolving these challenges using traditional methods diverts key staff and costs companies unnecessarily large amounts of time and budget, putting them at risk of losing out to competitors or failing to comply with regulatory requirements.

Tthe solution that is rapidly becoming established as roughly 50% to 75% faster, cheaper and more accurate, is the highly automated Metadata Ddriven Estate (MDE).

MDE’s high degree of automation minimises the need for costly/slow humans to write software. It accurately discovers and documents the “as is” movements and transformations of data that occur between the source systems (e.g. point of sale) through intermediate systems (e.g. data warehouse) then over to the business end users.  The industry term for this is end to end Data Lineage, it is a requirement of BCBS 239 regulatory compliance. MDE also analyses how data is used by business end users and by applications such as marketing campaigns.  This allows you to easily change the business rules that transform/re-purpose your data so that you can quickly deliver new functionality or add new data feeds in pace with business needs.

This highly advanced form of data-plumbing is enabling businesses to save millions of pounds by integrating faster, more accurately and more cheaply than ever, irrespective of technology, e.g. new hadoop data platforms, legacy mainframes, specialist Teradata appliances.

Meeting regulatory requirements

In banking, for example, the Metadata Driven Estate is allowing major institutions to meet the stringent requirements of regulations such as BCBS 239, where establishing data quality and provenance are essential. For example, this year a leading UK financial services institution has used MDE to reduce the total cost of ownership of their massive 2 petabyte data estate and deliver BCBS 239 data lineage and data governance, so improving competitiveness while achieving regulatory compliance, all in less than six months!  Prior to this they had tried for several years but were defeated by the complexity and sheer volumes of their >500 million queries into >4,000 Databases from > 3,600 Users. MDE used automation to accurately process this complex estate, quickly and cheaply.

The MDE approach uses automation to finger-print the data back to its point of origin, enabling the regulator to access data that is wholly and demonstrably accurate and in compliant format, with the option of drilling down to check accuracy.

Whereas achieving this with conventional methods has been taking many months, or in some cases, the entire year, the MDE approach to achieve what is required for regulatory compliance reporting is delivered as a fixed fee four week sprint.  It does not rely on you having subject matter experts or accurate documentation on hand, instead it reads the metadata that is left behind (footprints in the sand) by all IT systems as they process their day to day workload – so MDE gets its information “from the horses mouth”.

Separation of banking functions

Strategic re-organisation e.g. the separation of investment from retail and commercial banking to meet regulatory requirements is also placing huge demands on the ability of institutions to determine the provenance of data from thousands of databases, many of which will have been shared, formally or informally for years.  MDE automates the un-picking process and accelerates the re-purposing of data to support the new organisation(s).

The massive degree of duplication within financial institutions, where some datasets are reproduced in different guises as many as 60 times, also adds to the complexity of achieving compliant separation.  MDE identifies “safe duplicates” that can be deleted and queries automatically re-routed to the survivor instance.

The metadata-driven approach is again slashing the time and costs involved in these operations, in one case reducing the length of the project from a predicted six months to just two. No less importantly, the process of cleaning up the data and reducing duplication is enabling banks to reduce the size of their IT estates by as much as 30 per cent while at the same time having all the tools that allow regulators to see what they have done and how.

Revenue generation                                                                                                                                                                                                                                                                                                                                                                                                       Use of MDE to generate revenue is also growing – most obviously when it cuts the time it takes to deploy multi-channel marketing campaigns that require highly-flexible and quickly-configurable data-hungry applications.

Without MDE for banks the sheer length of time that it takes to provision marketing applications can be such a drawback that it often negates all the benefits. Although the marketing application can be fully functional in hours, using traditional methods it can take 18 months to provision it with reliably accurate data that gives the company a single view of the customer. Hardly surprising when it commonly involves sorting through petabytes of data coming from 20 or 30 different databases.

However, MDE means provisioning can take place in weeks, rather than months, regardless of the technology involved, slashing the cost and time to make marketing initiatives a reality, allowing the organisation to build its market share with much greater momentum.

The speed, flexibility and reliability of MDE will transform the ability of any data-driven organisation to meet all its challenges in regulation and innovation.

Surely any technology that hugely improves business performance in such short time and at such reduced cost, really is a revolution in its own right?

Latest blogs

Nóra Bézi Bitrise

The future of finance and banking apps: balancing security and innovation

The fintech revolution in the mobile space Fintech mobile applications are rapidly becoming the norm when it comes to handling our personal finances. As these apps grow in popularity and continue to define our relationship with money, the number Read more »

Will Hurst Monevo

How can lenders leverage open banking data to tackle Covid-19 pandemic challenges

We have seen extraordinary levels of Government-backed support made available to the public and businesses alike in responding to the health crisis by our banks through payment holidays, grants and the furlough scheme. While we keep collective Read more »

Nish Kotecha and Noslen Suárez PhD Finboot

How blockchain can help us have trust in the food we eat

Today’s food supply chains are global, connected and generally efficient, but the COVID-19 pandemic has shone a spotlight on areas of weakness. The urgent need for robust and resilient systems and processes has been brought sharply into focus, and Read more »

Chris Miller RSA Security

Back to Normality: Five Steps to Stay Resilient After Disruption

The financial services sector has lived through many global disruptions, but the nature of recent events has put an unprecedented strain on operational resilience; from needing to ensure critical functions could continue with skeleton staff and Read more »


How Technology Has Disrupted the Used Car Buying Experience

We’ve seen many fields change rapidly as a result of the integration of modern technological advancements over the last couple of decades. And it looks like more is coming on the horizon as well, judging by current trends. One of the markets that Read more »

Related Blogs

Abbey Shasore Factbook

Successful investment reporting: it’s always about the data

Heads of Performance or Client Reporting at any asset management firm are responsible for the outputs to the end consumer. Their biggest worry is always: ‘Can I trust the numbers (and text) on those outputs? Is it ‘good to go’? Has it been brought Read more »

Rossana Thomas Fiserv

Why Data Rules the Payment

Intelligent information is invaluable to businesses. In the payments industry, data can open the doors to innovation, improved operations, and better, quicker decision making. In fact, the value of payment data has begun to rival that of the payment Read more »

Shaun Collings Pure Storage

Change Starts from Within: Overcoming Digital Transformation Barriers in the Public Sector

Across the public sector there are various forms of digital transformation taking place, at different stages of progress.  Part of the driver for this is that citizens now expect a level of service akin to what they receive from private companies. Read more »

Daniel Fried Veeam

Data Protection Day 2020: De-Risking in the Era of Transparency

The issue of data protection and privacy was, until recently, a conversation confined to a specific group of people within an organisation. Unless you were an IT consultant or a corporate lawyer, privacy compliance was something somebody else took Read more »

Andrew Stevens Quadient

Banks to Use Mobile Channels and Customer Data to Differentiate Themselves in 2020

Digitally native neo-banks will continue to differentiate themselves in the market in 2020 by the experience they offer to customers. As a result, these banks will begin to pick up the critical minimum number of customers to become profitable, Read more »

Free Newsletter Sign-up
+44 (0) 208 819 32 53 +44 (0) 173 261 71 47
Download Our Mobile App
Financial It Youtube channel