The Big Data Landscape – My Predictions for 2018…

In 2018 we are rapidly entering what I would like to call ‘Big Data 3.0’. This is the age of ‘Converged Big Data’ where its various complementary technologies – Data Science, DevOps, Business Automation begin to all come together to solve complex industry challenges in areas as diverse as Manufacturing, Insurance, IoT, Smart Cities and Banking. 

(Image Credit – Simplilearn)

First, we had Big Data 1.0…

In the first pass of Big Data era, Hadoop was the low-cost storage solution. Companies saved tens of billions of dollars from costly and inflexible enterprise data warehouse (EDW) projects. Nearly every large organization has begun deploying Hadoop as an Enterprise Landing Zone (ELZ) to augment an EDW. The early corporate movers working with the leading vendors more or less figured out the kinks in the technology as applied to their business challenges.

We just passed Big Data 2.0…

As adoption patterns matured and Big Data included projects such as YARN, Spark, and Hive, customers began deploying Big Data to business challenges such as Fraud Detection, Customer Journey et al and began to realize business value from it. Adoption has indeed begun skyrocketing at verticals like Banking, Telecom, Manufacturing & Insurance. The monolithic Big Data market has begun segmenting into well-defined categories – Infrastructure providers, Streaming Data companies, Data Analysis providers, SQL on Hadoop solutions, full-fledged machine learning toolsets etc.

With that said, let us look at my Big Data predictions for 2018.

Trend #1 Big Data 3.0 – where Data fuels Digital Transformation…

Fortune 5000 process large amounts of customer information daily. This is especially true in areas touched by IoT – power and utilities, manufacturing and connected car. However, they have been sorely lacking in their capacity to interpret this in a form that is meaningful to their customers and their business. In areas such as Banking & Insurance, this can greatly help arrive at a real-time understanding of not just the risks posed by a customer/partner relationship (from a credit risk/AML standpoint) but also an ability to increase the returns per client relationship. Digital Transformation can only be fueled by data assets. In 2018, more companies will tie these trends together moving projects from POC to production.

The Six Strategic Questions Every Bank Should Answer with Big Data & AI in 2018…

Trend #2 ‘Predictive Analytics on Hadoop’ projects begin to proliferate…

I have written extensively about efforts to infuse business processes with machine learning. Predictive analytics have typically resembled a line of business project or initiative. The benefits of the learning from localized application initiatives are largely lost to the larger organization if one doesn’t  allow multiple applications and business initiatives to access the models built. In 2018, machine learning expands across more usecases from the mundane (fraud detection, customer churn prediction to customer journey) to the new age (virtual reality, conversational interfaces, chatbots, customer behavior analysis, video/facial recognition) etc. Demand for data scientists will increase.

In areas around Industrie 4.0, Oil & Energy, Utilities – billions of endpoints will send data over to edge nodes and backend AI services which will lead to better business planning, real-time decisions and a higher degree of automation & efficiency across a range of processes. The underpinning data capability around these will be a Data Lake.

This is an area both Big Data and AI have begun to influence in a huge way. 2018 will be the year in which every large and medium-sized company will have an AI strategy built on Big Data techniques. Companies will begin exposing their AI models over the cloud using APIs as shown above using a Models as a Service architecture.

Trend #3 Big Data begins to take baby steps towards replacing the Enterprise Data Warehouse

Infrastructure vendors have been aiming to first augment and then replace EDW systems. As the ability of projects that perform SQL-on-Hadoop, data governance and audit matures, Hadoop will slowly begin replacing EDW footprint. The key capabilities that Data Lakes usually lack from an EDW standpoint – around OLAP, performance reporting will be augmented by niche technology partners. While this is a change that will easily take years, 2018 is when it begins. Expect migrations where clients have not really been using the full power of EDWs beyond simple relational schemas and log data etc to be the first candidates for this migration.

Trend #4 Cybersecurity pivots into Big Data…

Big Data is now the standard by which forward-looking companies will perform their Cybersecurity and threat modeling. Let us take an example to understand what this means from an industry standpoint. For instance, in Banking, in addition to general network level security, we can categorize business level security considerations into four specific buckets –   general fraud, credit card fraud, AML compliance, and cybersecurity. The current best practice in the banking industry is to encourage a certain amount of convergence in the back-end data silos/infrastructure across all of the fraud types – literally in the tens.  Forward-looking enterprises are now building cybersecurity data lakes to aggregate & consolidate all digital banking information, wire data, payment data, credit card swipes, other telemetry data (ATM & POS)  etc in one place to do security analytics. This pivot to a Data Lake & Big Data can pay off in a big way.

The reason this convergence is helpful is that across all of these different fraud types, the common thread is that the fraud is increasingly digital (or internet based) and they fraudster rings are becoming more sophisticated every day. To detect these infinitesimally small patterns, an analytic approach beyond the existing rules-based approach is key to understand for instance – location-based patterns in terms of where transactions took place, Social Graph-based patterns and Patterns which can commingle real-time & historical data to derive insights. This capability is only possible via a Big Data-enabled stack.

Trend #5 Regulators Demand Big Data – PSD2,GPDR et al…

The common thread across virtually a range of business processes in verticals such as Banking, Insurance, and Retail is the fact that they are regulated by a national or supranational authority. In Banking, across the front, mid and back office, processes ranging from risk data aggregation/reporting, customer onboarding, loan approvals, financial crimes compliance (AML, KYC, CRS & FATCA), enterprise financial reporting  & Cyber Security etc – all need to produce verifiable, high fidelity and auditable reports. Regulators have woken up to the fact that all of these areas can benefit from universal access to accurate, cleansed and well-governed cross-organization data from a range of Book Of Record systems.

A POV on Bank Stress Testing – CCAR & DFAST..

Further, applying techniques for data processing such as in-memory processing, the process of scenario analysis, computing,  & reporting on this data (reg reports/risk scorecards/dashboards etc) can be vastly enhanced. They can be made more real time in response to data about using market movements to understand granular risk concentrations. Finally, model management techniques can be clearly defined and standardized across a large organization. RegTechs or startups focused on the risk and compliance space are already leveraging these techniques across a host of areas identified above.

Trend #6 Data Monetization begins to take off…

The simplest and easiest way to monetize data is to begin collecting disparate data generated during the course of regular operations. An example in Retail Banking is to collect data on customer branch visits, online banking usage logs, clickstreams etc. Once collected, the newer data needs to be fused with existing Book of Record Transaction (BORT) data to then obtain added intelligence on branch utilization, branch design & optimization, customer service improvements etc. It is very important to ensure that the right business metrics are agreed upon and tracked across the monetization journey. Expect Data Monetization projects to take off in 2018 with verticals like Telecom, Banking, and Insurance to take the lead on these initiatives.

The Tao of Data Monetization in Banking and Insurance & Strategies to Achieve the Same…

Trend #7 Data Native Architectures converge with Cloud Native Architectures…

Most Cloud Native Architectures are designed in response to Digital Business initiatives – where it is important to personalize and to track minute customer interactions. The main components of a Cloud Native Platform are shown below and the vast majority of these leverage a microservices based design. Given all this, it is important to note that a Big Data stack based on Hadoop (Gen 2) is not just a data processing platform. It has multiple personas – a real-time, streaming data, interactive platform that can perform any kind of data processing (batch, analytical, in memory & graph based) while providing search, messaging & governance capabilities. Thus, Hadoop provides not just massive data storage capabilities but also provides multiple frameworks to process the data resulting in response times of milliseconds with the utmost reliability whether that be real-time data or historical processing of backend data. My bet on 2018 is that these capabilities will increasingly be harnessed as part of a DevOps process to develop a microservices based deployment.


 Big Data will continue to expand exponentially across global businesses in 2018. As with most disruptive innovation, it will also create layers of complexity and opportunity for Enterprise IT. Whatever be the kind of business model – tracking user behavior or location sensitive pricing or business process automation etc – the end goal of IT architecture should be to create enterprise business applications that are heavily data insight and analytics-driven.

Why Kubernetes Will Be A Transformational Cloud Technology..

It is 2018 and Enterprise IT does not question the value of Containerized applications anymore. Given the move to adopting DevOps and Cloud Native Architectures, it is critical to leverage container oriented capabilities to bring together development and operations teams to solve Digital business challenges. However, the lack of a standard control plane for these containerized deployments was always going to be a challenge. Google’s Kubernetes (kube or k8s), an open source container orchestration platform,  is rapidly becoming the defacto standard on how Cloud Native applications are architected, composed, deployed, and managed.

Kubernetes outshines competition…

First off, a deep dive on Kubernetes is provided below for those who are beginning their evaluation of the platform.

Kubernetes – Container Orchestration for the Software Defined Data Center (SDDC)..(5/7)

With it’s Google pedigree, K8s is the only container orchestration platform that is proven at scale in the web-scale, cloud-native world. K8s predecessors Omega/Borg manage vast containerized deployments that deliver services such as Google Search, Gmail, and YouTube.

Let us enumerate both the technology and business advantages that are captured in the below illustration.

Technical Advantages…

With its focus on grouping containers together into logical units called pods, K8s enables lightweight deployment of microservice based multi-tier applications. The service abstraction then gives a set of logical pods an external facing IP address.A Service can be discovered by other services as well as scaled and load balanced independently. Labels (key, value) pairs can be attached to any of the above resources. K8s is designed for both stateless and stateful app as it supports mounting both ephemeral as well as persistent storage volumes.

Service as an architectural construct called (a group of pods exposed to the external world via an IP Address) enables a high-level focus on the deployment, performance, and behavior of an application rather than its underlying infrastructure.

Kubernetes also provides autoscaling (both up and down) to accommodate usage spikes. It also provides load balancing to ensure that usage across hosts is evenly balanced. The Controller also supports rolling updates/canary deployments etc to ensure that applications can be seamlessly and incrementally upgraded.

Developers and Operations can dictate whether the application works on a single container or a group of containers without any impact to the application.

These straightforward concepts enable a range of architectures from the legacy stateful to the microservices to IoT land – data-intensive applications & serverless apps – to be built on k8s.

A Robust Roadmap…

With Google and Red Hat leading a healthy community of contributors, the just-released Kubernetes 1.9 added many useful features. First, it provides a higher degree control over clusters, added detailed storage metrics and makes it an extensible architecture. It also improves many aspects of the API. It also moves Windows support into beta. Coupled with work ongoing in the Open Service Broker API, this moves the needle on support for hybrid architectures one step closer. Just to provide an idea of the robustness of development, this release is expected to include 38 features spanning security, cluster lifecycle management, APIs, networking, storage and additional functionality. [1]

Business & Ecosystem Advantages…

K8s as an open source orchestrator is now a foundational component of market-leading platforms such as Red Hat’s OpenShift and (IaaS Clouds such as) AWS ECS Container Service/Azure/VMWare Pivotal CloudFoundry. There is no fear of lockin around this container standard. 2017 saw a shakeout in this technology segment as competition to K8s essentially folded and announced plans to support the orchestrator. Platforms such as Docker, Mesos, CoreOS now integrate with & support Kubernetes at different levels.

Over the last three years, they have now emerged over 50 Kubernetes powered platforms and distributions. The Cloud Native Computing Foundation’s (CNCF) Kubernetes Conformance model includes API standards for networking and storage. The key benefit to developers is that applications coded for k8s are pretty much lockin free from both an orchestration and storage standpoint.

Credit – CNCF

In the last year, k8s has made tremendous strides in project documentation, developer help & quickstarts, and on improving the overall operator experience.  The 2017 KubeCon held in Austin, TX drew 4200 attendees and had multiple tracks covering everything from CI/CD Pipelines, Operational experience and Special Interest Groups (SIG) covering a range of non-functional areas such as performance and security.

The Road Ahead…

The Cloud Native landscape has an amazing amount of change every year but it is a safe bet that Kubernetes given its massive open source ecosystem and modular architecture and design is a safe bet to emerge as the defacto standard in container orchestration.

Four strategic areas of advances for Kubernetes in 2018 include –

  1. Playing the container factotum for a range of cloud architectures
  2. Refinement of k8s deployments around cloud native microservices based architectures. These include operating in an architecture with Service Meshes, Serverless Computing & Chaos Engineering concepts
  3. Increased vertical industry adoption especially around OpenStack NFV and Telco
  4. Adoption in hybrid cloud usecases


[1] Kubernetes 1.9 –

The 12 Software Architectures That Will Matter in Financial Services in 2018 & Beyond…

Over the last three years, we have examined a succession of business issues in the various sectors of financial services on this blog. These have ranged from the mundane (Trading, Risk management, Market Surveillance, Fraud detection, AML et al) to the transformative (Robo advisors, Customer Journeys, Blockchain, Bitcoin etc). We have also examined the changing paradigms in enterprise architecture – moving from siloed monolithic applications to cloud-native software. This blog summarizes the most 12 important technical posts on innovative application architectures.


Having spent the majority of my career working in Banking and Financial Services has made for a fascinating time. It is amazing to witness business transformation begin to occur across the landscape. However, this transformation is occurring on repeatedly discussed themes. A key challenge that CXOs and Enterprise Architecture teams face is how to deploy much-discussed technologies such as Cloud platforms, Big Data, Enterprise Middleware and AI in real-world architectures.  This blog post sums up eleven real-world application architectures that industry leaders can use as a good reference point for their own implementations.

The common theme to all of the below architectures –

  1. A focus on Cloud native concepts including microservices, lightweight backends, containers
  2. Design Patterns that encourage new age Data Management techniques including Hadoop and Spark
  3. Cloud-agnostic – whether that is public cloud or private cloud
  4. Integrating business process management and business rules engines as first-class citizens
  5. 100% Open Source

#1 Design and Architecture of a Real World Trading Platform…

Design and Architecture of a Real World Trading Platform.. (2/3)

#2 Big Data driven Architecture for Credit and Market Risk Management…

How a Pioneering Bank leverages Hadoop for Enterprise Risk Data Aggregation & Reporting..

#3 Reference Architecture for Big Data-enabled CyberSecurity…

Cybersecurity and the Next Generation Datacenter..(2/4)

#4 Reference Architecture for Payment Card Fraud Detection…

Hadoop counters Credit Card Fraud..(2/3)

#5 Design and Architecture of a Robo-Advisor Platform…

Design and Architecture of A Robo-Advisor Platform..(3/3)

#6 Reference Architecture for Customer Journeys and Single View of a Customer…

Demystifying Digital – Reference Architecture for Single View of Customer / Customer 360..(3/3)

#7 A Reference Architecture for the Open Banking Standard…

A Reference Architecture for The Open Banking Standard..

#8 The Architecture of Blockchain…

The Architecture of Blockchain..(4/5)

#9 The Architecture of Bitcoin…

The Architecture of Bitcoin..(2/5)

#10 How to Re-Architect a Wealth Management Office…

Next Gen Wealth Management Architecture..(3/3)

#11 Reference Architecture for Market Surveillance – CAT, MAR, MiFID II et al…

The Definitive Reference Architecture for Market Surveillance (CAT, UMIR and MiFiD II) in Capital Markets..

#12 Logical Architecture for Operational Risk Management…

Infographic: Logical Architecture for Operational Risk Management


With each passing quarter, financial services is a business that looks increasingly in danger of disintermediation. CXOs have no alternative but to digitize their businesses. IT will be forced to support cloud-native technologies in both key areas – applications and infrastructure in pursuit of business goals. Developers will also be at the forefront of this change. Eventually, quality of Enterprise Architecture decides business destiny.

My Final Post for 2017: How an Enterprise PaaS enables Enterprise Architecture…

With DevOps and Container based automation rapidly gaining industry mindshare in 2017, PaaS is emerging as a “fit for purpose” technology for Digital Projects. With the PaaS market beginning to mature, different product subcategories within the main umbrella are being proposed – Structured PaaS, Containers as a Service, Unstructured PaaS etc. For now, these subcategory definitions look largely academic as technology follows business challenges & any such segmentation should largely follow from the challenges being solved. PaaS is no different. My goal for this post then is to approach the market from the standpoint of the key (business) capabilities in an Enterprise Architecture that an industrial grade PaaS should enable, no matter where it falls on the spectrum of PaaS platforms.

Enterprise Architecture based on a PaaS…

Enterprise Architecture typically spans four different areas – 1) Business Architecture, 2) Data Architecture, 3) Application Design & 4) Deployment Architecture. Given the rapidly maturing cloud-based delivery models (IaaS and SaaS) – many EA standards now include compulsory cloud-native awareness and design across the four domains.

We posit that in 2018, PaaS has emerged as the most important driver of an enterprise architecture. PaaS technology can accomplish a majority of the goals of an EA in a variety of ways, as we will cover below.

The definition of what constitutes a Platform As a Service (PaaS) continue to vary. However, there is no disagreement that PaaS enables the easy but robust buildout of a range of Cloud Native architectures.  The vision of a PaaS is to ultimately enable massive gains in productivity for application developers that intend to leverage a cloud-based IaaS. At the same time, advances in open source technology in 2017 are ensuring management seamlessness & simplicity for Cloud Admins.

The below graphic illustrates the core building blocks of an enterprise architecture based on a PaaS.

The Foundational Services a PaaS provides Enterprise Architects cover a range of areas as depicted above..

Core Benefits of Adopting an Industrial Strength PaaS…

PaaS technology was originally developed as a way of helping developers with a smooth experience in developing polyglot applications. With the advent of Docker and Kubernetes, the focus has also shifted to enabling CI/CD pipelines and in achieving seamless deployment on a cloud-based infrastructure. The following areas confer significant PaaS capabilities that EA (Enterprise Architecture) teams would otherwise have to cobble themselves:

  • Cloud Native via Containers – An industrial grade PaaS abstracts away any & all underlying Hardware/IaaS concerns by leveraging containers. However, it also ensures that the PaaS can leverage the services of the underlying IaaS whether that is Amazon AWS, Microsoft Azure, OpenStack or VMWare. At a minimum, as long as the cloud supports defacto standards such as Linux and Docker, the PaaS can host any platform or application or package as well as support migrations across the underlying Clouds across Dev/Test/QA/Prod environments. Enterprise IT should be able to easily split workloads across these different clouds based on business needs.The key to all of this is to agree on the Container as the standard contract between the PaaS and the IaaS layers. Thus, the few leading PaaS vendors such as OpenShift have adopted standards-based container technology for development, packaging and deploying applications. Further, the availability of a Container registry is also very important to guarantee the provenance and safety of commonly used Docker Images.
  • Developer Services – A PaaS includes development tools that can vastly reduce the amount of time to develop complex n-tier applications. The developer experience needs to be smooth. These should include at a minimum either Docker images or, an easy plugin-based integration that covers a range of enterprise runtimes such as workflow, Big Data libraries, Identity Management, API Management, Broker based messaging integration, Search and Security services. Based on the architectural requirements of a given business project, the PaaS should be able to offer a natural stack of default options for the above services typically using a template such as a simple Dockerfile that calls out the default OS, JVM version & the other runtime dependencies of the application. The PaaS then generates a barebones application that the developer can then just fill in the blanks with their source code. This typically done using a command line, or web interface or by invoking an API. This unified experience then carries over across the CI/CD pipelines, deployment and then management. This way, everyone in the organization speaks & adheres to a common development vocabulary.
  • Mobile Application Development –For developers, a PaaS should encompass the easy provisioning of cloud resources through the application lifecycle while enabling application development using microservices. However, leading PaaS providers also include toolkits for cross-platform development capabilities for mobile devices and a range of browsers.
  • CI – A robust PaaS provides facilities for Continuous integration (CI). It does this in several ways. Firstly, code from multiple team members is checked (push and merge code pull requests) into a common source control repository (typically based on Git). This supports constant check-ins and automated checks/gates are added to run various kinds of tests. Further included are capabilities such as developer workflow based on includes Git where a push event causes a Docker image build.
  • Continuous Delivery – The PaaS can then automate all steps required to deliver the application binaries from a CI standpoint to delivery using CD. These involve supporting automated testing, code dependency checks etc and seamlessly promoting images from one environment to the other.
  • Continuous Deployment – Once the PaaS has containerized workloads & deploy them, the next step is to orchestrate them. The PaaS includes capabilities that can then deploy the application on a family of containers & load balance/manage their runtime footprint. This capability is typically provided by a container orchestration layer such as Kubernetes or Mesos. A range of services around HA, service discovery etc are provided by this layer.
  • Runtime Characteristics – The PaaS finally simplifies how complex n-tier applications are scheduled and then deployed across tiers, how these groups of containers that constitute an application leverage the network & the underlying storage, how they’re exposed to consuming applications via request routing, how the health of various groups of containers (called Pods in the case of Kubernetes) is managed, ensuring high availability and finally, zero downtime deployments.


PaaS provides enterprise architecture teams with a range of capabilities that enable Cloud Native application development and delivery. These range from i) enabling CI/CD capabilities for developers via application automation ii) providing a range of container orchestration capabilities. These enable rapid deployment, version control, rolling updates etc. All of these ultimately enable rapid digital application development. 2018 onwards, Enterprise Architects can only neglect a serious look at PaaS at their peril.

The Six Strategic Questions Every Bank Should Answer with Big Data & AI in 2018…

After a decade of focusing on compliance with regulatory mandates, Banks are back at fixating on technology innovation. The reason is obvious – over the last five years, Silicon Valley majors and FinTechs have begun to rapidly encroach on the highest profit areas of the banking business. The race is on to create next-generation financial services ecosystems in a variety of areas ranging from Retail Banking, Capital Markets, and Wealth Management. The common thread to all these is massive volumes of Data & Advanced analytics on the data. Given that almost every large and small bank has a Big Data & AI strategy in place, it makes sense for us to highlight six key areas where they should all first direct and then benchmark their efforts from an innovation standpoint.

Global Banking in 2016-17…

As 2017 draws to a close, the days of growth and sky-high stock market valuations seem to be largely back. McKinsey Research posits that while the global banking industry appears quite healthy outwardly, profits are at best flat or even falling across geographies[1]. For the seventh year in a row, the industry’s ROE (Return on Equity) was between 8-10%. For 2016, the industry’s ROE was down a full percentage point from 2015, raising concerns about profitability across the board. There are however innovators that are doing well due to their strong focus on execution.

Banks have overall been very slow to respond to the onslaught of the digital business led by Amazon, Google/Alphabet, PayPal and the horde of FinTechs. What all of these disruptors do better than Banks is to harness customer data to drive offerings that appeal to neglected banking consumers who are already used to using these services every waking hour in their lives.

As technology continues to advance and data becomes more available, the twin forces of competition & regulation, are driving overall innovation in across banking. Capital Markets players are using AI in a range of areas from optimising trading execution, contract pricing, strategy backtesting to risk & compliance.

In the staider Retail Banking & Asset Management areas, profitable areas such as customer lending, consumer payments &  wealth management are slowly being disrupted at the cost of established banks. What also lies behind this disruption is the FinTech’s ability to pick and choose the (profitable) areas they want to compete in, their minimal overhead as opposed to & an advanced ability to work with data generated constantly by customer interactions by deploying algorithms that mine historical data & combine it in ways that reveal new insights.

I posit that there are six strategic questions that Banking institutions of all stripes need to glean from their Big Data (& AI) projects. This with a view to attaining sustainable growth for the foreseeable future  –

    • How do we know more about our customers?
    • How do we manage regulation and turn it into a source of lasting competitive advantage?
    • How can we increase our digital quotient in a way that enables us to enter new businesses?
    • How can this deluge of information drive business insight?
    • How can we drive Business transformation both within the Bank and disarm competition?
    • How can this information drive agility in customer responsiveness?
Every Bank has to aim to answer these six questions using Big Data & AI.

Question #1 How much do we know about our customers..really?

Financial institutions, including retail banks, capital markets, payment networks etc process large amounts of customer information daily. However, they have been sorely lacking in their capability to understand their customer profiles as one whole and to interpret this in a form that is meaningful to their business. The ability to do this can result in an understanding of not just the risks posed by this relationship (from a credit risk/AML standpoint) but also an ability to increase the returns per client relationship. This is an area Big Data and AI can influence in a huge way.

Demystifying Digital – Why Customer 360 is the Foundational Digital Capability – ..(1/3)

Question #2 How do we manage the Regulatory Onslaught and Turn it into Competitive Advantage?

There exist two primary reasons for Enterprises such as Banks, Insurers, Payment Providers and FinTechs to pursue best in class Risk Management Processes and Platforms. The first need in Capital Markets is compliance driven by various regulatory reporting mandates such as the Basel Reporting Requirements, the FRTB, the Dodd‐Frank Act, Solvency II, CCAR and CAT/MiFID II in the United States & the EU. The second reason (common to all kinds of Banking) is the need to drive top-line sales growth for both individual and institutional clients.

We have long advocated for the implementation of Big Data across both the areas. The common thread across virtually every business processes across the front, mid and back office is risk management.  Processes ranging from risk data aggregation/reporting, customer onboarding, loan approvals, financial crimes compliance (AML, KYC, CRS & FATCA), enterprise financial reporting  & Cyber Security etc can benefit from universal access to accurate, cleansed and well-governed cross-organization data from a range of Book Of Record systems. Further, applying techniques for data processing such as in-memory processing, the process of scenario analysis, computing,  & reporting on this data (reg reports/risk scorecards/dashboards etc) can be vastly enhanced. They can be made more real time in response to data about using market movements to understand granular risk concentrations. Finally, model management techniques can be clearly defined and standardized across a large organization. RegTechs or startups focused on the risk and compliance space are already leveraging these techniques across a host of areas identified above.

Risk Management – Industry Insights & Reference Architectures…

Question #3 Increase your Digital Quotient…

For decades, Banks have had a monopoly on the financial business. The last few years have seen both FinTechs and other players such as Amazon, Alibaba, Facebook etc enter lucrative areas in banking. These areas include Consumer lending, financial advisory etc. The keyword in all of this is ‘Digital Disintermediation’ and regulators have also begun to take note. In the EU and the UK, regulators are at the forefront of pushing mandates such as SEPA (Single European Payments Area), Open Banking Standard, and PSD-2.  All of these regulations will ensure that Banks are forced to unlock their customer data in a way that encourages consumer choice. The hope is that agile players can then use this data to exploit inefficiencies in the banks business model using technology. Services such as account aggregation, consumer loans, credit scoring services, personal financial management tools, and other financial advisory become easy to provide via Open APIs.

If incumbent Banks don’t respond, they will lose their monopoly on being their customers primary front end. As new players take over areas such as mortgage loans (an area where they’re much faster than banks in granting loans), Banks that cannot change their distribution and product models will be commodified. The challenges start with reworking inflexible core banking systems. These maintain customer demographics, balances, product information and other BORT (Book Of Record Transaction) data that store a range of loan, payment and risk information. These architectures will slowly need to transition from their current (largely) monolithic architectures to compose-able units. There are various strategies that Banks can follow to ‘modernize the core’ but adopting Big Data native mindset is. Banks will also seek to work with FinTechs to create islands of cooperation where they can learn from the new players.

Question #4 Drive Business Insight…

There are two primary areas where business insights need to be driven out of. The first is internal operations and the second is customer service.  This category encompasses a wide range of strategic choices that drive an operating model – product ideation, creation, distribution strategies across channels/geographies etc. Whatever be the right product and strategy focus, the ability to play in select areas of the value chain depends upon feedback received from day to day operations. Much like in a manufacturing company, this data needs to be harnessed, analyzed with a view to ultimately monetizing it.

Question #5 Business Transformation…

There is no question that FinTechs are able to take ideas from nothing to delivery in a matter of months. This is the key reason banks need to transform their business. This is critical in key areas such as sales, wealth management, and origination. There is surely a lot of confusion around how to drive such initiatives but no one questions the need for centralizing data assets.

In my mind, the first and most important reason to move to a unified strategy is to evolve standalone Digital capabilities into a coherent Platform. Different lines of business can use these capabilities to develop an ecosystem of applications that can be offered as a SaaS (Software as a Service). The end state of Digital Platforms is to operate business systems at massive scale in terms of customers, partners, and employees.

Question #6 Enhance Customer Service…

Customer Service is clearly an area of differentiation for nimbler players as compared to Banks. Banks are still largely dealing with ensuring that consistent views of customer accounts & balances can be maintained across channels. On the other hand, FinTechs have moved onto Chatbots and Robo-advisors all built around Big Data & AI. A Chatbot is a virtual assistant that helps clients perform simple transactions using mediums such as text or voice. They’re based on Natural Language Processing and Machine Learning and are being deployed in simple scenarios such as balance checks and other simpler customer service processes. However, as time goes by they will inevitably get more sophisticated and will eventually supplant human service for the vast majority of the service lifecycle.

Big Data Driven Disruption – The Robo-Advisor..(1/3)

Surely, areas such as automated customer service and investment management are still in early stages of maturity. However, they are unmistakably the next big trend in the financial industry and one that players should begin developing capabilities around. 


Increasingly, a Bank’s technology platform(s) centered around Big Data represents a significant competitive differentiator that can generate substantial revenues from existing customers and help acquire new ones. Given the strategic importance and revenue potential of this resource, the C-suite must integrate Big Data & AI into their strategic planning in 2018.


[1] McKinsey – “Remaking the Bank for an Ecosystem World” –

Global Payments Industry in 2018 – Breaking Through to New Horizons..

The Global Payments Industry in 2018…

In 2017, the Payments industry largely kept its promise of leading financial services – This evidenced in two important categories – consumer adoption and technology innovation. From a numbers standpoint, this has been accompanied by healthy growth in both the volumes and the count of payments. Mckinsey estimates that (as of 2017) the payments industry now makes up 34% of the global banking industry.[1] The coming year will find that three key forces – Digital technology, Consumer demands & Regulatory change – will continue to drive growth in the industry. With that in mind, let us consider the top industry themes and trends for 2018.

Image Credit –

Background and my predictions for 2017..

Payment Providers – How Big Data Analytics Provides New Opportunities in 2017

Trend #1 Digital Payments volumes continue to surge in 2018

Consumer payment volumes are now beginning to overtake business payments. McKinsey forecasts that from a volumes standpoint, by 2021, global payment volumes will surpass 2.2 trillion US dollars [1] –  a massive increase from just 450 billion US dollars in 2017.  The key drivers for this increase are the number of consumers rapidly coming online in countries such as China and India (with the latter alone contributing a base of 500 million internet subscribers) in 2017.  In India, the demonetization campaign conducted a year ago has resulted in a surge of digital transactions.

Banks have not been sitting still in the face of the instant payment paradigm. Across the globe, Banks have been nudging more consumers to begin using digital payments as a way of providing speed while managing both cost & risk – often at the expense of cheque payments. In the UK, Fast Payments Service which was launched in 2008, processing the five billionth payment in 2015.

The Faster Payments Scheme – UK

Currently, four types of payments can be processed through Faster Payments – immediate payments, forward dated payments, standing payment orders and Direct Corporate Access (single business payments with upto 250k pounds per transaction). Nearly every large and medium-sized Bank in the United Kingdon supports Faster Payments including the likes of Barclays and HSBC.

In response to all this rapid change, players across the payments spectrum and in adjacent verticals such as Retail & Telco will need to begin enhancing their mobile apps & in-store payments. Established card schemes such as Mastercard and Visa have begun rolling out API driven interfaces.

Trend #2 The Internet Leaders take increasing share of the consumer and corporate payments…

We are also witnessing a whole range of nimble competitors such as FinTechs and other financial institutions jockeying to sell both closed and open loop payments products to customers.

The likes of Apple, Amazon, Facebook, Alibaba, and Google are originating payments from not just their online portals and mobile apps but also from sensors, personal assistants (Echo, Siri etc) and voice-driven interfaces. These players will also drive capabilities into a range of payments related usecases – from Single View of Customer to Data Monetization to AML/Risk & Fraud detection.

Euromonitor contends that the leading mobile-centric nation in the world is China. In 2015, Chinese consumers made more purchases through mobile phones than using traditional computers. As of 2016, this number had increased to 2/3rd of all online purchases. Chinese players led by AliPay and WeChat are increasingly looking to replicate their domestic success abroad. This is being helped by global travel by Chinese consumers who are expected to take 225 million international trips in 2030, at a compound annual growth rate (CAGR) of 7.3% over 2016-2030. [3]

Trend #3 Regulators push for  Open Data Sharing & Innovation…

With Payment Systems Directive 2 (PSD2), the European Parliament has adopted the legal foundation for the creation of an EU-wide single payments area (SEPA).  While the goal of the PSD is to establish a set of modern, digital industry rules for all payment services in the European Union; it has significant ramifications for the financial services industry as it will surely current business models & foster new areas of competition. The key message from a regulatory standpoint is that consumer data can be opened up to other players in the payment value chain. This will lead to a clamor by players to own more of the customer’s data with a view to selling business services (e.g. accurate credit scoring, access to mortgage & other consumer loans and mutual funds etc) on that information.

Why the PSD2 will Spark Digital Innovation in European Banking and Payments….

Trend #4 Customer Experience drives volumes growth…

Customers are demanding smarter UX capabilities and intuitive cross-channel interfaces. Technology built around ensuring that payment providers can create a single view of a Cardholder across multiple accounts & channels of usage will result in more ecross-sellross sell/upsell and better customer segmentation.

For instance, the 360 degree view is a snapshot of the below types of data –

  • Customer’s Demographic information – Name, Address, Age etc
  • Length of the Customer-Enterprise relationship
  • Products and Services purchased overall
  • Preferred Channel & time of Contact
  • Marketing Campaigns the customer has responded to
  • Major Milestones in the Customers relationship
  • Ongoing activity – Open Orders, Deposits, Shipments, Customer Cases etc
  • Ongoing Customer Lifetime Value (CLV) Metrics and the Category of customer (Gold, Silver, Bronze etc)
  • Any Risk factors – Likelihood of Churn, Customer Mood Alert, Ongoing issues etc
  • Next Best Action for Customer

Payment providers have been sitting on petabytes of customer data and have only now begun waking up to the possibilities of monetizing this data. An area of increasing interest is to provide sophisticated analytics to merchants as a way of driving merchant rewards programs. Retailers, Airlines, and other online merchants need to understand what segments their customers fall into as well as what the best avenues are to market to each of them. E.g. Web apps, desktop or tablet etc. Using all of the Payment Data available to them, Payment providers can help Merchant Retailers understand their customers better as well as improve their loyalty programs.

Trend #5 Cross-Border Payments offer a lot of business growth but Compliance and Security remain huge challenges…

While Cross-border transactions still generate substantially higher margins than domestic. This blog has cataloged a range of Risk/Fraud and KYC/Compliance usecases in the cards industry. We increasingly find that banks across the spectrum are putting in strong capabilities around real-time fraud detection, risk management and AML (Anti Money Laundering).

Big Data Counters Payment Card Fraud (1/3)…

Advanced analytics and business reporting are being used to target money launderers and fraudsters. These projects, however, have large and complex outlays and needs advanced capabilities around Big Data & Artificial Intelligence.


The Payments industry is the most dynamic portion of Global Banking. Players will need a lot of creativity to connect the twin worlds of slow-moving finance and fast-moving technology. In 2018, the pressure will be on players to deliver higher rates of adoption, margins leveraging technology driven innovation.


[1] McKinsey – Payments Insights

[2] Faster Payments UK –

[3] Forbes – “Three Payment Trends that will change how we pay in 2018”

Why Enterprises should build Platforms and not just Standalone Applications…

                                                    Image Credit – Shutterstock 


The natural tendency in the world of Corporate IT is to create applications in response to business challenges. For instance, take any large Bank or Insurer or Manufacturer – you will find thousands of packaged applications that aim to solve a range of challenges from departmental level issues to enterprise-wide business problems. Over years these have given rise to application and infrastructure sprawl.

The application mindset creates little business value over the long run while creating massive technology headaches. For instance, the rationalization of these applications over time becomes a massive challenge in and of itself. At times, IT does not even understand how relevant some of these applications are to business users, who are even using them and the benefits derived. Over the last 15 years, Silicon Valley players such as Apple, Google, and Facebook et al have begun illustrating the power of building platforms that connect a range of users to the businesses that serve them. As the Network Effects connected to using these platforms have grown exponentially, so have the users.

What Corporate IT & business need to learn to do is to move to a Platform mindset.

The Platform Strategy…

Amazon is the perfect example of how to conceive and execute a platform strategy over a couple of decades. It began life as a retailer in 1994 and over time morphed into other complementary offerings such as Marketplace, AWS, Prime Video, Payments etc. These platforms have led to an ever-increasing panoply of services, higher revenues, promoted more directed consumer interactions and higher network effects. Each platform generates its own revenue stream and is a large standalone corporation in its own right. However, the sum of these platforms is higher than the sum of the individual products and this has led to Amazon becoming the most valuable company in the world (as of late 2017).

So what are the key business benefits and drivers of a platform oriented model?

Driver #1 Platforms enable you to build business ecosystems

Platforms enable enterprise business to orient their core capabilities better and to be able to deliver on those. Once that is done to a high degree of success, partners and other ecosystem players can plug in their capabilities.  The functionality that the platform provides is the ability to inter The challenge most times is that large companies always seem to play catchup with business models of nimbler players. When they do this, they often choose an application based approach which does not enable them to take a holistic view of their enterprise and the business ecosystems around them. In the Platform approach, IT departments move to more of a service model while delivering agile platforms and technology architectures for business lines to develop products around.

E.g. Post the PSD2 regulation, innovators in the European Banking system will become a prime example of platform led business ecosystems.

Why the PSD2 will Spark Digital Innovation in European Banking and Payments….

Driver #2 Platforms enable you to rethink and better the customer experience thus driving new revenue streams

The primary appeal of a platform based architecture is the ability to drive cross-sell and upsell opportunities. This increases not the number of products adopted by a given customer but also (and ultimately) the total revenue per customer.

The below blog post discusses how Payment Providers are increasingly using advanced analytics on their business platforms to generate not only increased topline/sales growth but also to defend against fraud and anti-money laundering (AML).

Payment Providers – How Big Data Analytics Provides New Opportunities in 2017

Driver #3 Platforms enable you to experiment with business models (e.g. Data Monetization)

The next progressive driver in leveraging both internal and external data is to use it to drive new revenue streams in existing lines of business.  This is also termed Data Monetization. Data Monetization is the organizational ability to turn data into cost savings & revenues in existing lines of business and to create new revenue streams. This requires fusing both internal and external data to create new analytics and visualization.

The Tao of Data Monetization in Banking and Insurance & Strategies to Achieve the Same…

Driver #4 Platforms destroy business process silos

One of the chief reasons that hold back an enterprise ability to innovate is the presence of both business and data silos. This is directly a result of an Application based approach. When underlying business processes & data sources are both fragmented, communication between business teams moves over to other internal & informal mechanisms such as email, chat and phone calls etc. This is an overall recipe for delayed business decisions which are ultimately ineffective as they depend more on intuition than are backed by data. The Platforms approach drives the organization towards unification and rationalization of both the data and the business process that creates it thus leading to a unified and consistent view of both across the business.

Why Data Silos Are Your Biggest Source of Technical Debt..

Driver #5 Platforms move you to become a Real-time Enterprise

Enterprises that are platform oriented does more strategic things right than wrong. They constantly experiment with creating new and existing business capabilities with a view to making them appealing to a rapidly changing clientele. They refine these using constant feedback loops and create platforms comprised of cutting-edge technology stacks that dominate the competitive landscape. The Real-Time enterprise demands that workers at many levels ranging from the line of business managers to executives have fresh, high quality and actionable information on which they can base complex yet high-quality business decisions.

The Three Habits of Highly Effective Real Time Enterprises…


A business and IT strategy built on platform approaches enable an organization to take on a much wider & richer variety of business challenges.  This enables an organization to achieve outcomes that were not really possible with the Application model.

My take on Gartner’s Top 10 Strategic Trends for 2018 & beyond..

My vision for the future state of the digital economy – I see a movie. I see a story of everybody connected with very low latency, very high speed, ultra-dense connectivity available. Today you’re at the start of something amazing… I see the freeing up, not just of productivity and money, but also positive energy which can bring a more equal world.” -Vittorio Colao, CEO, Vodafone, Speaking at the World Economic Forum – Davos, Jan 2015

As is customary for this time of the year, Gartner Research rolled out their “Top 10 Strategic Technology Trends for 2018” report a few weeks ago – Rather than exclusively cover the IT technology landscape as in past years, Gartner has also incorporated some of the themes from the 2016 US Presidential election, namely fake news and content.My goal for this blogpost is to provide my frank take on these trends to the reader. Also, as always – to examine the potential impact of their recommendations from an enterprise standpoint.

Previous Gartner Reviews…


My take on Gartner’s Top 10 Strategic Technology Trends for 2016


My take on Gartner’s Top 10 Strategic Technology Trends for 2017

The predictions themselves can be organized in five specific clusters – Web-scale giants, Cryptocurrencies, Fake News & AI,  IT job markets & IoT/Security.

Let us consider  –

Prediction Cluster #1 -Of  Web Scale Giants, Bots & E-Commerce… 

This year, Gartner makes two key predictions from the standpoint of the webscale giants, namely the FANG (Facebook, Amazon, Netflix and Google/Alphabet) companies plus Apple. These companies now dominate whatever business areas they choose to operate in largely due to the general lack of traditional enterprise competition to their technology-infused business models. They have not only gained market leadership status in their core markets but are also branching into creating blue ocean business models. Gartner’s prediction is that by 2020, these giants – which will largely remain unchallenged –  will need to innovate via self-disruption to stay nimble and competitive.

This prediction is hard to disagree with and is fairly obvious to someone who has followed their growth over the years. Virtually every major advance in consumer technology, mobile business models, datacenter architectures, product development methodologies over the last ten years has originated at these companies. The question is how much of this forecasted organic disruption will happen due to their cannibalizing existing product lines or creating entirely new markets e.g. self-driving tech, VR/AR etc.

The critical reason these companies have such a wide business moat is that they’ve incubated the Digital Native customer category. Their users are highly comfortable with technology and use services offered (such as Google’s range of products, Facebook services such as the classical social media platform, Instagram,  Uber, Netflix, Amazon Prime etc) almost hourly in their daily lives. As I have noted before, these customers expect a similar seamless & contextual experience while engaging with the more mundane and traditional enterprises such as Banks, Telcos, Retailers, Insurance companies. They expect primarily expect a digital channel experience. These companies then have a dual fold challenge – not only to provide the best user expereince but also to store all this data as well as harness it for real-time insights in a way that is connected with internal marketing & sales.

As many studies have shown, companies that constantly harness data about their customers, internal operations and perform speedy analytics on this data often outshine their competition. Does that seem a bombastic statement? Not when you consider that almost half of all online dollars spent in the United States in 2016 were spent on Amazon and almost all digital advertising revenue growth in 2016 was accounted by two biggies –Google and Facebook.

Which leads us to the second prediction, that – by 2021 early adopter brands that redesign their websites to support visual and voice search will increase digital commerce revenue by 30%.

This prediction is also bolstered by the likes of comScore which notes that voice & visual search have rapidly become the second and third leg of online search. Every serious mobile app now supports both these modes. Further Amazon with their Alexa assistant is bringing this capability to bear in diverse areas such as home automation.

Virtual reality (VR) and augmented reality (AR) are technologies that will completely change the way humans interact with one another and with intelligent systems that make up the Digital Mesh.  Uses of these technologies will include gamification (to improve customer engagement with products and services), other customer & employee-facing applications etc.

Prediction Cluster #2 – By 2022, Cryptocurrencies create $1B of value in the Banking market…

We have discussed the subject of bitcoin and blockchain to some degree of depth over the last year and this prediction will seem safe and obvious to many. The explosion of market value in Bitcoin and other alt-currencies also supports the coming of age of cryptocurrencies. However, Gartner pegging cryptocurrency led business value at just $1B by 2022 seems way on the lower end. Cryptocurrencies are not only widely accepted in various forms of banking. E.g. Payments, Consumer Banking loans, Mortgages etc but they are on the verge of gaining Central Bank support. I expect an explosion in their usage and institutionalization over the next two-three year horizon. Every enterprise needs an Altcurrency and Blockchain strategy.

Blockchain For the Enterprise: Key Considerations..

Prediction Cluster #3 – Fake News and Counterfeit Reality run amok…

Keeping in line with the dominant theme of the US Presidential election of 2016, fake news has become a huge challenge across multiple social media platforms. This news is being manufactured by skilled writers working for foreign and often hostile governments as well as AI driven bots. Gartner forecasts that by 2022, the majority of news consumed in developed economies will be fake. This is a staggering indictment of the degree of criminality in creating a counterfeit reality. Germany has led the way in passing legislation that goes after criminals who sow racial discord by planting fake news on internet platforms which have more than 2 million users. [1] The law applies to online service providers who operate platforms that enable sharing and dissemination of data. If offending material is not removed from social network platforms within 24 hours, fines of upto  €50 million can be levied by the regulator.

Enterprises need to guard similarly against fake news being shared with a view to harming their corporate or product image. Putting in place strong cyber defenses and operational risk systems will be key.

Prediction Cluster #4 – IT jobs in the Digital Age…

We have spoken about the need for IT staff to retool themselves as Digital transformation & bimodal IT projects increasingly take a seat in the corporate agenda. Accordingly, IT needs to increasingly understand and communicate in the language of the business. Gartner increasingly forecasts that IT staff will become versatilists across the key disciplines of Infrastructure, Operations, and Architecture.

What Lines Of Business Want From IT..

Gartner also forecasts that AI related jobs will experience healthy growth staring in 2020. Until then AI will result in widespread time and effort savings with AI augmenting existing workers with time and productivity savings.

Prediction Cluster #6 – IoT and Security… 

There are two key predictions included this year from an IoT standpoint. The first is that by 2022, half of IoT security budgets will be spent towards remediation and device safety recalls rather than in providing protection. Clearly, as threat vectors increase into an enterprise by their adoption of IoT, it is key to put appropriate governance mechanisms to ensure perimeter defense and to ensure appropriate patching & security policies are followed. You are only as secure as the weakest devices inside your organizational perimeter.

Secondly, In three years or less, Gartner predicts that IoT capabilities will be included in 95% of new electronic designs. This is not a surprise given the proliferation of embedded devices and the improvements in operating systems such as embedded Linux. However, the key gains will be made in platforms that harness and make this data actionable.

A Digital Reference Architecture for the Industrial Internet Of Things (IIoT)..

The Numbers…

This year’s Gartner’s predictions have largely underwhelmed in three broad areas.

Firstly, the broad coverage of all leading tech trends that were evident in the earlier years is clearly missing. For instance, sensor technology enabling autonomous vehicles such as LIDAR (Light Detection and Ranging) being pioneered by the likes of Alphabet and Tesla is conspicuous by its absence on the list. Elon Musk has been on record saying that self-driven transportation is just two or three years away from being introduced by the car makers.  Next, any mention of 5G wireless capabilities which enable a range of IoT workloads is expected to be a reality in 2020. This is another obvious miss by Gartner.

Secondly, some of the most evident areas of enterprise innovation such as FinTechs, InsurTechs are conspicuous by their absence.

Thirdly, Gartner has included quantitative data such as percentages and dates that with each trend that can leave one scratching their head. It is unclear what methodology and logic were employed in arriving at such exact numbers.


[1] “Germany’s Bold Gambit to Prevent Online Hate Crimes and Fake News Takes Effect” – Evelyn Douek, Lawfare