Demystifying Digital – Why Customer 360 is the Foundational Digital Capability – ..(1/3)

The first post in this three part series on Digital Foundations introduces the concept of Customer 360 or Single View of Customer (SVC). We will discuss the need for & the definition of the SVC as part of the first step in any Digital Transformation endeavor. We will also discuss specific benefits from both a business & operational state that are enabled by SVC. The second post in the series introduces the concept of a Customer Journey. The third & final post will focus on a technical design & architecture needed to achieve both these capabilities.
 
In an era of exploding organizational touch points, how many companies can truly claim that they know & understand their customers, their needs & evolving preferences deeply and from a realtime perspective?  
How many companies can claim to keep up as a customers product & service usage matures and keep them engaged by cross selling new offerings. How many can accurately predict future revenue from a customer based on their current understanding of their profile?
The answer is not at all encouraging.
Across industries like Banking, Insurance, Telecom & Manufacturing, the ability to get a unified view of the customer & their journey is at the heart of the the enterprise ability to promote relevant offerings & detect customer dissatisfaction. 
  • Currently most industry players are woeful at putting together this comprehensive Single View of their Customers (SVC). Due to operational silos, each department possess a siloed & limited view of the customer across multiple channels. These views are typically inconsistent, lack synchronization with other departments & miss a high amount of potential cross-sell and up-sell opportunities.
  • The Customer Journey problem has been an age old issue which has gotten exponentially more complicated over the last five years as the staggering rise of mobile technology and the Internet of Things (IoT) have vastly increased the number of enterprise touch points that customers are exposed to in terms of being able to discover & purchase new products/services. In an OmniChannel world, an increasing number of transactions are being conducted online. In verticals like Retail and Banking, the number of online transactions approaches an average of 40%. Adding to the problem, more and more consumers are posting product reviews and feedback online. Companies thus need to react in realtime to piece together the source of consumer dissatisfaction.
Another large component of customer outreach are Marketing analytics & the ability to run effective campaigns to recruit customers.

The most common questions that a lot of enterprises fail to answer accurately are –

  1. Is the Customer happy with their overall relationship experience?
  2. What mode of contact do they prefer? And at what time? Can Customers be better targeted at these channels at those preferred times?
  3. What is the overall Customer Lifetime Value (CLV) or how much profit we are able to generate from this customer over their total lifetime?
  4. By understanding CLV across populations, can we leverage that to increase spend on marketing & sales for products that are resulting in higher customer value?
  5. How do we increase cross sell and up-sell of products & services?
  6. Does this customer fall into a certain natural segment and if so, how can we acquire most customers like them?
  7. Can different channels (Online, Mobile, IVR & POS) be synchronized ? Can Customers begin a transaction in one channel and complete it in any of the others without having to resubmit their data?

The first element in Digital is the Customer Centricity & it must naturally follow that a 360 degree view is a huge aspect of that.

Customer360

                                       Illustration – Customer 360 view & its benefits

So what information is specifically contained in a Customer 360 –

The 360 degree view is a snapshot of the below types of data –

  • Customer’s Demographic information – Name, Address, Age etc
  • Length of the Customer-Enterprise relationship
  • Products and Services purchased overall
  • Preferred Channel & time of Contact
  • Marketing Campaigns the customer has responded to
  • Major Milestones in the Customers relationship
  • Ongoing activity – Open Orders, Deposits, Shipments, Customer Cases etc
  • Ongoing Customer Lifetime Value (CLV) Metrics and the Category of customer (Gold, Silver, Bronze etc)
  • Any Risk factors – Likelihood of Churn, Customer Mood Alert, Ongoing issues etc
  • Next Best Action for Customer

How can Big Data technology help?

Leveraging the ingestion and predictive capabilities of a Big Data based platform, banks can provide a user experience that rivals Facebook, Twitter or Google and provide a full picture of customer across all touch points.

Big Data enhances the Customer 360 capability in the following ways  –  

  1. Obtaining a realtime Single View of the Customer (typically a customer across multiple channels, product silos & geographies) across years of account history 
  2. Customer Segmentation by helping businesses understand customer segments down to the individual level as well as at a segment level
  3. Performing Customer sentiment analysis by combining internal organizational data, clickstream data, sentiment analysis with structured sales history to provide a clear view into consumer behavior.
  4. Product Recommendation engines which provide compelling personal product recommendations by mining realtime consumer sentiment, product affinity information with historical data.
  5. Market Basket Analysis, observing consumer purchase history and enriching this data with social media, web activity, and community sentiment regarding past purchase and future buying trends.

Customer 360 can help improve the following operational metrics of a Retailer or a Bank or a Telecom immensely.

  1. Cost to Income ratio; Customers Acquired per FTE; Sales and service FTE’s (as percentage of total FTE’s), New Accounts Per Sales FTE etc
  2.  Sales conversion rates across channels, Decreased customer attrition rates etc.
  3. Improved Net promotor scores (NPS), referral based sales etc

Customer 360 is thus basic digital capability every organization needs to offer their customers, partners & internal stakeholders. This implies a re-architecture of both data management and business processes automation.

The next post will discuss the second critical component of Digital Transformation – the Customer Journey.

Embedding A Culture of Business Analytics into the Enterprise DNA..

IT driven business transformation is always bound to fail” – Amber Storey, Sr Manager, Ernst & Young

The value of Big Data driven Analytics is no longer in question both from a customer as well as an enterprise standpoint. Lack of investment in an analytic strategy has the potential to impact shareholder value negatively.  Business Boards and CXOs are now concerned about their overall levels and maturity of investments in terms of business value – i.e increasing sales, driving down business & IT costs & helping create new business models. It is thus an increasingly accurate argument that smart applications & ecosystems built around them will increasingly dictate enterprise success.

Such examples among forward looking organizations abound across industries. These range from realtime analytics in manufacturing using IoT data streams across the supply chain, the use of natural language processing to drive patient care decisions in healthcare, more accurate insurance fraud detection & driving Digital interactions in Retail Banking etc to quote a few. 

However , most global organizations currently adopt a fairly tactical approach to ensuring the delivery of of traditional business intelligence (BI) and predictive analytics to their application platforms.  This departmental is quite suboptimal in ways as scaleable data driven decisions & culture not only empower decision-makers with up to date and realtime information but also help them develop long term insights into how globally diversified business operations are performing.  Scale is the key word here due to rapidly changing customer trends, partner, supply chain realities & regulatory mandates.

Scale implies speed of learning,  business agility across the organization in terms of having globally diversified operations turn on a dime thus ensuring that the business feels empowered.

A quick introduction to Business (Descriptive & Predictive) Analytics –

Business intelligence (BI) is a traditional & well established analytical domain that essentially takes a retrospective look at business data in systems of record. The goal for BI is to primarily look for macro or aggregate business trends across different aspects or dimensions such as time, product lines, business unites & operating geographies.

BI is primarily concerned with “What happened and what trends exist in the business based on historical data?“. The typical use cases for BI include budgeting, business forecasts, reporting & key performance indicators (KPI).

On the other hand, Predictive Analytics (a subset of Data Science) augments & builds on the BI paradigm by adding a “What could happen” dimension to the data in terms of –

  • being able to probabilistically predict different business scenarios across thousands of variables
  • suggesting specific business actions based on the above outcomes

Predictive Analytics does not intend to nor will it replace the BI domain but only adds significant business capabilities that lead to overall business success. It is not uncommon to find real world business projects leveraging both these analytical approaches.

Creating an industrial approach to analytics – 

Strategic business projects typically begin imbibing a BI/Predictive Analytics based approach as an afterthought to the other aspects of system architecture and buildout. This dated approach then ensures that analytics becomes external to and eventually operating in a reactive mode in the operation of business system.

Having said that, one does need to recognize that an industrial approach to analytics is a complex endeavor that depends on how an organization tackles the convergence of the below approaches –

  1. Organizational Structure
  2. New Age Technology 
  3. A Platforms Mindset
  4. Culture

Creating_An_Analytic_Culture

        Illustration – Embedding A Culture of Business Analytics into the Enterprise DNA..

Lets discuss them briefly – 

Organizational Structure – The historical approach has been to primarily staff analytics teams as a standalone division often reporting to a CIO. This team has responsibility for both the business intelligence as well as some silo of a data strategy. Such a piecemeal approach to predictive analytics ensures that business & application teams adopt a “throw it over the wall” mentality over time.

So what needs to be done? 

In the Digital Age, enterprises should look to centralize both data management as well as the governance of analytics as core business capabilities. I suggest a hybrid organizational structure where a Center of Excellence (COE) is created which reports to the office of the Chief Data Officer (CDO) as well as individual business analytic leaders within the lines of business themselves.

 This should be done to ensure that three specific areas are adequately tackled using a centralized approach- 

  • Investing in creating a data & analytics roadmap by creating a center of excellence (COE)
  • Setting appropriate business milestones with “lines of business” value drivers built into a robust ROI model
  • Managing Risk across the enterprise with detailed scenario planning

New Age Technology –

The onset of Digital Architectures in enterprise businesses implies the ability to drive continuous online interactions with global consumers/customers/clients or patients. The goal is not just provide engaging visualization but also to personalize services clients care about across multiple modes of interaction. Mobile applications first begun forcing the need for enterprise to begin supporting multiple channels of interaction with their consumers. We have seen how how exploding data generation across the global economy has become a clear & present business & IT phenomenon. Data volumes are rapidly expanding across industries. However, while the production of data itself that has increased but it is also driving the need for organizations to derive business value from it. This calls for the collection & curation of data from dynamic,  and highly distributed sources such as consumer transactions, B2B interactions, machines such as ATM’s & geo location devices, click streams, social media feeds, server & application log files and multimedia content such as videos etc – using Big Data.

Cloud Computing is the ideal platform to provide the business with self service as well as rapid provisioning of business analytics. Every new application designed needs to be cloud native from the get go.

The onset of Digital Architectures in enterprise businesses implies the ability to drive continuous online interactions with global consumers/customers/clients or patients. The goal is not just provide engaging Visualization but also to personalize services clients care about across multiple modes of interaction. Mobile applications first begun forcing the need for enterprise to begin supporting multiple channels of interaction with their consumers. For example Banking now requires an ability to engage consumers in a seamless experience across an average of four to five channels – Mobile, eBanking, Call Center, Kiosk etc.

A Platforms Mindset – 

As opposed to building standalone or one-off business applications, a Platform Mindset is a more holistic approach capable of producing higher revenues. Platforms abound in the webscale world at shops like Apple, Facebook & Google etc. Applications are constructed like lego blocks  and they reuse customer & interaction data to drive cross sell and up sell among different product lines. The key components here are to ensure that one starts off with products with high customer attachment & retention. While increasing brand value, it is key to ensure that customers & partners can also collaborate in the improvements in the various applications hosted on top of the platform.

Culture – Business value fueled by analytics is only possible if the entire organization operates on an agile basis in order to collaborate across the value chain. Cross functional teams across new product development, customer acquisition & retention, IT Ops, legal & compliance must collaborate in short work cycles to close the traditional business & IT innovation gap. Methodologies like DevOps who’s chief goal is to close the long-standing gap between the engineers who develop and test IT capability and the organizations that are responsible for deploying and maintaining IT operations – must be adopted. Using traditional app dev methodologies, it can take months to design, test and deploy software. No business today has that much time—especially in the age of IT consumerization and end users accustomed to smart phone apps that are updated daily. The focus now is on rapidly developing business applications to stay ahead of competitors that can better harness Big Data’s amazing business capabilities.

Summary- 

Enterprise wide business analytic approaches designed around the four key prongs  (Structure, Culture, Technology & Platforms)   will create immense operational efficiency, better business models, increased relevance and ultimately drive revenues. These will separate the visionaries, leaders from the laggards in the years to come.

What Lines Of Business Want From IT..

Relationship

                    Illustration: Business- IT Relationship (Image src – Pat.it)

Previous posts in this blog have discussed the fact that technological capabilities now make or break business models. It is critical for IT to operate in a manner that maximizes their efficiency while managing costs & ultimately delivering the right outcomes for the organization.

It is clear and apparent to me that the relationship lines of business (LOBs) have with their IT teams – typically central & shared – is completely broken at a majority of large organizations. Each side cannot seem to view either the perspective or the passions of the other. This dangerous dysfunction usually leads to multiple complaints from the business. Examples of which include –

  • IT is perceived to be glacially slow in providing infrastructure needed to launch new business initiatives or to amend existing ones. This leads to the phenomenon of ‘Shadow IT’ where business applications are  run on public clouds bypassing internal IT
  • Something seems to be lost in translation while conveying requirements to different teams within IT
  • IT is too focused on technological capabilities – Virtualization, Middleware, Cloud, Containers, Hadoop et al without much emphasis on business value drivers

So what are the top asks that Business has for their IT groups? I wager that there are five important focus areas –

  1. Transact in the language of the business –Most would agree that there has been too much of a focus on the technology itself – how it works,  what the infrastructure requirements are to host applications – cloud or on-prem, data engines to ingest and process it etc etc . The focus needs to be on customer needs that drive business value for an organization’s customers, partners, regulators & employees. Technology at it’s core is just an engine and does not exist in a vacuum. The most vibrant enterprises understand this ground reality and always ensure that business needs drive IT and not the other way around. It is thus highly important for IT leadership to understand the nuances of the business to ensure that their roadmaps (long and medium term) are being driven with business & competitive outcomes in mind. Examples of such goals are a common organization wide taxonomy across products, customers, logistics, supply chains & business domains. The shared emphasis on both business & IT should be on goals like increased profitability per customer, enhanced segmentation of both micro and macro customer populations with the ultimate goal of increasing customer lifetime value (CLV).
  2. Bi-Modal or “2 Speed” IT et al need to be business approach centric – Digital business models that are driving agile web-scale companies offer enhanced customer experiences built on product innovation and data driven business models. They are also encroaching into the domain of established industry players in verticals like financial services, retail, entertainment, telecommunications, transportation and insurance  by offering contextual & trendy products tailored to individual client profiles. Their savvy use of segmentation data  and realtime predictive analytics enables the delivery of bundles of tailored products across multiple delivery channels (web, mobile, point of sale, Internet, etc.). The enterprise approach has been to adopt a model known as Bi-Modal IT championed by Gartner. This model envisages two different IT camps – one focused on traditional applications and the other focused on innovation. Whatever be the moniker for this approach – LOBs need to be involved as stakeholders from the get-go & throughout the process of selecting technology choices that have downstream business ramifications. One of the approaches that is working well is increased cross pollination across both teams, collapsing artificial organizational barriers by adopting DevOps & ensuring that business has a slim IT component to rapidly be able to fill in gaps in IT’s business knowledge or capability.
  3. Self Service Across the board of IT Capabilities – Shadow IT (where business goes around the IT team) is not just an issue with infrastructure software but is slowly creeping up to business intelligence and advanced analytics apps. The delays associated with provisioning legacy data silos combined with using tools that are neither intuitive nor able to scale to deal with the increasing data deluge are making timely business analysis almost impossible to perform.  Insights delivered too late are not very valuable. Thus, LOBs are beginning  to move to a predominantly online SaaS (Software As A Service) model across a range of business intelligence applications. Reports, visual views of internal & external datasets are directly served to internal consumers based on data uploaded into a cloud based BI provider. These reports and views are then directly delivered to end users. IT needs to enable this capability and make it part of their range of offerings to the business.
  4. Help the Business think Analytically  – Business Process Automation (BPM) and Data Driven decision making are proven approaches used at data-driven organizations. When combined with Data and Business Analytics, this tends to be a killer combination. Organizations that are data & data metric driven are able to define key business processes that provide native support for key performance indicators (KPIs) that are critical and basic to their functioning. Applications developed by IT need to be designed in such a way that these KPIs can be communicate and broadcast across the organization constantly. Indeed a high percentage of organizations now have senior executive in place as the champion for BPM, Business Rules and Big Data driven analytics. These applications are also mobile native so that they can be provided access through a variety of mobile platforms for field based employees & back into the corporate firewall.
  5. No “Us vs Them” mentality – it is all “Us”  –  None of the above are only possible if the entire organization operates on an agile basis in order to collaborate across the value chain. Cross functional teams across new product development, customer acquisition & retention, IT Ops, legal & compliance must collaborate in short work cycles to close the traditional business & IT innovation gap.  One of chief goals of agile methodologies is to close the long-standing gap between the engineers who develop and test IT capability and business requirements for such capabilities.  Using traditional app dev methodologies, it can take months to design, test and deploy software – which is simply unsustainable. 

Business & IT need to collaborate. Period. –

The most vibrant enterprises that have implemented web-scale practices not only offer “IT/Business As A Service” but also have instituted strong cultures of symbiotic relationships between customers (both current & prospective), employees , partners and developers etc.

No business today has much time to innovation—especially in the age of IT consumerization where end users accustomed to smart phone apps that are often updated daily. The focus now is on rapidly developing business applications to stay ahead of competitors that can better harness technology’s amazing business capabilities.

How Robo-Advisors work..(2/3)

Millennials want “finance at their fingertips”..they want to be able to email and text the financial advisors and talk to them on a real-time basis,” – Greg Fleming, Ex-Morgan Stanley
The first post in this series on Robo-advisors touched on the fact that Wealth Management has been an area largely untouched by automation as far as the front office is concerned. Automated investment vehicles have largely begun changing that trend and they helping create a variety of business models in the industry. This three part series explored the automated “Robo-advisor” movement in the first post.This second post will focus on the overall business model & main functions of a Robo-advisor.
Introduction
FinTechs led by Wealthfront and Betterment have pioneered the somewhat revolutionary concept of Robo-advisors. To define the term – a Robo-advisor is an algorithm based automated investment advisor that can provide a range of Wealth Management services tailored to a variety of life situations.
Robo-advisors offer completely automated financial planning services. We have seen how the engine of the Wealth Management business is new customer acquisition. The industry is focused on acquiring the millennial or post millennial HNWI (High Net Worth Investor) generation. The technology friendliness of this group ensures that are the primary target market for automated investment advice. Not just the millenials, anyone who is comfortable with using technology and wants lower cost services can benefit from automated investment planning. However,  leaders in the space such as – Wealthfront & Betterment – have disclosed that their average investor age is around 35 years. [1]
Robo-advisors provide algorithm-based portfolio management methods around investment creation, provide automatic portfolio rebalancing and value added services like tax-loss harvesting as we will see. The chief investment vehicle of choice seems to be low-cost, passive exchange-traded funds (ETFs).

What are the main WM business models

Currently there are a few different business models that are being adopted by WM firms.

  1. Full service online Robo-advisor that is a 100% automated without any human element
  2. Hybrid Robo-advisor model being pioneered by firms like Vanguard & Charles Schwab
  3. Pure online advisor that is primarily human in nature

What do Robo-advisors typically do?

The Robo-advisor can be optionally augmented & supervised by a human adviser. At the moment, owing to the popularity of Robo-advisors among the younger high networth investors (HNWI), a range of established players like Vanguard, Charles Schwab as well as a number of FinTech start-ups have developed these automated online investment tools or have acquired FinTech’s in this space.e.g Blackrock. The Robo-advisor is typically offered as  a free service (below a certain minumum) and typically invests in low cost ETFs.  built using digital techniques – such as data science & Big Data.

Robo_Process

                                  Illustration: Essential functions of a Robo-advisor

The major business areas & client offerings in the Wealth & Asset Management space have been covered in the first post in this series at http://www.vamsitalkstech.com/?p=2329

Automated advisors only cover a subset of all of the above at the moment. The major usecases are as below –

  1. Determine individual Client profiles & preferences—e.g. For a given client profile- determine financial goals, expectations of investment return, diversification etc
  2. Identify appropriate financial products that can be offered either as pre-packaged portfolios or custom investments based on the client profile identified in the first step
  3. Establish correct Investment Mix for the client’s profile – these can included but are not ,limited to equities, bonds, ETFs & other securities in the firm’s portfolios . For instance, placing  tax-inefficient assets in retirement accounts like IRA’s as well as  tax efficient municipal bonds in taxable accounts etc.
  4. Using a algorithmic approach, choose the appropriate securities for each client account
  5. Continuously monitor the portfolio & transactions within it to tune performance , lower transaction costs, tax impacts etc based on how the markets are doing. Also ensure that a client’s preferences are being incorporated so that appropriate diversification and risk mitigation is being performed
  6. Provide value added services like Tax loss harvesting to ensure that the client is taking tax benefits into account as they rebalance portfolios or accrue dividends.
  7. Finally ,ensure the best user experience by handling a whole range of financial services – trading, account administration, loans,bill pay, cash transfers, tax reporting, statements in one intuitive user interface.

000-graph

                             Illustration: Betterment user interface. Source – Joe Jansen

To illustrate these concepts in action, leaders like Wealthfront & Betterment are increasingly adding features where  highly relevant, data-driven advice is being provided based on existing data as well as aggregated data from other providers. Wealthfront now provides recommendations on diversification, taxes and fees that are personalized not only to the specific investments in client’s account, but also tailored to their specific financial profile and risk tolerance. For instance, is enough cash being set aside in the emergency fund ? Is a customer holding too much stock in your employer? [1]

The final post will look at a technology & architectural approach to building out a Robo-advisor. We will also discuss best practices from a WM & industry standpoint in the context of Robo-advisors.

References:

  1. Wealthfront Blog – “Introducing the new Dashboard”

Global Retail Banking Needs a Digital Makeover

If you don’t like change, you will like irrelevance even less.” -General Eric Shinseki, Former  US Secretary of Veterans Affairs

This blog has spent time documenting the ongoing digital disruption across the industry especially financial services. Is there proof that creative destruction is taking a hold in Banking? The answer is a clear & unequivocal “Yes”. Clearly, Retail Banking is undergoing a massive makeover. This is being driven by many factors – changing consumer preferences, the advent of technology, automation of business processes & finally competition from not just the traditional players but also the Fintechs. The first casualty of this change is the good old Bank Branch. This post looks at the business background of Retail Banking across the world & will try to explain my view on what is causing this shift in how Banks and consumers perceive financial services.

This blog post will be one of a series of five standalone posts on Retail Bank transformation. The intention for the first post is to discuss industry dynamics, the current state of competition and will briefly introduce the forces causing a change in the status quo. The second post will categorize FinTechs across the banking landscape with key examples of how they disinter-mediate established players. The remaining posts will examine each of the other forces (Customer  in more detail along with specific and granular advice to retail banks on how to incorporate innovation into their existing technology, processes and organizational culture.

Introduction – 

Retail Banking is perhaps one of the most familiar and regular services that everyday citizens use in the course of their lives. Money is a commodity we touch every day in our lives when we bank, shop, pay bills, borrow etc. Retail lines of banking typically include personal accounts, credit cards, mortgages and auto loans. 

For large financial conglomerates that have operations spanning Commercial Banking, Capital Markets, Wealth & Asset Management etc, retail operations have always represented an invaluable source of both stability as well as balance sheet strength. The sheer size & economic exposure of retail operations ensures that it is not only staid yet stable but also somewhat insulated from economic shocks. This is borne out by the policies of respective national central banks & treasury departments. Indeed one of main the reasons regulators have bailed out banks in the past is due to the perception that Main Street & the common citizen’s banking assets becoming a casualty of increased risk taking  by traders in the capital markets divisions. This scenario famously played out during the Great Depression in the late 1920s and was a major factor in causing widespread economic contagion. A stock market crash quickly cascaded into a nation-wide economic depression. 

Thus, retail banking is crucial to not just to the owning corporation but also to diverse stakeholders in the world economy – deposit holders, the regulators led by the US Federal Reserve (in the US) & a host of other actors.  

The State of Global Retail Banking – 

In the financial crisis of 2008, retail banks not only held their own but also assumed a bigger share of revenues as the recovery got underway in the following years. According to a survey by Boston Consulting Group (BCG), retail banking activities accounted for 55 percent of the revenues generated across a global cohort of 140 banks, up from 45 percent in 2006.[1] 

However, the report also contends that retail revenues since 2008 have been slowly falling as investors have begin shifting their savings to deposits as a reaction to high profile financial scandals thus putting pressure on margins. Higher savings rates have helped offset this somewhat & retail banks ended up maintaining better cost to income (CIR) ratios than did other areas of banking.Retail banks also performed better on a key metric return on assets (ROA). The below graphic from the BCG captures this metric. In the Americas region, the average ROA was 162 percent higher than the average group ROA in 2008. From 2001 through 2006, it was 51 percent higher. Global banking revenues stood at $ 1.59 trillion in 2015 – a figure that is expected to hold relatively steady across the globe [2]

It is also important to note that global performance of retail banks across the five major regions: the Americas, Europe, the Middle East, Asia, and Australia has generally varied based on a multitude of factors. And even within regions, banking performance has varied widely.[2]

Retail Banking - BCG

                                      Illustration 1 – Retail Banking is profitable and stable 

As stable as this sector seems, it is also be roiled by four main forces that are causing every major player to rethink their business strategy. Left unaddressed, these changes will cause huge and negative impacts on competitive viability, profitability & also impact all important growth over the next five years. 

What is the proof that retail banking is beginning to change? The below graphic from CNN [1] says it all –

BofA_Branches_CNN

Bank of America has 23% fewer branches and 37% fewer employees than in 2009.  That downward trend across both metrics is expected to continue as online transactions from (deposits to checks to online loans) grown by a staggering 94%. The bank is expected to cut more positions in reflection of a shrinking headcount and branch footprint[1].

Pressure from the FinTechs:

The Financial Services and the Insurance industry are facing an unprecedented amount of change driven by factors like changing client preferences and the emergence of new technology—the Internet, mobility, social media, etc. These changes are immensely profound, especially with the arrival of “FinTech”—technology-driven applications that are upending long-standing business models across all sectors from retail banking to wealth management & capital markets. Further, members of a major new segment, Millennials, increasingly use mobile devices, demand more contextual services and expect a seamless unified banking experience—something akin to what they  experience on web properties like Facebook, Amazon, Uber, Google or Yahoo, etc. They do so by expanding their wallet share of client revenues by offering contextual products tailored to individual client profiles. Their savvy use of segmentation data and predictive analytics enables the delivery of bundles of tailored products across multiple delivery channels (web, mobile, call center banking, point of sale, ATM/kiosk etc.).

Retail Banking must trend Digital to respond – 

The definition of Digital is somewhat nebulous, I would like to define the key areas where it’s impact and capabilities will need to be felt for this gradual transformation to occur.

A true Digital Bank needs to –

  • Offer a seamless customer experience much like the one provided by the likes of Facebook & Amazon i.e highly interactive & intelligent applications that can detect a single customer’s journey across multiple channels
  • offer data driven interactive services and products that can detect customer preferences on the fly, match them with existing history and provide value added services. Services that not only provide a better experience but also foster a longer term customer relationship
  • to be able to help the business prototype, test, refine and rapidly develop new business capabilities
  • Above all, treat Digital as a Constant Capability and not as an ‘off the shelf’ product or a one off way of doing things

The five areas that established banks need to change across are depicted below..

RetailBank_Value_Drivers

  1. Convert branches to be advisory & relationship focused instead of centers for transactions – As the number of millennials keeps growing, the actual traffic to branches will only continue to decline.  Branches still have an area of strength in being intimate customer touch points. The branch of the future can be redesigned to have more self service features along with relationship focused advisory personnel instead of purely being staffed by tellers and managers. They need to be reimagined as Digital Centers, not unlike an Apple store, with highly interactive touch screens and personnel focused on building business through high margin products.
  2. Adopt a FinTech like mindset – FinTechs (or new Age financial industry startups) offer enhanced customer experiences built on product innovation and agile business models. They do so by expanding their wallet share of client revenues by offering contextual products tailored to individual client profiles. Their savvy use of segmentation data and predictive analytics enables the delivery of bundles of tailored products across multiple delivery channels (web, mobile, Point Of Sale, Internet, etc.). Like banks, these technologies support multiple modes of payments at scale, but they aren’t bound by the same regulatory and compliance regulations as are banks, who operate under a mandate that they must demonstrate that they understand their risk profiles. The best retail banks will not only seek to learn from, but sometimes partner with, emerging fintech players to integrate new digital solutions and deliver exceptional customer experience. To cooperate and take advantage of fintechs, banks will require new partnering capabilities. To heighten their understanding of customers’ needs and to deliver products and services that customers truly value, banks will need new capabilities in data management and analytics.
  3. Understand your customer – Banks need to move to a predominantly online model, providing consumers with highly interactive, engaging and contextual experiences that span multiple channels—branch banking, eBanking, POS, ATM, etc. Further goals are increased profitability per customer for both micro and macro customer populations with the ultimate goal of increasing customer lifetime value (CLV).
  4. Business Process improvement – Drive Automation across lines of business  – Financial services are fertile ground for business process automation, since most banks across their various lines of business are simply a collection of core and differentiated processes. Examples of these processes are consumer banking (with processes including on boarding customers, collecting deposits, conducting business via multiple channels, and compliance with regulatory mandates such as KYC and AML); investment banking (including straight-through-processing, trading platforms, prime brokerage, and compliance with regulation); payment services; and wealth management (including modeling model portfolio positions and providing complete transparency across the end-to-end life cycle). The key takeaway is that driving automation can result not just in better business visibility and accountability on behalf of various actors. It can also drive revenue and contribute significantly to the bottom line. Automation enables enterprise business and IT users to document, simulate, manage, automate and monitor business processes and policies. It is designed to empower business and IT users to collaborate more effectively, so business applications can be changed more easily and quickly.
  5. Agile Culture – All of the above are only possible if the entire organization operates on an agile basis in order to collaborate across the value chain. Cross functional teams across new product development, customer acquisition & retention, IT Ops, legal & compliance must collaborate in short work cycles to close the traditional business & IT innovation gap.  One of DevOps’s chief goals is to close the long-standing gap between the engineers who develop and test IT capability and the organizations that are responsible for deploying and maintaining IT operations. Using traditional app dev methodologies, it can take months to design, test and deploy software. No business today has that much time—especially in the age of IT consumerization and end users accustomed to smart phone apps that are updated daily. The focus now is on rapidly developing business applications to stay ahead of competitors that can better harness Big Data’s amazing business capabilities.

How can all of this be quantified? –

The results of BCG’s sixth annual Global Retail-Banking Excellence benchmarking illustrate the value drivers. Forward looking banks are working on some of the above aspects are able to reduce cycle times for core processes thus improving productivity. The leaders in the survey are also reallocating resources from the mid and office to customer facing roles.[3]

Again, according to the BCG, digital reinvention comes with huge benefits to both the top and bottom-lines. Their annual survey across the global retail banking sector estimates an average reduction in operating expenses from 15% to 25%, increases in pretax profit by 20% to 30% and an average increase in margins before tax from 5% to 10%. [3] These numbers are highly impressive at the scale that large banks operate.

The question thus is, can the vast majority of Banks change before it’s too late? Can they find the right model of execution in the Digital Age before their roles are either diminished or dis-intermediated by competition?

We will dive deep into the FinTech’s in the next post in the series.

References

[1] CNN Money – Bank of America has 23% fewer branches than 2009

[2]BCG Research- Winning Strategies Revisited for Retail Banking

[3] BCG Research- Global Capital Markets 2016: The Value Migration

Capital Markets Pivots to Big Data in 2016

Previous posts in this blog have discussed how Capital markets firms must create new business models and offer superior client relationships based on their vast data assets. Firms that can infuse a data driven culture in both existing & new areas of operation will enjoy superior returns and raise the bar for the rest of the industry in 2016 & beyond. 

Capital Markets are the face of the financial industry to the general public and generate a large percent of the GDP for the world economy. Despite all the negative press they have garnered since the financial crisis of 2008, capital markets perform an important social function in that they contribute heavily to economic growth and are the primary vehicle for household savings. Firms in this space allow corporations to raise capital using the underwriting process. However, it is not just corporations that benefit from such money raising activity – municipal, local and national governments do the same as well. Just that the overall mechanism differs – while business enterprises issue both equity and bonds, governments typically issue bonds. According to the Boston Consulting Group (BCG), the industry will grow to annual revenues of $661 billion in 2016 from $593 billion in 2015 – a healthy 12% increase. On the buy side, the asset base (AuM – Assets under Management) is expected to reach around $100 trillion by 2020 up from $74 trillion in 2014.[1]

Within large banks, the Capital Markets group and the Investment Banking Group perform very different functions.  Capital Markets (CM) is the face of the bank to the street from a trading perspective.  The CM group engineers custom derivative trades that hedge exposure for their clients (typically Hedge Funds, Mutual Funds, Corporations, Governments and high net worth individuals and Trusts) as well as for their own treasury group.  They may also do proprietary trading on the banks behalf for a profit – although it is this type of trading that Volcker Rule is seeking to eliminate.

If a Bank uses dark liquidity pools (DLP) they funnel their Brokerage trades through the CM group to avoid the fees associated with executing an exchange trade on the street.  Such activities can also be used to hide exchange based trading activity from the Street.  In the past, Banks used to make their substantial revenues by profiting from their proprietary trading or by collecting fees for executing trades on behalf of their treasury group or other clients.

Banking and within it, capital markets continues to generate insane amounts of data. These producers range from news providers to electronic trading participants to stock exchanges which are increasingly looking to monetize data. And it is not just the banks, regulatory authorities like the FINRA in the US are processing peak volumes of 40-75 billion market events a day http://www.vamsitalkstech.com/?p=1157 [2]. In addition to data volumes, Capital Markets has always  possessed a variety challenge as well. They have tons of structured data around traditional banking data, market data, reference data & other economic data. You can then factor in semi-structured data around corporate filings,news,retailer data & other gauges of economic activity. An additional challenge now is the creation of data from social media, multimedia etc – firms are presented with significant technology challenges and business opportunities.

Within larger financial supermarkets, the capital markets group typically leads the way in  being forward looking in terms of adopting cutting edge technology and high tech spends.  Most of the compute intensive problems are generated out of either this group or the enterprise risk group. These groups own the exchange facing order management systems, the trade booking systems, the pricing libraries for the products the bank trades as well as the tactical systems that are used to manage their market and credit risks, customer profitability, compliance and collateral systems.  They typically hold about one quarter of a Banks total IT budget. Capital Markets thus has the largest number of use cases for risk and compliance.

Players across value chain on the buy side, the sell side, the intermediaries (stock exchanges & the custodians) & technology firms such as market data providers are all increasingly looking at leveraging these new data sets that can help unlock the value of data for business purposes beyond operational efficiency.

So what are the  different categories of applications that are clearly leveraging Big Data in production deployments.

CapMkts_UseCases

                      Illustration – How are Capital Markets leveraging Big Data In 2016

I have catalogued the major ones below based on my work with the majors in the spectrum over the last year.

  1. Client Profitability Analysis or Customer 360 view:  With the passing of the Volcker Rule, the large firms are now moving over to a model based on flow based trading rather than relying on prop trading. Thus it is critical for capital market firms to better understand their clients (be they institutional or otherwise) from a 360-degree perspective so they can be marketed to as a single entity across different channels—a key to optimizing profits with cross selling in an increasingly competitive landscape. The 360 view encompasses defensive areas like Risk & Compliance but also the ability to get a single view of profitability by customer across all of their trading desks, the Investment Bank and Commercial Lending.
  2. Regulatory Reporting –  Dodd Frank/Volcker Rule Reporting: Banks have begun to leverage data lakes to capture every trade intraday and end of day across it’s lifecycle. They are then validating that no proprietary trading is occurring on on the banks behalf.  
  3. CCAR & DFast Reporting: Big Data can substantially improve the quality of  raw data collected across multiple silos. This improves the understanding of a Bank’s stress test numbers.
  4. Timely and accurate risk management: Running Historical, stat VaR (Value at Risk) or both to run the business and to compare with the enterprise risk VaR numbers.
  5. Timely and accurate liquidity management:  Look at the tiered collateral and their liquidity profiles on an intraday basis to manage the unit’s liquidity.  They also need to look at credit and market stress scenarios and be able to look at the liquidity impact of those scenarios.
  6. Timely and accurate intraday Credit Risk Management:  Understanding when  & if  deal breaches a tenor bucketed limit before they book it.  For FX trading this means that you have about 9 milliseconds  to determine if you can do the trade.  This is a great place to use in memory technology like Spark/Storm and a Hadoop based platform. These usecases are key in increasing the capital that can be invested in the business.  To do this they need to convince upper management that they are managing their risks very tightly.
  7. Timely and accurate intraday Market Risk Management:  Leveraging Big Data to market risk computations ensures that Banks have a real time idea of any market limit breaches for any of the tenor bucketed market limits.
  8. Reducing Market Data costs: Market Data providers like Bloomberg, Thomson Reuters and other smaller agencies typically charge a fee each time data is accessed.  With a large firm, both the front office and Risk access this data on an ad-hoc fairly uncontrolled basis. A popular way to save on cost is to  negotiate the rights to access the data once and read it many times.  The key is that you need a place to put it & that is the Data Lake.
  9. Trade Strategy Development & Backtesting: Big Data is being leveraged to constantly backtest trading strategies and algorithms on large volumes of historical and real time data. The ability to scale up computations as well as to incorporate real time streams is key to
  10. Sentiment Based Trading: Today, large scale trading groups and desks within them have begun monitoring economic, political news and social media data to identify arbitrage opportunities. For instance, looking for correlations between news in the middle east and using that to gauge the price of crude oil in the futures space.  Another example is using weather patterns to gauge demand for electricity in specific regional & local markets with a view to commodities trading. The realtime nature of these sources is information gold. Big Data provides the ability to bring all these sources into one central location and use the gleaned intelligence to drive various downstream activities in trading & private banking.
  11. Market & Trade Surveillance:Surveillance is an umbrella term that usually refers to a wide array of trading practices that serve to distort securities prices thus enabling market manipulators to illicitly profit at the expense of other participants, by creating information asymmetry. Market surveillance is generally out by Exchanges and Self Regulating Organizations (SRO) like the FINRA in the US – all of which have dedicated surveillance departments set up for this purpose. However, capital markets players on the buy and sell side also need to conduct extensive trade surveillance to report up internally. Pursuant to this goal, the exchanges & the SRO’s monitor transaction data including orders and executed trades & perform deep analysis to look for any kind of abuse and fraud. Big Data shines at this usecase as discussed here – http://www.vamsitalkstech.com/?p=1212
  12. Buy Side (e.g. Wealth Management) – A huge list of usecases I have catalogued here – https://dzone.com/articles/the-state-of-global-wealth-management-part-2-big-d 
  13. AML Compliance –  Covered in various blogs and webinars.
    http://www.vamsitalkstech.com/?s=AML
    https://www.boozallen.com/insights/2016/04/webinar-anti-money-laudering – 

The Final Word

A few tactical recommendations to industry CIOs:

  • Firstly, capital markets players should look to create centralized trade repositories for Operations, Traders and Risk Management.  This would allow consolidation of systems and a reduction in costs by providing a single platform to replace operations systems, compliance systems and desk centric risk systems.  This would eliminate numerous redundant data & application silos, simplify operations, reduce redundant quant work, improve and understanding of risk.
  • Secondly, it is important to put in place a model to create sources of funding for discretionary projects that can leverage Big Data.
  • Third, Capital Markets groups typically have to fund their portion of AML, Dodd Frank, Volcker Rule, Trade Compliance, Enterprise Market Risk and Traded Credit Risk projects.  These are all mandatory spends.  After this they typically get to tackle discretionary business projects. Eg- fund their liquidity risk, trade booking and tactical risk initiatives.  These defensive efforts always get the short end of the stick and are not to be neglected while planning out new initiatives.
  • Finally, an area in which a lot of current players are lacking is the ability to associate clients using a Lightweight Entity Identifier (LEI). Using a Big Data platform to assign logical and physical entity ID’s to every human and business the bank interacts can have salubrious benefits. Big Data can ensure that firms can do this without having to redo all of their customer on-boarding systems. This is key to achieving customer 360 views, AML and FATCA compliance as well as accurate credit risk reporting.

It is no longer enough for CIOs in this space to think of tactical Big Data projects, they must be thinking around creating platforms and ecosystems around those platforms to be able to do a variety of pathbreaking activities that generate a much higher rate of return.

References

[1] “The State of Capital Markets in 2016” – BCG Perspectives

[2] FINRA Technology –
http://technology.finra.org/

Big Data Driven Disruption – The Robo-Advisor..(1/3)

Wealth Management is the highest growth businesses for any medium to large financial institution. It also is the highest customer touch segment of banking and is fostered on long term (read extremely lucrative advisory) relationships. This three part series explores the automated “Robo-advisor” movement in the first post. We will cover the business background and some definitions . The second post will focus on the overall business model & main functions of a Robo-advisor. The final post will look at a technology & architectural approach to building out a Robo-advisor. We will also discuss best practices from a WM & industry standpoint in the context of Robo-advisors.

roboadvisor

(Image Credit – Forbes)

The term ‘Wealth Management‘ broadly refers to an aggregation of financial services that are typically bespoke and offered to highly affluent clients.  These include financial advisory,  personal investment management, financial advisory, and planning disciplines directly for the benefit of high-net-worth (HNWI) clients.  This term can refer to a wide range of possible functions and business models.

A wealth manager is a specialized financial advisor who helps a client construct an entire investment portfolio and advises on how to prepare for present and future financial needs. The investment portion of wealth management normally entails both asset allocation of a whole portfolio as well as the selection of individual investments. The planning function of wealth management often incorporates tax planning around the investment portfolio as well as estate planning for individuals as well as family estates.

The ability to sign up wealthy individuals & families; then retaining them over the years by offer those engaging, bespoke & contextual services will largely provide growth in the Wealth Management industry in 2016 and beyond.

However,  WM as an industry sector has lagged other areas within banking from a technology & digitization standpoint. Multiple business forces ranging from increased regulatory & compliance demands, digital demands & expectations from younger, technology savvy customers and new Age FinTechs have led to firms slowly begin a makeover process. Let us examine these trends in more detail. 

Business Trends Driving the need for Robo/Automated Investment Advisors –

These trends  are a combination of industry reality as well as changing preferences on behalf of the HNWI clientele –

  1. Growth in the Wealth Management business largely depends on the ability to sign up new clients. Previously WM shops would not be interested in signinup up clients with less than a certain value of investable assets (typical threshold being $ 1 million). However the need to on-ramp these folks onto a long term relationship means being able to offer lower cost automated business models that better fit their mindsets
  2. The mentality of younger clientele has also evolved over the years. These clients are technologically savvy, they largely have a DIY (Do It Yourself) mindset and their digital needs are largely being missed by the wealth management community. This rising segment demands digital services that are highly automated & 24/7 in nature without needing to pay the premium charged by a human advisor
  3. Regulatory, cost pressures are rising which are leading to commodification of services
  4. Innovative automation and usage techniques of data assets among new entrants aka the FinTechs are leading to the rise of automated advisory services thus challenging incumbent firms. At traditional brokerage firms like  Morgan Stanley, Bank of America Corp. and Wells Fargo & Co. about 46,000 human advisers were employed as of 2016. The challenge for these incumbent firms will be to develop such automated investing tools as well as offer more self-service channels for customers [2]
  5. A need to offer aggregated & holistic financial services tailored to the behavioral needs of the HNWI investors on an individual basis

So where is the biggest trend in this disruption? It is undoubtedly, the Robo-advisor.

Introducing the Automated Advisor (affectionately called the Robo-advisor) –

FinTechs led by Wealthfront and Betterment have pioneered the somewhat revolutionary concept of Robo-advisors. To define the term – a Robo-advisor is an algorithm based automated investment advisor that can provide a range of Wealth Management services described below. The Robo-advisor can be optionally augmented & supervised by a human adviser. At the moment, owing to the popularity of Robo-advisors among the younger high networth investors (HNWI), a range of established players like Vanguard, Charles Schwab as well as a number of FinTech start-ups have developed these automated online investment tools or have acquired FinTech’s in this space.e.g Blackrock. The Robo-advisor is built using digital techniques – such as data science & Big Data – as we will explore in the next post.

What service models can Robo-advisors satisfy –

Full service Wealth Management firms broadly provide services in the following core areas which Robo-advisors can slowly begin supplementing –

  1. Investment Advisory – Helping a client construct an investment portfolio that helps her/him prepare for life changes based on their respective risk appetites & time horizons. The financial instruments invested in range from the mundane – equities, bonds etc to the arcane – hedging derivatives etc
  2. Retirement Planning – Retirement planning is a obvious function of a client’s personal financial journey & one that lends itself to automation. From a HNWI standpoint, there is a need to provide complex retirement services while balancing taxes, income needs & estate prevention etc. Robo-advisors are able to bring in market trends and movements of securities to ensure that client’s retirement holdings are not  skewed toward particular sectors of the marke.
  3. Estate Planning Services – A key function of wealth management is to help clients pass on their assets via inheritance. The Robo-advisor can assist a human wealth managers helps construct wills that leverage trusts and suggest suitable forms of insurance etc to help facilitate a smooth process of estate planning
  4. Tax Planning – Robo-advisors can help clients manage their wealth in such a manner that tax impacts are reduced from a taxation (e.g IRS in the US) perspective. As the pools of wealth increase, even small rates of taxation can have a magnified impact either way. The ability to achieve the right mix of investments from a tax perspective is a key capability and one that can be automated to a high degree
  5. Insurance Management – A Robo-advisor can help suggest and manage  the kinds of insurance purchased by their HNWI clients so that the appropriate hedging services could be put in place based on the client’s specific investment mix & exposures
  6. Institutional Investments– Institutional Robo-advisors can provide investment services to investors like pension funds, hedge funds etc while automating them a variety of backoffice functions

Currently most Robo-advisors limit themselves to providing the first function only i.e portfolio management (i.e. allocating investments among asset classes) without addressing issues such as estate and retirement planning and cash-flow management, which are also the domain of financial planning.[1]

Expect this to change as the technology rapidly matures in the years to come with advances in cognitive computing that will enable . At one of the earliest Robo-advisors, Betterment,  as of early 2016 – more than half of their $3.3 billion of assets under management comes from people with more than $100,000 at the firm. Another early starter, Wealthfront estimated more than a third of its almost $3 billion in assets in accounts requiring at least $100,000. Schwab, one of the first established investment firms to produce an automated product, attracted $5.3 billion to its offering in its first nine months.[2]

Robo

Robo-advisory business models

Currently there are a few different business models that are being adopted by firms.

  1. Full service online Robo-advisor that is a 100% automated without any human element
  2. Hybrid Robo-advisor model being pioneered by firms like Vanguard & Charles Schwab
  3. Pure online advisor that is primarily human in nature

Conclusion –

As one can see clearly, automated investing methods are still in early stages of maturity. However, they are unmistakably the next big trend in the WM industry and one that players should begin developing capabilities around. According to AT.Kearney, by 2020, Roboadvisors will manage around $2.2 trillion in global HNWI assets.[2]

The next post in this three part series will focus on the pivotal role of Big Data in creating a Robo-advisor. We will discuss system requirements & propose a reference architecture. 

References

  1. Wikipedia – https://en.wikipedia.org/wiki/Robo-advisor
  2. Bloomberg – “The Rich are already using Roboadvisors and that scares the banks..”

Across Industries, Big Data Is Now the Engine of Digital Innovation..

The data fabric is the next middleware.” –Todd Papaioannou, CTO at Splunk

Enterprises across the globe are confronting the need to create a Digital Strategy. While the term itself may seen intimidating to some, it essentially represents  an agile culture built on customer centricity & responsiveness. The only way to attain Digital success is to understand your customers at a micro level while making strategic decisions on your offerings to the market. Big Data has become the catalyst in this massive disruption as it can help business in any vertical solve their need to understand their customers better. It aids this by providing foundational  platform for amazing products.

We have seen how how exploding data generation across the global has become a clear & present business & IT phenomenon. Data volumes are rapidly expanding across industries. However, while the production of data itself that has increased but it is also driving the need for organizations to derive business value from it. This calls for the collection & curation of data from dynamic,  and highly distributed sources such as consumer transactions, B2B interactions, machines such as ATM’s & geo location devices, click streams, social media feeds, server & application log files and multimedia content such as videos etc. It needs to be noted that data volumes here consist of multi-varied formats, differing schemas, transport protocols and velocities.

Internet of Things (IoT) has become an entire phenomenon to itself. It is truly a horizontal vertical (no pun intended) as the proliferation of applications of sensors is causing rapid change in system & application architectures. The system of IoT is burgeoning from the initial sensors, digital devices, mechanical automatons to cars, process monitoring systems, browsers, television, traffic cameras etc etc.

Big Data is thus crossing the innovation chasm. A vast majority of early adopter projects are finding business success with a strong gain in ROI (Return On Investment). The skills gap is beginning to slowly decrease with Hadoop ecosystem becoming a skill that every modern application developer needs to have. Increasingly customers are leading the way by deploying Big Data in new and previously uncharted areas like cybersecurity leading to massive cross vertical interest.

DT_Vectors

The five elements in Digital Transformation, irrespective of the business vertical you operate in, are –

  1. Customer Centricity
  2. Realtime multichannel analytics
  3. Operational improvements – Risk, Fraud & Compliance
  4. Ability of the business to visualize data
  5. Marketing & Campaign optimization

The first element in Digital is the Customer centricity.

Big Data drives this in myriad ways  –  

  1. Obtaining a realtime Single View of an entity (typically a customer across multiple channels, product silos & geographies)
  2. Customer Segmentation by helping businesses understand their customers down to the individual level as well as at a segment level
  3. Customer sentiment analysis by combining internal organizational data, clickstream data, sentiment analysis with structured sales history to provide a clear view into consumer behavior.
  4. Product Recommendation engines which provide compelling personal product recommendations by mining realtime consumer sentiment, product affinity information with historical data.
  5. Market Basket Analysis, observing consumer purchase history and enriching this data with social media, web activity, and community sentiment regarding past purchase and future buying trends.

Realtime Multichannel Analytics is the second piece of a Digital Strategy.

Mobile applications first begun forcing the need for enterprise to begin supporting multiple channels of interaction with their consumers. For example Banking now requires an ability to engage consumers in a seamless experience across an average of four to five channels – Mobile, eBanking, Call Center, Kiosk etc. The healthcare industry stores patient data across multiple silos – ADT (Admit Discharge Transfer) systems, medication systems, CRM systems etc but all of this must be exposed across different mediums of access. Data Lakes provide an ability to visualize all of the patients data in one place thus improving outcomes. Every customer facing application needs to be both multi-channel as well as one that supports  a unified 360 degree customer view across all these engagement points. Applications developed in 2016 and beyond must take a 360 degree based approach to ensuring a continuous client experience across the spectrum of endpoints and the platforms that span them from a Data Visualization standpoint. Every serious business needs to provide a unified view of a customer across tens of product lines and geographies. Big Data not only provides the core foundational elements for a realtime view of the moving parts of the business but also enables businesses to listen to their customers.

A strategic approach to improving Risk, Fraud & Compliance analytics  can add massive value and competitive differentiation in three distinct categories as shown below.

  1. Exponentially improve existing business processes. e.. Risk data aggregation and measurement, HIPAA/SOX/Manufacturing compliance, fraud detection
  2. Help create new business models and go to market strategies – by monetizing multiple data sources – both internal and external
  3. Vastly improve regulatory compliance by generating fresher and more accurate insights across silos of proprietary data

The onset of Digital Architectures in enterprise businesses implies the ability to drive continuous online interactions with global consumers/customers/clients or patients. The goal is not just provide engaging Visualization but also to personalize services clients care about across multiple modes of interaction. Mobile applications first begun forcing the need for enterprise to begin supporting multiple channels of interaction with their consumers. For example Banking now requires an ability to engage consumers in a seamless experience across an average of four to five channels – Mobile, eBanking, Call Center, Kiosk etc. Healthcare is a close second where caregivers expect patient, medication & disease data at their fingertips with a few finger swipes on an iPad app.

The ability of outbound Marketing campaigns to reach engaged customers in a proactive manner using the right channel has been a big gap in their effectiveness. The old school strategy of blasting out direct mailers and emails does not work anymore both from a cost as well as a customer engagement standpoint. Nowadays, campaigns for exciting new products & promotions need to be built on the rich customer intelligence assets that Big Data enables you to build. Examples of these capabilities are replete in sectors like Retail where offering a positive purchase experience in terms of personalized offers, price comparisons, social network based sharing of experiences et al drive higher customer engagement & loyalty.

The Final Word

My goal for this post was to communicate a business revelation that I have had in past year. While the semantics of business processes, the usecases & the data sources, elements, formats may vary from industry to industry ( e.g. Banking to Healthcare to Manufacturing to Telecom) – the approaches as well as the benefits from leveraging a data & analytics driven business model essentially remain the same. These capabilities are beginning to separate the winners from the rest of the pack.

Open Enterprise Hadoop – as secure as Fort Knox

Previous posts in this blog have discussed customers leveraging Open Source, Big Data and Hadoop related technologies for a range of use cases across industry verticals. We have seen how a Hadoop-powered “Data Lake” can not only provide a solid foundation for a new generation of applications that provide analytics and insight, but can also increase the number of access points to an organization’s data. As diverse types of both external and internal enterprise data are ingested into a central repository, the inherent security risks must be understood and addressed by a host of actors in the architecture. Security is thus highly essential for organizations that store and process sensitive data in the Hadoop ecosystem. Many organizations must adhere to strict corporate security polices as well as rigorous industry guidelines. So how does open source Hadoop stack upto demanding standards such as PCI-DSS? 

We have from time to time, noted the ongoing digital transformation across industry verticals. For instance, banking organizations are building digital platforms that aim to engage customers, partners and employees. Retailers & Banks now recognize that the key to win the customer of the day is to offer a seamless experience across multiple channels of engagement. Healthcare providers want to offer their stakeholders – patients, doctors,nurses, suppliers etc with multiple avenues to access contextual data and services; the IoT (Internet of Things) domain is abuzz with the possibilities of Connected Car technology.

The aim of this blogpost is to disabuse those notions which float around from time to time where a Hadoop led 100% open source ecosystem is cast as being somehow insecure or unable to fit well into a corporate security model. It is only to dispel such notions about open source, the Open Source Alliance has noted well that – “Open source enables anyone to examine software for security flaws. The continuous and broad peer-review enabled by publicly available source code improves security through the identification and elimination of defects that might otherwise be missed. Gartner for example, recommends the open source Apache Web server as a more secure alternative to closed source Internet Information servers. The availability of source code also facilitates in-depth security reviews and audits by government customers.” [2]

It is a well understood fact that data is the most important asset a business possess and one that nefarious actors are usually after. Let us consider the retail industry- cardholder data such as card numbers or PAN (Primary Account Numbers) & other authentication data is much sought after by the criminal population.

The consequences of a data breach are myriad & severe and can include –

  • Revenue losses
  • Reputational losses
  • Regulatory sanction and fines etc

Previous blogposts have chronicled cybersecurity in some depth. Please refer to this post as a starting point for a somewhat exhaustive view of cybersecurity. This awareness has led to an increased adoption in risk based security frameworks. E.g ISO 27001, the US National Institute of Standards and Technology (NIST) Cybersecurity Framework and SANS Critical Controls. These frameworks offer a common vocabulary, a set of guidelines that enable enterprises to  identify and prioritize threats, quickly detect and mitigate risks and understand security gaps.

In the realm of payment card data – regulators,payment networks & issuer banks themselves recognize this and have enacted compliance standard – the PCI DSS (Personal Cardholder Information – Data Security Standards). PCI is currently in its third generation incarnation or v3.0 which was introduced over the course of 2014. It is the most important standard for a host of actors –  merchants, processors, payment service providers or really any entity that stores or uses payment card data. It is also important to note that the core process of compliance all applications and systems in a merchant or a payment service provider.

The  PCI standards council recommends the following 12 components for PCI-DSS as depicted in the below table.

PCI_DSS_12_requirements_grande

Illustration: PCI Data Security Standard – high level overview (source: shopify.com)

While PCI covers a whole range of areas that touch payment data such as POS terminals, payment card readers, in store networks etc – data security is front & center.

It is to be noted though that according to the Data Security Standards body who oversee the creation & guidance around the PCI , a technology vendor or product cannot be declared as being cannot “PCI Compliant.”

Thus, the standard has wide implications on two different dimensions –

1. The technology itself as it is incorporated at a merchant as well as

2. The organizational culture around information security policies.

My experience in working at both Hortonworks & Red Hat has shown me that open source is certified at hundreds of enterprise customers running demanding workloads in verticals such as financial services, retail, insurance, telecommunications & healthcare. The other important point to note is that these customers are all PCI, HIPPA and SOX compliant across the board.

It is a total misconception that off the shelf and proprietary point solutions are needed to provide broad coverage across the above pillars. Open enterprise Hadoop offers comprehensive and well rounded implementations across all of the five areas and what more it is 100% open source. 

Let us examine how security in Hadoop works.

The Security Model for Open Enterprise Hadoop – 

The Hadoop community has thus adopted both a top down as well as bottom up approach when looking at security as well as examining at all potential access patterns and across all components of the platform.

Hadoop and Big Data security needs to be considered across the below two prongs – 

  1. What do the individual projects themselves need to support to guarantee that business architectures built using them are highly robust from a security standpoint? 
  2. What are the essential pillars of security that the platform which makes up every enterprise cluster needs to support? 

Let us consider the first. The Apache Hadoop project contains 25+ technologies in the realm of data ingestion, processing & consumption. While anything beyond a cursory look is out of scope here, an exhaustive list of the security hooks provided into each of the major projects are covered here [1].

For instance, Apache Ranger manages fine-grained access control through a rich user interface that ensures consistent policy administration across Hadoop data access components. Security administrators have the flexibility to define security policies for a database, table and column, or a file, and can administer permissions for specific LDAP-based groups or individual users. Rules based on dynamic conditions such as time or geolocation, can also be added to an existing policy rule. The Ranger authorization model is highly pluggable and can be easily extended to any data source using a service-based definition.[1]

Administrators can use Ranger to define a centralized security policy for the following Hadoop components and the list is constantly enhanced:

  • HDFS
  • YARN
  • Hive
  • HBase
  • Storm
  • Knox
  • Solr
  • Kafka

Ranger works with standard authorization APIs in each Hadoop component, and is able to enforce centrally administered policies for any method used to access the data lake.[1]

 Now the second & more important question from an overall platform perspective. 

There are five essential pillars from a security standpoint that address critical needs that security administrators place on data residing in a data lake. If any of these pillars is vulnerable from an implementation standpoint, it ends up creating risk built into organization’s Big Data environment. Any Big Data security strategy must address all five pillars, with a consistent implementation approach to ensure their effectiveness.

Security_Pillars

                             Illustration: The Essential Components of Data Security

  1. Authentication – does the user possess appropriate credentials? This is implemented via the Kerberos authentication protocol & allied concepts such as Principals, Realms & KDC’s (Key Distribution Centers).
  2. Authorization – what resources is the user allowed to access based on business need & credentials?  Implemented in each Hadoop project & integrated with an organizations LDAP/AD/.
  3. Perimeter Security – prevents unauthorized outside access to the cluster. Implemented via Apache Knox Gateway which extends the reach of Hadoop services to users outside of a Hadoop cluster. Knox also simplifies Hadoop security for users who access the cluster data and execute jobs.
  4. Centralized Auditing  – implemented via Apache Atlas and it’s integration with Apache Ranger.
  5. Security Administration – deals with the central setup & control all security information using a central console.  uses Apache Ranger to provide centralized security administration and management. The Ranger Administration Portal is the central interface for security administration. Users can create and update policies, which are then stored in a policy database.

ranger_centralized_admin

                                           Illustration: Centralized Security Administration

It is also to be noted that as Hadoop adoption grows at an incremental pace, workloads that harness data for complex business analytics and decision-making may need more robust data-centric protection (namely data masking, encryption, tokenization). Thus, in addition to the above Hadoop projects as Apache Ranger, enterprises can essentially take an augmentative approach.  Partner solutions that offer data centric protection for Hadoop data such as Dataguise DgSecure for Hadoop which clearly complement an enterprise ready Hadoop distribution (such as those from the open source leader Hortonworks) are definitely worth a close look.

Summary

While implementing Big Data architectures in support of business needs, security administrators should look to address coverage for components across each of the above areas as they design the infrastructure. A rigorous & bottom-up approach to data security makes it possible to enforce and manage security across the stack through a central point of administration, which will likely prevent any potential security gaps and inconsistencies. This approach is especially important for newer technology like Hadoop where exciting new projects & data processing engines are always being incubated at a rapid clip. After all, the data lake is all about building a robust & highly secure platform on which data engines – Storm,Spark etc and processing frameworks like Mapreduce function to create business magic. 

 References – 

[1] Hortonworks Data Security Guide
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Security_Guide/

[2] Open Source Alliance of America
http://opensourceforamerica.org/learn-more/benefits-of-open-source-software/

Why Software Defined Infrastructure & why now..(1/2)

The ongoing digital transformation in key verticals like financial services, manufacturing, healthcare and telco has incumbent enterprises fending off a host of new market entrants. Enterprise IT’s best answer is to increase the pace of innovation as a way of driving increased differentiation in business processes. Though data analytics & automation remain the lynchpin of this approach – software defined infrastructure (SDI) built on the notions of cloud computing has emerged as the main infrastructure differentiator & that for a host of reasons which we will discuss in this two part blog.

Software Defined Infrastructure (SDI) is essentially an idea that brings together  advances in a host of complementary areas spanning both infrastructure software, data as well as development environments. It supports a new way of building business applications. The core idea in SDI is that massively scalable applications (in support of diverse customer needs) describe their behavior characteristics (via configuration & APIs) to underlying datacenter infrastructure which simply obeys those commands in an automated fashion while abstracting away the underlying complexities.

SDI as an architectural pattern was originally made popular by the web scale giants – the so-called FANG companies of tech — Facebook , Amazon , Netflix and Alphabet (the erstwhile Google) but has begun making it’s way into the enterprise world gradually.

Common Business IT Challenges prior to SDI – 
  1. Cost of hardware infrastructure is typically growing at a high percentage every year as compared to  growth in the total  IT budget. Cost pressures are driving an overall re look at the different tiers across the IT landscape.
  2. Infrastructure is not completely under the control of the IT-Application development teams as yet.  Business realities that dictate rapid app development to meet changing business requirements
  3. Even for small, departmental level applications, still needed to deploy expensive proprietary stacks which are not only cost and deployment footprint prohibitive but also take weeks to spin up in terms of provisioning cycles.
  4. Big box proprietary solutions leading to a hard look at Open Source technologies which are lean and easy to use with lightweight deployment footprint.Apps need to dictate footprint; not vendor provided containers.
  5. Concerns with acquiring developers who are tooled on cutting edge development frameworks & methodologies. You have zero developer mindshare with Big Box technologies.

Key characteristics of an SDI

  1. Applications built on a SDI can detect business events in realtime and respond dynamically by allocating additional resources in three key areas – compute, storage & network – based on the type of workloads being run.
  2. Using an SDI, application developers can seamlessly deploy apps while accessing higher level programming abstractions that allow for the rapid creation of business services (web, application, messaging, SOA/ Microservices tiers), user interfaces and a whole host of application elements.
  3. From a management standpoint, business application workloads are dynamically and automatically assigned to the available infrastructure (spanning public & private cloud resources) on the basis of the application requirements, required SLA in a way that provides continuous optimization across the life cycle of technology.
  4. The SDI itself optimizes the entire application deployment by both externally provisioned APIs & internal interfaces between the five essential pieces – Application, Compute, Storage, Network & Management.

The SDI automates the technology lifecycle –

Consider the typical tasks needed to create and deploy enterprise applications. This list includes but is not limited to –

  • onboarding hardware infrastructure,
  • setting up complicated network connectivity to firewalls, routers, switches etc,
  • making the hardware stack available for consumption by applications,
  • figure out storage requirements and provision those
  • guarantee multi-tenancy
  • application development
  • deployment,
  • monitoring
  • updates, failover & rollbacks
  • patching
  • security
  • compliance checking etc.
The promise of SDI is to automate all of this from a business, technology, developer & IT administrator standpoint.
 SDI Reference Architecture – 
 The SDI encompasses SDC (Software Defined Compute) , SDS (Software Defined Storage), SDN (Software Defined Networking), Software Defined Applications and Cloud Management Platforms (CMP) into one logical construct as can be seen from the below picture.
FS_SDDC

                      Illustration: The different tiers of Software Defined Infrastructure

The core of the software defined approach are APIs.  APIs control the lifecycle of resources (request, approval, provisioning,orchestration & billing) as well as the applications deployed on them. The SDI implies commodity hardware (x86) & a cloud based approach to architecting the datacenter.

The ten fundamental technology tenets of the SDI –

1. Highly elastic – scale up or scale down the gamut of infrastructure (compute – VM/Baremetal/Containers, storage – SAN/NAS/DAS, network – switches/routers/Firewalls etc) in near real time

2. Highly Automated – Given the scale & multi-tenancy requirements, automation at all levels of the stack (development, deployment, monitoring and maintenance)

3. Low Cost – Oddly enough, the SDI operates at a lower CapEx and OpEx compared to the traditional datacenter due to reliance on open source technology & high degree of automation. Further workload consolidation only helps increase hardware utilization.

4. Standardization –  The SDI enforces standardization and homogenization of deployment runtimes, application stacks and development methodologies based on lines of business requirements. This solves a significant IT challenge that has hobbled innovation at large financial institutions.

5. Microservice based applications –  Applications developed for a SDI enabled infrastructure are developed as small, nimble processes that communicate via APIs and over infrastructure like messaging & service mediation components (e.g Apache Kafka & Camel). This offers huge operational and development advantages over legacy applications. While one does not expect Core Banking applications to move over to a microservice model anytime soon, customer facing applications that need responsive digital UIs will need definitely consider such approaches.

6. ‘Kind-of-Cloud’ Agnostic –  The SDI does not enforce the concept of private cloud, or rather it encompasses a range of deployment options – public, private and hybrid.

7. DevOps friendly –  The SDI enforces not just standardization and homogenization of deployment runtimes, application stacks and development methodologies but also enables a culture of continuous collaboration among developers, operations teams and business stakeholders i.e cross departmental innovation. The SDI is a natural container for workloads that are experimental in nature and can be updated/rolled-back/rolled forward incrementally based on changing business requirements. The SDI enables rapid deployment capabilities across the stack leading to faster time to market of business capabilities.

8. Data, Data & Data –  The heart of any successful technology implementation is Data. This includes customer data, transaction data, reference data, risk data, compliance data etc etc. The SDI provides a variety of tools that enable applications to process data in a batch, interactive, low latency manner depending on what the business requirements are.

9. Security –  The SDI shall provide robust perimeter defense as well as application level security with a strong focus on a Defense In Depth strategy.

10. Governance –  The SDI enforces strong governance requirements for capabilities ranging from ITSM requirements – workload orchestration, business policy enabled deployment, autosizing of workloads to change management, provisioning, billing, chargeback & application deployments.

The next & final blog in this series will look at current & specific technology choices – as of 2016 – in building out an SDI.