Why Retailers Fail to Adopt Advanced Data Analytics

For years now, executives have been told that advanced analytics can provide better answers to almost every business question. Yet in retail, at least, surprisingly few companies have taken full advantage of the opportunity.

Even as Walmart, Amazon, and a few other leading retailers operate at the leading edge of the analytics frontier, making many important decisions based on an ever-growing supply of real-time and historical data, most of their competitors still use very basic tools that are far better able at tracking where they’ve been than where they should be going.

This is already having real consequences for the industry. During the pandemic, McKinsey estimates, the 25 top-performing retailers — most of whom are digital leaders — were 83% more profitable than laggards and took home more than 90% of the sector’s gains in market capitalization. Although you cannot prove a negative, it does seem likely that laggards are leaving a lot of money on the table. In grocery retail, for instance, McKinsey estimates that implementing advanced analytics would add 2% to grocers’ earnings — a potential windfall for a tough, low-margin business.

This won’t come as news to most people. The executives of even the slowest-moving company must be aware at some level that they are missing out. Yet despite understanding the advantages that analytics have given their competitors, and despite knowing that academics and consultants keep developing more and more advanced analytics solutions, most laggards seem unlikely to catch up with the leaders anytime soon.

Why are so many companies having such a hard time making this leap forward? What is holding them back?

Six Sticking Points

To find out, we interviewed a diverse set of global retail executives (senior executives of retailers, distributors, consulting firms, and analytics providers active in the Americas, Europe, and Asia). The 24 business leaders we interviewed, whose companies varied in their analytics maturity, cited six factors as the primary sticking points:

Culture. Most companies suffer from risk aversion and have no clear goals for an analytics project. “Is data important?” one interviewee told us. “Everyone says yes. If you ask why, many don’t know.” Others look down on analytics, considering their work to be more art than science. One department store executive recalled a buyer asking, “Will an algorithm tell me what dresses to buy? I know what dresses to buy.”

Organization. Many noted that their companies struggle to maintain a balance between centralization and decentralization, both of which are essential: centralization for efficiency, economies of scale, and consistency; and decentralization for flexibility, a greater ability to adapt to local environments, and receptivity to a wider range of ideas.

People. The larger problem, however, respondents suggested to us, is perhaps this: The analytics function is often run by people who do not really understand the business. As one executive wrote, “When during an interaction with the problem owners someone from analytics gives the impression (s)he does not understand the business at all, something happens that I’d like to call organ withdrawal: they stop taking this person serious[ly] altogether.”

Most executives, particularly those in mid-sized businesses or from emerging economies, told us they face a critical lack of employees with the right skills to design and use analytics tools. What they need most are employees who can bridge functional gaps — translators, that is, between analytics and the business. In the Netherlands alone, thousands of econometricians and data scientists are needed, but only a few hundred are coming on the market each year. Right now, LinkedIn lists more than 4,000 business analytics openings in the Netherlands, nearly 50,000 across Europe, and more than 100,000 in the United States.

Processes. Firms do not have unlimited resources to achieve their goals. Some of our interviewees noted that analytics projects often take too long and lack clear priorities. Analytics initiatives can benefit from processes that are well-defined with unambiguous lines of accountability for the overall objective.

Systems. Many firms must currently make do with a hodgepodge of legacy systems. Some complain that they lack the ability to keep up with the exponential growth of data available. Mismatches between the sophistication of the data and the sophistication of the tools are also common.

Data. Respondents told us that their biggest problem was data quality and data management: Data is often siloed in various places around the firm and not managed in an organized way. Some companies are not even collecting the data they need. “There is a lot of data we are not even generating,” one executive said. “[We] don’t have sensors in our transportation units, don’t have GPS in all of them, or RFID in inventory to know where the merchandise is.”

Of course, many executives at lagging companies are dissatisfied with their current situation and hope to change it. They want to invest in cloud-based storage and computing, better asset tracking, and more technology to enhance the customer experience and track consumer behavior. Video technology is also high on many wish lists, followed by mobile apps. Other executives are looking for mineable product-attribute data that can offer answers to such questions as why certain products are being returned or what customers like or dislike most today.

Most executives also told us they are looking forward to the day when they have higher-quality data and smarter machine-learning tools. They want decision-making support at more granular levels, such as store by store. A number told us they would like more help with demand planning, modeling, and solution strategies. They also wanted help with the integration of additional non-traditional data, such as census and demographic data and data about the weather, in-store customer activity, social-media activity, clickstreams, and online search trends. But it is not good enough to get the data, you also need the tools to convert that data into actionable knowledge.

Getting Started

We recommend two ways:

First, take stock of where you stand. What are the most common important decisions you make? How advanced is the analytics used to make them? Is your culture ready to adopt an evidence-based approach to decision-making? Are you organized to let individual units experiment and innovate in the use of analytics, while at the same time learning from those experiences to expand successful insights from local to company-wide applications? Do you have people with skills to translate back and forth — from business issues to analytics problems, for example, and then analytics output to business recommendations? Do we have a systems infrastructure to collect, store, organize, access, and process all the information required for analytics initiatives?

Second, ask what processes can be improved with better analytics using existing data. How can you improve the analytics used to analyze that data? How can you make them more forward-looking and more advanced in the methods employed?

Once this initial assessment is complete, the hard work begins. The way forward entails organizational redesign and strategic investment. We discuss each in turn below.

Organizational Redesign

The analytical frontrunners we’ve studied exhibit an organizational culture that celebrates experimentation. The mantra “Think big, start small, and scale fast” was commonplace. For these companies, data and analytics are seemingly part of their DNA. Reimaging the organizational culture is no small task, but we recommend starting with a restatement of organizational values in relation to analytics.

Specifically, leaders can spearhead an internal campaign emphasizing that analytics are meant to empower decision-makers, not replace them. Foster a culture whereby employees are rewarded for understanding the predictions and prescriptions generated by analytical tools instead of merely executing the recommendations and rewarding compliance. Overall, any internal resistance to the widespread adoption of analytics can be combatted by opening the algorithmic black box and recognizing that managers are more likely to use analytics solutions when they have some first-hand knowledge of the underlying approach. The goal is to have evidence-based decision-making be one of the most important cornerstones of the firm’s culture.

Structurally, we observed firms on the analytics frontier to complement this experimental culture with a winning organizational design. Many utilized a hub-and-spoke structure in which some expertise is embedded within particular business functions and some is located in a center of excellence dedicated to analytics. This organizational design achieves many benefits. The center of excellence can provide a community to those working on analytics, facilitate oversight, foster knowledge-sharing, and pool resources. And by having some members of the team collocated in the business units, the firm avoids the risks present when the center of excellence works in isolation — notably, the risk that teams will work on problems that are technically appealing rather than practically relevant.

Strategic Investment

It was very clear to us that the firms on the leading edge of analytics that we observed had made substantial investments in their systems. Most had made the strategic choice to replace their legacy systems with cloud-based systems. Why does this matter? It avoids a key challenge that exists when updating legacy systems — namely, the integration of the new and the old. Complaints about the ability of new modules to interface with existing systems abound. New cloud-based systems avoid such challenges and can be designed to scale and utilize the growing availability of big data.

Relatedly, data governance is a key strength of leading-edge firms. Data quality is a priority as is centralized storage. One of the obstacles we identified to the advancement of analytics was the siloed nature of existing data, making it difficult to incorporate enterprise-wide data into decision-making. The decisions of tomorrow involve crossing organizational boundaries (for example, marketing and operations). Breaking down data silos so that, say, pricing teams can incorporate operational factors such as delivery capacity or lead time can only enhance organizational performance.

Finally, and in our opinion, most importantly, firms need to invest in key talent and develop a pipeline for such talent. There are many ways to do so. One possibility is to collaborate with universities offering data-science degrees or similar programs. These programs often seek real-world projects on which their students can work. The benefit of this is two-fold. The students obtain practical knowledge about a business problem and can practice communicating analytical solutions to business leaders, and the firm can learn about the latest tools and preview some of the talent for future hiring opportunities. Another possibility is to develop training programs for existing employees. Tailor-made in-house programs can teach business owners some of the fundamentals of analytics and/or impart business-domain knowledge to those in strictly analytical roles.

• • •

Technological revolutions tend to arrive in two overlapping stages: the introduction of a new set of tools, and then the acquisition of the know-how required to handle them. This second stage, developing the know-how to exploit the new tools, often slows down adoption. There weren’t many electricians around at the start of Thomas Edison’s career, and the Wright Brothers were bicycle mechanics. In this respect, the data-analytics revolution is no different. What is different is the speed with which these new tools are being designed. In the age of data abundance, those who learn to profit from its insights first are almost certain to gain a powerful operational advantage over their competitors.

Note: This article draws on research the authors originally published in the October 2022 issue of the journal Production and Operations Management.