Why has everyone started talking about the “new term” Data Lake over the past two years?

My view is that user needs still drive data services. The fundamental reason for interest in data lakes is a qualitative change in those needs: traditional warehouse architectures and their components can no longer keep up.

Trends

One particularly important trend is:

• Real-time data

Others include lower costs and cloud-native design, but real-time data is, in my view, the hottest and most visible trend of the past year or two, with the most readily apparent benefits.

The traditional warehouse model is familiar: divide it into ODS, DWD, and DWS layers, use Hive for storage, and use Spark or MapReduce for data cleaning. It is clear and manageable, so we have happily followed this model for roughly ten years.

During that decade, mainstream internet companies’ data practices changed less than one might think: user profiles for recommendations, product tags for e-commerce, graphs for social propagation, and financial risk-control systems. From a higher perspective, a user-profile table needed ten years ago is still needed for a recommendation service today. A decade of accumulated talent, knowledge, and experience makes these tasks easier. Expertise in recommendation data that was hard to recruit ten years ago may now be merely the industry average.

As these tasks become easier and talent more plentiful, we want greater refinement. Recommending short videos or encouraging purchases has no final endpoint; it can always improve. Previously, learning a user’s interests the next day was enough. Once that became easy, we wanted behavior from the first ten seconds of a visit to reveal their preferences. Broad campaigns targeting everyone give way to finer operations and personalized results for each user.

Technical Evolution 1

Real-time data sounds good, but what does it mean technically? Do we need a real-time data system modeled on the batch warehouse?

Many companies indeed divide real-time streams into layers resembling their batch warehouses. The data carrier, however, is no longer Hive or HDFS: it is a real-time message queue such as Kafka. This brings problems:

• Message queues have limited retention
• Message queues lack analytical querying capabilities
• Historical reprocessing is less efficient than with filesystems

Beyond the data carrier, introducing a real-time warehouse raises questions about unifying it with the batch warehouse:

• Must governance and permission management be implemented separately?
• How can real-time and batch computations use consistent definitions?
• Do two systems waste resources and increase costs?

For a practical example, suppose a real-time metric is wrong and yesterday’s data must be corrected. Usually, we write a separate batch job, read data from the batch warehouse, recompute, and write the results to storage. Every real-time requirement therefore also requires a batch job, a huge engineering cost.

Technical Evolution 2

The cost of real-time systems is one reason many companies hesitate. This warehouse-building approach cannot be right: it requires twice the talent, perhaps more, and twice the time for a feature that might improve the business by only 10 percent. Technically, different stacks prevent engineering unification. Data lakes address this: can one batch job produce both real-time and batch metrics, as in the following diagram?

Beyond computation, can we unify data management, including schemas for intermediate tables and access permissions? A unified architecture can minimize duplication between streaming and batch, potentially adding almost no extra cost when meeting real-time requirements.

We are actively exploring this too. Most domestic internet companies remain at the “Technical Evolution 1” stage. With continued effort, I believe strong, successful implementations will emerge soon.