The data lake, or the promise of (finally!) breaking down information system silos
The concept of the data lake, which is closely linked to the Big Data movement, involves setting up a unified storage facility for the various types of information held within a company.
Unlike its distant cousin, the data warehouse, the data lake has a key characteristic, namely the absence of a strict schema imposed on incoming data flows. The result is a high degree of flexibility, making it easy to interact with the data, whether raw or refined. Another key feature of the data lake is that it can easily process and transform information to accelerate innovation cycles, and thus support the company's various data projects.
The emergence of data lakes is due to the new Big Data technologies, which have brought about an economic breakthrough in the cost of data.
A large number of companies are currently looking into the possibility of setting up a data lake, whether for their customer relations (exploiting all interactions and creating a 360° view of the customer), their industrial facilities (collecting manufacturing data, but also data linked to product use, for the purposes of preventive maintenance or product optimisation) or their financial flows.
And it's a safe bet that in the age of the Internet of Things, the scope of data lakes will continue to expand.
A complex technical foundation
Hadoop may seem the obvious choice for building a data lake, but it would be simplistic to think of this technology alone, which is mainly storage-oriented. Other software components are needed for data processing, both in batch and real-time mode, and for peripheral needs such as visualisation, data science and data governance. Integrating all these components (which, being Open Source, are still constantly evolving) requires specialised skills that are rare on the market.
Setting up a data lake also means mastering Cloud environments. When it comes to building a data lake, the Cloud is undoubtedly the best option, as it allows resources to be provisioned on demand, so that the infrastructure can grow as and when required.
Data lake as a service: ready-to-use Big Data processing
Although companies today have access to almost infinite quantities of data, they do not have fast, inexpensive solutions for exploiting it.
To meet these expectations and avoid embarking on a lengthy and costly on-premise deployment project, certain players such as Bigstepor Cloud Temple in France, offer 'data lake as a service' solutions. They provide their customers with a set of software components in the Cloud, ready to use because they are managed, so that they can concentrate on developing their data applications.
Given the scale of the task that a data lake represents, "Data lake as a Service" provides a managed service that offers a ready-to-use platform capable of growing with the needs of the business, and concentrating on the production and exploitation of the company's data.
Pierre Schaller, DGD Cloud Temple
See also :