Providing a cost-effective and accessible DataLake

The Client

Aiming to maintain price stability throughout the European Union and contribute to the safety of its banking system, our client for this project is a state bank that manages government accounts. They also process treasury payments, support public companies, and handle public security auctions for the European Central Bank.

The Challenge:

As part of the AnaCredit project introduced by the European Central Bank, our client needed a new way to store the data required by the initiative – namely, a monthly collection of 94 attributes attached to 21 million lines of credit.

Putting a halt to business as usual to find the resources to store and organise massive amounts of granular data was out of the question. Part of our client’s request was to produce an outcome that would increase employee productivity and data analysis capabilities.

A health check of current practices had to be conducted to reach the best solution. Options like creating bespoke software to complement current databases, such as our data management solution deepeo, were considered – but ultimately our client needed an on-premises to store, process and secure large amounts of data in its original formats.

So, we recommended and began to build a multi-disciplinary DataLake.

Project Outcome

The successful delivery of the data lake project was underpinned by cutting-edge technologies, an expert Agile team, and a commitment to innovation.

Technologies Leveraged

The data lake was built using a robust technology stack, including:

• Big Data Tools : Hadoop, HDFS, Hive, HBase, Oozie, and Atlas for scalable data storage and management.
• Security Frameworks : Ranger and Knox to ensure secure access and governance.
• Analytics & Reporting : ElasticSearch, Kibana, PostgreSQL, and modern tools like Java and R for advanced data analysis and visualization.
• Development & CI/CD : WildFly, GitLab, and Jenkins for seamless development and deployment.

The project achieved remarkable outcomes that transformed the client’s data operations:

• Enhanced User Experience : Supported the evolution of user usage with intuitive tools and features.
• Innovative Customer Services : Delivered agile and cutting-edge solutions tailored to client needs.
• Streamlined Data Management : Simplified data sharing and facilitated the collection and analysis of large datasets.
• Modernized Reporting : Introduced advanced reporting tools for Banque de France agents.
• Boosted Productivity : Increased data analysis capabilities, improving overall efficiency.
• Digital Transformation : Contributed significantly to the client’s journey towards a modern, data-driven organization.

This project not only addressed the immediate need for a stable and scalable data lake but also positioned the client for sustained success in their digital transformation efforts.

Our staff can sometimes feel daunted by the amount of data we acquire and how to best access it to keep it safe. We hadn’t considered a solution like a DataLake, and the expertise offered was as clear as it was reliable.

External Project Manager