Data processing system for an Ecommerce and Emarketing platform for an insurance provider

• Industry:
Insurance
• Revenue:
€1.5 billion
• Employees:
3,100
• Location:
Worldwide

Design and development of a data-management platform to collect, transform, and distribute data between a large variety of sources and targets.
The marketers use the software to manage insurance products sold on the Internet and to prepare and run campaigns.
The software has to be interfaced to the corporate, subsidiary, and partners’ systems using heterogeneous technologies.

• Create a back end system to faster collect and transform high data volumes from numerous sources
• Distribute the data in a normalized format to different sets of e-commerce and e-marketing platforms
• Deploy the back end from scratch for any subsidiary in less than two months, including local specifics and regulation
• Allow marketers to create brand-new products from scratch without IT support

• Deal with a lot of heterogeneous data sources, most of them being unknown at the beginning of the project
• Propose solutions, despite no clear vision nor requirements from the business team
• Often change the short-term planning to adapt the software to new opportunities or strategies
• Process millions of records in minutes and serve them in seconds
• Imposed teamwork with a third-party IT provider in charge of the front ends
• Work with different database systems

I designed and developed a Java-based back end based upon the Spring framework and using technologies as standardized as possible: JPA, REST, SOAP, among others.
The whole module-oriented platform was a web application running on Tomcat 8 and using MySQL, PostgreSQL, and Oracle DB.

More specifically, I used the following:

• Spring Core for the dependency injection
• Spring Batch for the data stream processing
• Spring Security
• Jersey for the REST implementation
• Hibernate for the low-volume database operations
• A tailor-made JPA-annotation-based module, able to persist thousands of records per second to the database for the high-volume operations
• A tailor-made plugin strategy to easily integrate external modules with a few lines of code
• Vaadin for the administration UI

The system was highly configurable and extensible. Thanks to Maven, each build only contained the required components for the targeted environment.
Finally, I set up the Scrum framework in order to match project management with the speed and flexibility expectations from the business function.

The software overreached the goals and allowed deployments from scratch in hours thanks to continuous delivery. Most of the time was spent on configuring the environment and monitoring. The modular architecture allows new features to be developed, tested, and integrated very fast. New subsidiaries can be operational shortly after their decision to use the software, improving their velocity, time-to-market, and ability for change.
Its performance and reliability and the usage of mature technologies made it cheap to host and maintain. No major bug was detected since operation started.
Furthermore, the detailed monitoring of all the activity on the platform made the investigation of potential issues incredibly fast and made it possible to detect the real cause of the issues in minutes.
Each service of the platform is under constant monitoring to detect anomalies in the usage or execution. The number of calls and the execution time were mainly used to detect inconsistencies in the front ends.

• Short ramp-up to learn about the customer, its business, and its culture
• Usage of agile practices to constantly match the developments to the expectations, even when very volatile
• Good technology stack: robust, flexible, and adapted to the specific requirements
• High software quality improved customer satisfaction and accelerated the development of new features