Big data on AWS – a project success story
An Altis project team in Sydney went live this week, with a new big data solution on AWS for a family restaurant chain with over 900 locations across Australia.
What made this a big data project and a success? Typically big data is described in terms of the 3 Vs – velocity, volume and variety. This project loaded 11 billion rows of historical transaction data (that’s right 11 billion rows) and will be adding 15 million rows per day to the solution. Velocity and volume definitely get a tick there.
Before embarking on this project, this client faced common problems we see in organisations grappling with how to make the most of their data assets:
- manual data integration
- replication of information into silos
- difficulty in analysing detailed and granular information in different ways
- duplication of business rules, or inconsistent implementation of rules
The solution was developed and delivered using a combination of AWS and Talend technologies. It’s a cost-effective yet high-performance solution based on a hybrid use of AWS RedShift, AWS S3, Spark and Lambda architecture to take advantage of both batch and stream processing, that will be able to easily scale without compromising it’s benefits. Using a train the trainer approach, we’ve transferred knowledge of the solution and data model to a few nominated power users to help them facilitate self-service reporting.
Some of the key business benefits that this solution will offer are:
- access to detailed, granular data that was never available before
- greater insights into how promotions are directly and indirectly affecting sales
- better understanding of what product offerings are delivering the best ROI
- enabling rollup and multi-level, flexible reporting of custom metrics on top of low level sales transaction data
- improved data governance and consistent business logic
- a robust, maintainable platform using contemporary tools and technology that will scale
The solution is bedding in at the moment so look out for a later blog around some of our lessons learnt in working with these big data and cloud solutions.