The Altis Buzz – January 2017
Our Big Data and the Cloud War Stories
Last year, over 40% of our delivery work utilised Big Data technologies such as Spark and Hadoop or was delivered in the cloud using AWS or Microsoft Azure platforms. There were two themes that stood out for these projects.
- Clients are comfortable with an agile “learn quick/fail fast” approach. With the number of open source, cloud services, and commercial products available to solve a problem; you need to have an appetite to try different approaches and frameworks to determine the best fit for your business problem.
- Turning data into business decisions faster was the primary objective. This was common across both the above clients and our traditional warehouse clients. While our data warehouse clients continue to utilise our accelerator packs, our Big Data clients leveraged our integration and agile delivery experience.
Below are a few of our recent project experiences that demonstrate the above principles that you can benefit from.
We are delivering a Big Data Project using Spark for an ASX top 50 company and their objective is the speedy dissemination of information to multiple systems for improved client experience. Using three week sprints and tools such as Spark and Kinesis, Altis is designing and building streaming interfaces, in-memory ingestion and transformation rules while exposing the information via an API. In addition, we are storing the information in Hadoop and preparing the platform to support future machine learning capabilities. Along the way we are testing a variety of tools and AWS hardware configurations to find the most cost effective throughout.
In another similar example in the financial services sector, we are designing and building the ability to visualise a ‘single view of customer’ (yes, that old chestnut) enabling front line staff to have a near-instant view of the client they are working with versus yesterday’s information. In this case, we are leveraging Hadoop, HBase and a Scala framework and again are trailing a variety of configurations and tools to optimise the solution.
For one of the banks, we designed and built a Tableau reporting and discovery front end interfacing via Impala with Hadoop. The source systems for Hadoop include their existing data warehouse, new technology stack data warehouse, and unstructured data sources.
In the utilities sector, leveraging three two-week sprints, we designed and built an IoT stream analytics and real-time reporting solution using the Microsoft Azure streaming stack.
Speaking of AWS and Microsoft Azure, we have been involved in a significant number of new Information Management solutions and/or migrations of existing workloads to these platforms over the last 12 months.
Altis has now delivered two green field enterprise data warehouses in Azure utilising our cloud accelerator packs. There were definitely some learnings along the way of what works well and what does not. In addition, we tested the feasibility of building a logical/virtual data warehouse.
In the AWS world, we have delivered multiple Kinesis, Redshift and Aurora projects leveraging both AWS data and analytic services and open source products. As a result, we are currently leading one of the largest Point of Sale analytics projects in Australia.
If you want to find out more about our experiences, offerings or other war stories including a massive client program that reset itself from 60 people where we provided a few consultants to a team of 3 that we are now leading, leveraging a LAMBDA architecture and lots of RAM, then stop by our booth at any of these events in February:
- Gartner Data and Analytics Summit in Sydney – 20th – 21st February
- BI Summit in Auckland 13th – 14th February
Or give Katrina (ACT & NT), Emilio (NSW, QLD & WA), Andy (VIC, TAS, SA), Alex (NZ) or Peter (UK) a call.
Altis now has a dedicated innovation group headed up by Mythili Baker who are looking at developing technologies and processes to help make our clients lives easier. Here is just one of the innovations that our team have been working on.
Location Cleansing using GNAF
We have developed a Microsoft template which will quickly enable you to cleanse and geocode your data. Some of the features of our template include:
- A pre-built connection utilising the free GNAF (Geocoded National Address File) published by the Australian government
- Pre-built logic allowing fuzzy matching of your location data to the GNAF data set
- The ability to append longitude and latitude information to your existing data
- A proven solution, previously implemented at a government department to automate a highly manual process, with a match rate of 99%
View a printable version of this newsletter here