AWS Practice – Altis Consulting

How we help

We help you take advantage of cloud platforms such as AWS to support Big Data and Analytics solutions.

AWS Advisory Services

The AWS Practice Team can deliver strategic advice for organisations looking to utilise the cloud for Big Data and Analytics loads. Our services include:

• Strategy, Roadmaps and Architecture Guidance.
• Delivery Services.
• Managed Services.

Strategy, Roadmaps and Architecture Guidance

We can help you design your solution architecture, taking into account your requirements and constraints around Data Volume, Data Latency, Fault Tolerance, Solution Availability and Security.

Delivery Services

The AWS Practice Team helps delivering AWS Big Data and Analytics projects. Our services include: analysis, design, development, testing, optimisation and handover. Typical solution delivery projects include:

Big Data and Data Streaming

Different technologies need to be used when handling large Data volumes and/or when there is a need to analyse the data in near real-time.

AWS Kinesis Service Suite (real-time data processing) and AWS EMR (large data volume processing and storage) are key technologies for Big Data and Data Streaming use cases. AWS S3 is typically used in this architecture for persistent storage.

Data Lake

A Data Lake is used when there is a need to store large amounts of data in a way that will allow a wide variety of usage of data in the future. As opposed to a Data Warehouse, the data format in a Data Lake is not dictated by the usage by downstream systems. The data does need to be catalogued and searchable. Data Lakes normally include the following components:

• • Collection and Storage: This is the main purpose of a Data Lake. In the AWS world, S3 is used for the Storage component. S3 allows storing virtually unlimited amounts of data at a reduced cost. Ingestion of the data can be done via Kinesis Firehose or SFTP for example.
  • Processing: Once the Data is stored on S3, it needs to be available for further processing in downstream systems. Typically on AWS, an EMR instance (AWS Managed Hadoop Platform) running frameworks like Spark, Hive or Pig can be used to process the data. The result of this process can either be written back to the Data Lake on S3 or pushed to other downstream Data Stores for further use.
  • Analytics: Part of the Data Lake data will need to be exposed to end users for Analytics requirements. A common pattern is for this data to be pushed to AWS Redshift and then accessed via Reporting and Visualisation tools. There are many options to bring data from the Data Lake to the Data Warehouse, from traditional ETL to Spark jobs running on EMR.
  • Authentication and Security: AWS services such as IAM and KMS are used for User Authentication and Key Management. Server Side Encryption available in numerous services is leveraged to encrypt the data at rest.
  • Catalogue and Search: Data in the Data Lake should be indexed and searchable. A Dynamo DB can be used to support the Data Catalogue. The Catalogue is refreshed once new data is added to the Data Lake. AWS Elastic Search can be used to search the Catalogue. Lambda functions can be created to keep the Data Store, the Catalogue and the Search Engine in sync.

Advanced Data Warehousing

Leverage AWS technologies to achieve both performance and simplicity, such as AWS Redshift, a fully managed petabyte scale Data Warehouse and S3, AWS Simple object storage used to land data extracted from Data Sources.

Many options exist for the ETL engine, either EC2 based using a RedShift dedicated ELT tool such as Matillion, leveraging AWS fully managed services such as Kinesis Firehose or Spark processing on EMR.

Similarly, there are many possibilities in the reporting and analysis area. A host of AWS Technology Partner tools are readily available on the AWS Marketplace. For a fully managed reporting and analytics capabilities, AWS Quicksight is another option.

Another increasingly popular building block of advanced Data Warehouse is AWS Lambda that gives server-less compute capabilities.

Below is an architecture Altis has implemented for a global company running around 1,000 restaurants in Australia. The client wanted to analyse billions of Point of Sale transactions using RedShift.

In this context, there were two main advantages in using AWS Lambda:

• Firstly, we didn’t have to provision additional servers for file orchestration in our architecture.
• Secondly, it allowed automated near real-time data processing from the arrival of an XML file in S3, to the trigger of the XML parser on EMR all the way to the landing of parsed and cleansed data to AWS RedShift. A combination of S3 events calling Lambda functions made this approach possible.

Traditional Data Warehousing

Mainly leveraging EC2 servers to allow any flavor of technology, solutions typically cover:

• • Database to host the Data Warehouse (Oracle, SQL Server, Teradata).
  • ETL (Extraction, Transform and Load) Engine (MS SSIS, Talend, IBM DataStage).
  • Reporting Layer (MS SSRS, IBM Cognos, SAP Business Objects).
  • Data Discovery and Visualisation (Tableau, Power BI, Qlik).

This approach is generally recommended in a lift and shift scenario where the solution hosted on AWS needs to be very close to the legacy on-premises solution.

Managed Services

After your solution is live in a Production Environment, our AWS trained and certified Managed Services team can provide support of your AWS hosted solution.  Support typically includes:

• Monitoring of instances and services.
• Failure detection.
• Alerts and notifications managements.
• Failure rectification.

Practice Lead

Guillaume Jaudouin

Guillaume is the AWS Practice Lead at Altis and is part of our Sydney consulting team. Guillaume is a certified AWS Architect and has expertise in Big Data, Data Warehousing and Analytics.

He loves travelling and staying in a place long enough to discover the unexpected. Scuba and sport keep him fit and believing ‘all men who achieve great things have been great dreamers’ keeps him dreaming.

Tel: +61 2 9211 1522
guillaumej@altis.com.au