The need for Data Integration has never been greater. Organizations are faced with an overwhelming growth in data, an increasing range of diverse data sources and the continued proliferation of systems within the enterprise.

In this blog I will talk about the hot topics and key trends for Data Integration in 2012 and beyond.

First, a definition – ‘Data integration refers to an organisation’s ability to gather data and information residing in multiple sources, combine it and provide a unified view to support business goals and initiatives’.

So what are the Data Integration hot topics and trends?
• Data Integration Platforms – the right mix of capabilities
• Managing social data
• Profiting from ‘Big data’ and analytics
• Driving efficiency from Cloud Data integration
• Data quality – the essential ingredient when taking data from multiple sources.

This week we will focus on trends in Data Integration Platforms.
Data Integration requires a range of capabilities including Extract, Transform and Load (ETL) working with complementary capabilities for data sharing via XML messages using an Enterprise System Bus (ESB) and data federation (or virtual data integration) utilising Enterprise Information Integration (EII).

A true Data Integration solution needs to include all these different capabilities to provide a complete platform to support all of the organisation’s Data Integration business initiatives e.g. Data Warehousing, Master Data Management, Enterprise Application Integration (EAI), SOA and Data Migration.
Companies who are successfully managing their data and achieving efficiency and effective business results are recognising that to succeed in Data Integration no single capability will address all business Data Integration initiatives. For example, an Enterprise System Bus (ESB) is not the right capability for bulk movement of data and complex transformations; this must be performed using an Extract, Transform and Load (ETL) capability.
Many Data Integration projects have failed due to an incorrect selection of the right Data Integration capability. With a Data Integration Platform in place, with the right mix of capabilities, this should no longer be an excuse for a failed project.

Next time we will look at trends in Social Data.

Andrew Painter
Senior Consultant Altis Consulting

Data Quality Blog – A Never Ending Story

                         Where’s a spell-checker when you need it?

I was recently invited to provide one or more data quality anecdotes for a chance to win some glorious prize. I was hooked, more for the opportunity to contribute an anecdote and less for the prize. I was also amused that there’s still such widespread appeal for these embarrassing events and, more worryingly, that there are still plenty of embarrassing events happening.

I decided that a blog focussing on data quality stories would provide some insight into how data quality issues can impact business and why it’s important we don’t just accept poor data but we actively work to fix the problems (and stop these lifelong embarrassing moments).

This blog is the first in a series that will focus in a light-hearted way on DQ anecdotes and stories, to examine how such errors occur and what can be done to prevent them in the future.

To set the scene let’s look at what the term Data Quality means. My colleague Veronica Coyne recently identified the dimensions of Data Quality, in her blog ‘What Does Data Quality Mean?’ (http://altis.com.au/what-does-data-quality-mean/) Veronica agreed with the importance of the dimensions identified by Larry English (http://www.infoimpact.com/) these are:

• Existence and Completeness
• Validity (conforms to business rules)
• Accuracy (correctly represents characteristic of real world object or event)
• Precision (values are correct to the right level of detail)
• Non-duplication (each record appears only once in a data store)
• Equivalence (of redundant or distributed data)
• Timeliness (for all knowledge workers)
• Currency (data is up-to-date)
• Presentation Clarity (data is presented in a way that clearly conveys the truth)
• Relevance
• Definition Conformance (the value of an attribute is consistent with the attribute’s definition, e.g in type and format).
• Plus her own useful additions:
• Auditable and
• Trusted.
• Consistent

The purpose of Veronica’s message was that measuring data is necessary to managing it and for that you need to have clearly defined measures or in this case data quality dimensions. My purpose is to look at what happens when you get it wrong and then focus on the next step “What do we do about it?”

These blogs are to help highlight the DQ dimension(s) embedded and show their potential impact on your business.

A tip for the uninitiated here: A DQ occurrence is hardly ever likely to have a single outcome. The chances are you get it wrong once you get it wrong over and over.

Data Quality Felix

Here’s my first example – ok it’s not real, it’s a TV commercial, but you can easily see how real this example could be:

Remember this ad from a couple of years back? A certain large bank had Tony Barber accompanied by an obvious senior manager cum trophy holder; photographers, media hack and cheer-squad. They knock on the door of some house and tell the unsuspecting door-opener that he was being presented with an award because the bank was the umpteenth winner of Home Lender of the Year. The joke was they had the wrong person – the guy they want is next door. Poor data quality – you betcha! Did the management at the bank know they were making a joke out of poor data quality?  Seems to have eluded their management – or did it?

I wondered how the experience of Altis consultants could contribute a number of real-life anecdotes. When you work in the DW/BI space long enough you can’t help but capture interesting stories along the way. That said, you may only need to be a new entrant to pick up a story or two, because the issue’s not going away. It’s almost as if (poor) data quality is an implied cost of doing business and organisations have given up trying to address it and simply, even happily, accept the situation.

To set the scene, here are some references to locations where you can see various data quality results posted.

Now it’s over to our Altis story-tellers to come forth with their tales. Stay tuned.

Larry English – Information Impact

Daragh O Brien – IQ Trainwrecks

Darryl Collins – Data Quality Management

About the author:

                                    Paul Priestly is a Senior Consultant with Altis Consulting and has over 20 years’ experience in Information Management and IT project management within public and private sector organisations nationally and internationally. Paul specialises in strategic initiatives for DW/BI, data quality, MDM and data security.

I am excited to formally announce the opening of the London office for Altis Consulting. We are taking our specialist expertise in business analytics, business intelligence, data management and data warehousing to the UK market and I am proud to say I will be leading the UK practice.

Our goal is to offer a service that is vendor-independent, more flexible and offers greater value than our competitors.

With a blend of expertise across numerous products in the Data Warehouse and Business Intelligence space, combined with a deep commitment to the skills of our staff, we believe Altis is uniquely positioned to offer this service to UK clients.

Many BI projects end up as shelf-ware and never fulfill their potential. But our approach is different. Unlike other data management consultants we are not tied to specific vendors, so we can provide completely impartial advice, services and training – we make sure we are up to date with the offerings provided by a whole range of leading vendors. This enables our clients to better address key challenges and realise their objectives through better reporting and utilisation of business metrics. We pride ourselves on taking a less technical approach, and because we are privately owned we are more flexible, agile and provide less expensive consultancy than the larger companies in the UK.

Although we have experience across all sectors our initial focus in the UK will be on the Construction industry, Utilities and Higher Education, where there is huge demand to report and analyse information that can be applied to drive efficiencies in these industries.

I have over 15 years of experience in the Data Warehouse and Business Intelligence fields, having worked both client-side and for a number of consultancies including CMG (now owned by Logica). For the last six years I have been at Altis in Australia, but as I am originally from the UK I was the logical choice to head up the Altis expansion.

Altis provides an exceptional level of training to our employees and clients. Our philosophy is based on looking after our staff so they are better equipped to look after our customers. Our partnership with experts in the field such as Kimball University, for leading Data Warehouse training, and Stephen Few, one of the World’s leading experts on data visualisation and the author of several well-known books including ‘Show us the Numbers’, enables us to provide training in data warehousing and business intelligence that is second to none. All of our staff are offered the chance to attend these courses to ensure they are up to date with leading best practice, and we will be making the best use of these strategic partnerships in the UK market.

I am also excited about the opportunities for staff who may want to live and work in the UK and for our UK staff looking for opportunities in Australia. We see this cross-fertilisation of talent as one of the key benefits of this move.

Check out the press release at sourcewire.

There’s a lot of restructuring going on in IT companies at present and many people in our industry are looking for work. This can be an unsettling time for IT professionals who are not used to job seeking, but it can also be a great opportunity to find a new role in the company of your choice.

 If you’re looking for a consulting role in the DW/BI space, here are some recommended do’s and don’ts:

DO – throw your net far and wide. Opportunities may exist where you don’t think of looking. In particular, consider contacting companies directly. It’s true that some larger companies will only deal with agencies, but this is often not the case. Go to a potential employer’s website and check out their careers portal. Even if you don’t see the right job for you, it generally indicates that they accept direct applications.

DO – research the companies you would like to work for. Check out their employment practices, what their people say about working there, do they invest in their people, do they provide a career path, do they have a culture compatible with your values? If so make a direct approach.

DO – cover your bases and contact agencies, but don’t limit yourself to a single agency – they only have so many jobs on offer and they may not have the right one for you.

DO – make sure that your resume is up to date and that any written communications you have with agencies and employers are faultless. Agencies and employers alike see large numbers of resumes, so it’s important that yours stands out. In particular:
• Use concise English. Waffle is a turn-off.
• Do a spelling and grammar check. Errors lead to a poor initial impression.
• Don’t put a photograph on your resume (unless you’re exceptionally good looking).
• If business writing is not your forte, consider getting professional help writing your resume

DON’T write off a company that doesn’t respond immediately to your email or phone call. It may be that a message was not delivered, or that your application was misdirected. Try again, persistence works!

DO approach Altis. We talk continuously to candidates whether or not we have a position available. We have a direct recruitment model and it’s not uncommon for us to maintain a dialogue with candidates who we employ later on. And if we don’t believe you have the right background or skills, you’ll get a polite response and nothing’s lost.

Roy Hammett

People and Performance Manager, Altis Consulting

Being aware of and managing subtle cultural differences can make all the difference when managing a successful Global project.

 Recently I held a post project review session as part of closing out a global Business Intelligence project. The objective was to share and collate feedback on what had gone well or not so well and why – so this could be passed on to future project teams. At the end of the call, I found I could summarise the type of responses I would get on a topic from each region:

• One region was very positive and seemed careful to respect any ‘organisational rankings’ within the team in terms of their feedback.
• Another region’s feedback was extensive and mostly negative.
• The other region has some no-shows and virtually no comments or feedback despite during the project being the hardest region to implement.

 Perhaps if you’ve also worked on global teams you can guess which region was which?

 For me it was a perfect summary of the subtle cultural differences we deal with on global projects. There is definitely a downside to global projects. We have calls at unsocial hours (it’s always someone’s bed time, someone else’s waking up time and another’s lunchtime). The 24hr turnaround always means tasks take longer than expected to complete. And the effort of communication is always so much higher.

 On the other hand, diverse teams can generate improved productivity, creativity and results. There can also be a sense of excitement and discovery working with global partners which can drive team performances.

 So are you getting the most out of diverse teams?

Below are some tips to make the pain worth it!

•  Try to include a budget for an onsite team meeting. This can be a real team-building exercise where you can agree how you will work together, how to exploit each team’s strengths, and it’s amazing what a difference it makes talking to someone over the phone when you can picture their face.
• Agree the processes for working together, how you will report, resolve conflict and what is expected from each participant and region.
•  Spend significant time upfront sharing each region’s vision for success.
•  Avoid difficult topics (including introducing anything new) via email; remember 70% of communication is non-verbal, and subtleties are better communicated when you see each other.         
•  Utilise technology such as web cameras, online conferencing with video, as well as custom conference call facilities. These are definitely a step up from phone calls and can be used to substitute for travel.
• Involve the regions as much as possible in planning and talk about how you will communicate. Explain and use techniques like sharing personal anecdotes at the beginning of a meeting as ice- breakers.

So those are my experiences and tips, I’d be very interested in any stories or tips you can share.

 Mythili Baker
Principal Consultant

Last week we were fortunate to have Warren Thornthwaite and Margy Ross from the Kimball Group deliver an excellent Data Warehouse Lifecycle In Depth training course in Canberra. Seizing the opportunity to gain insight from an industry thought leader in data warehousing, Altis hosted an executive briefing where Warren shared some of the lessons he’d learnt from his extensive experience with DW/BI implementations.

Warren’s presentation entitled “DW/BI Wisdom of the Ages” provided insightful and useful points in dealing with the many challenges of implementing a DW/BI solution, and his talk strongly resonated with his audience.

Some of my key take-aways of the common challenges included:
- Not having a defined measurable goal or outcome. (Ask yourself what is the purpose of the DW/BI solution)
- Business systems or solutions being driven by technology alone
- Organisation priorities changing faster than the core services can keep up
- Failing to manage expectations
- Failing to publicise solution and successes

So what can be done about it?
1. Expectation management and clearly defining the outcomes go hand in hand. Narrow down the initial scope to a manageable deliverable that must add business value. If you aren’t delivering business value, the purpose of the DW/BI solution is easily called into question. After initial delivery, celebrate the successes and move on to the next priority iteration. Iterative developments will allow greater flexibility to keep pace with changing requirements and moving priorities.
2. No solution should be driven by technology in isolation of the business. Build a bridge between business and IT and work as strategic partners in the organisation (rather than adversaries). This relationship is key, regardless of the tools or technologies involved.
3. The best solution in the world won’t increase its customer base by word of mouth alone. Educate your potential users in what is available to them, develop user champions within the business and constantly communicate what’s being done, what the DW/BI system can do and where it’s headed.

Following on from the presentation, conversations ensued about how many of the executives in the audience had experienced the challenges described by Warren and how they saw opportunities to prevent them becoming a problem for their current implementations.

To ensure you’re invited to future briefings or courses contact
Altis Consulting
www.altis.com.au
+61 2 9211 1522

In line with its annual major release cycle, Tableau has today released version 7 of its software. This marks the seventh major release in the company’s seven year history which is quite an impressive achievement.

I have been playing with Tableau 7 during the Beta program and had an in-depth preview at the Tableau Customer Conference in Las Vegas in October 2011. Tableau 7 has some cool new features from both a front-end and back-end perspective, some of which I’ll share in this post.

This is by no means a full list of the new features. Visit the Tableau web site for a full list: http://www.tableausoftware.com/new-features/7.0

Filled Maps

Filled maps are available at the county level for the U.S. and country and state/province level for the rest of the world.

Growth in Employment image copied from the Tableau web site.

Profit by State

Population by Country

Area Charts

Area charts fill the area below the line in a line chart. They can be used to show the change in value for individual lines as well as the growth of the whole.

Flights Delayed at Chicago O’Hare image copied from the Tableau web site.

Show Me

By default, Show Me now remains open as a floating palette to simplify changing views (e.g. between a Text Table and a Bar graph) or to add new fields to a viz. It also now tells you how many Dimensions and Measures you need for a particular viz. Bar charts now default to horizontal.

Geocoding and Mapping Enhancements

Locations that are unknown to Tableau no longer unexpectedly appear at latitude zero, longitude zero off the West coast of Africa (what will this mean for the inhabitants of The Republic of Null Island?! http://nullisland.com). Tableau now displays a message at the bottom of the map stating the number of unknown locations and giving you the option to map them to locations that Tableau is aware of.

Tableau also caters for poorly spelt as well as foreign spelling of locations. Locations that it cannot understand are treated as per above. Custom latitude and longitude values can be entered on the fly for locations Tableau is not aware of.

New Main Menus

The main menus have been restructured. Example, there is now a new Worksheet and Dashboard menu making it easier to find things. The Table and Bookmark menus have been removed with their contents moved to other menus.

Dynamic Sorting

Tableau now dynamically updates the sort order of marks on the screen, even as the underlying data values change. This was possible in Tableau 6.1 via calculated sorting but it now also works via the one-click sort buttons on the toolbar.

The viz below shows Country sorted in descending order by population. I used the one-click sort button to achieve this.

I manually changed China’s population to be 1 billion and refreshed the viz and it re-sorted the Countries so that China moved to second place.

Enhanced Tooltips

Tooltips now include statistics for the marks currently selected. Tooltips offer increased functionality with “Keep Only”, “Exclude”, one-click sorting and view data capabilities.

Dashboard improvements

The placement of worksheets onto a dashboard has been improved by the grey visual feedback indicating where the sheet is going to be placed.

Tableau Data Server

Tableau Data Server is a new component of Tableau Server 7. Data Server provides the ability for database connections and Tableau Data Extracts to be centrally managed. Users of Tableau Desktop can now:

• Connect to the Tableau Data Extracts on Tableau Server without having to download the data extract. The queries to populate the data extract are executed on Tableau Server rather than on the desktop. I.e. the data extract is created once and can then be used by multiple workbooks. This has the performance benefit that the data is not being extracted from the data source multiple times.
• Connect to a centralised connection (“proxy connection”) stored on Tableau Server. The benefit of this is that users don’t have to define connections themselves and nor are they having to have various pieces of database connectivity software installed, configured and updated on their computers. The database connectivity software is installed on the Server and then the connection centrally configured and made accessible to users.

Image copied from Tableau web site.

Conclusion

Tableau 7 has a lot to offer end users in terms of new and improved visualisation capabilities and easier to use functionality. It also has something for the IT department in terms of improved central management with the new Data Server.

This is a solid release from Tableau and I look forward to teaching it to classes and using it in the field.

Visit the Tableau web site for a full list of new features: http://www.tableausoftware.com/new-features/7.0

About the Author

Chris Kearns has been in the IT industry for over 20 years and has spent the last 15 years focussed on Data Warehousing and Business Intelligence.

Chris joined Altis Consulting as the first employee in 1998 and has worked as a Consultant, Quality Assurance Manager, Education Practice Lead, Account Manager, Tableau Certified Trainer, and Tableau Practice Lead. He has helped to deliver solutions to many customers across a variety of industries in Australia and New Zealand.

When not at work, Chris can be found on the tennis court, playing the guitar, singing or swing dancing;  usually not all at once.

I’ve previously had a simplistic view of Master Data Management (MDM), that its aim was to build and maintain a ‘Master Entity’ that is clean, unique and free of data quality problems. Something that proves costly on most Data Warehouse Business Intelligence (DWBI) implementations. However after attending William McKnight’s Master Data Management course I now realise there’s much more to it.

MDM is a more integral data solution than current DW implementations. Traditional DW solutions are a one-way street of information traffic: extracting, transforming and loading data from one system to another.

In this approach, latency and distribution are issues. MDM, on the other hand, allows consumption and update of source records in real time, and at the same time publishing quality data to other applications downstream such as DWs, Analytics, etc.

Master Data Management surfaced as much more than reference data. MDM provides clean, quality information that can be ‘distributed’ throughout the enterprise applying total governance on such information. The distribution concept introduces actors in the information management chain such as ‘consumers’ and ‘subscribers’. The advocacy was towards an MDM engine closed to the source systems applying data quality upon data entry.

MDM is blurring the boundaries of Data Warehouse to some degree, questioning the relevance of it in the future. According to McKnight, MDM subject areas enable analytical applications straight off the MDM data hub.

Although possible, I question the effort needed to implement an analytical MDM model that will provide the same capabilities of a well-designed DWBI solution, considering that the industry does not provide for example, MDM data models advocating best practices.

From an implementation perspective, vendors push hard their own models but in most cases, the final cost to customize those models for particular business models is extremely high.

MDM advocates the role of a data steward. Someone who is the business owner of corporate data, responsible for sign-off, approval and updating their value. This is done through what McKnight calls Master Data Workflow.

The workflow essentially passes the ball to the business prompting them to make the decision if a particular data item is fit-for-purpose and can be given the green light for its use enterprise-wide.

Enterprise Master Hub. MDM enables data as a service (DaaS). This ultimately provides the holy grail of quality information services: “build once, use many times”. On this front, MDM clearly provides great benefits up and down streams of data flow.

Obviously there are many other important points on MDM. It is not difficult to see why the value proposition of an MDM solution is more than interesting, particularly in situations where businesses are embarking on the upgrade or decommission of their ERP and CRM applications.

MDM can be applied at any time, however these represent prime opportunities to start the MDM journey. The context is perhaps a little more prohibitive or challenging in private enterprises, as well as government departments with mature DWBI capabilities. In these situations, it is wise to tackle such a journey with baby steps.

Perhaps the simplistic view that I started with is not such a bad idea after all. We should start by taking care of core enterprise entity data quality problems, applying quality processes and governance around conformed dimensions in order to acquire their golden status where the entity is clean, unique and free of data quality problems. After a successful implementation, and having the backing of the business, then move onto the next stage towards a complete MDM solution.

About the author: Florencio Espejo is a Senior Consultant with Altis Consulting, and has more than 10 years experience delivering IT solutions specialising in Information Management. He has a varied background working with organisations across a range of fields including manufacturing and healthcare services, as well as Government.

A colleague and I recently attended Sharepoint Saturday in Canberra. A day-long conference of all things MS Sharepoint. The sessions were broad ranging, appealing to business focussed and highly technical alike.

Most business intelligence tools in the market today are accessible via a browser-based interface or portal. With the increasing use of collaboration tools, integration between these and business intelligence tools to give users a single platform to meet all their information needs is an interesting direction.

My key takeaways from the day were that many of the principles advocated for best practice implementations of Sharepoint 2010 have significant parallels to data warehouse and business intelligence best practices, and that the rise of unstructured data is challenging the traditional view of a DW/BI solution.

Information Management Best Practices advocate:

• Finding information easily, and ensuring it is the right version (Getting the right information to the right person at the right time)

• Maintaining a single version of the truth

• Reducing duplication of processes or recreating information repeatedly to be more efficient

• Using an iterative approach to developing the solution

Fundamentally these principles are common sense but need to be underpinned by good information management governance. The key theme that came across from many of the presentations was that technology alone, in this case Sharepoint 2010, will never be the entire solution. This technology provides a flexible and easy to use platform that encourages self-service and collaboration.

Unstructured Data

With Sharepoint 2010 Microsoft is offering a platform that allows business users to access both their structured and unstructured data from one place. Centralising emails, documents and spreadsheets (the bane of many a traditional data warehouse) into one place, and offering an integration of these different data sources with structured data in a way that appears seamless (to the user) is a very attractive promise. The proof will be in the success of implementations that can deliver upon this.

We look forward to the possibility of a user group session that explores Sharepoint 2010 for business intelligence in more detail in the future. Thanks to the Sharepoint User Group in Canberra for arranging this session.

About the Author: Nadine Hamilton is a Senior Consultant with Altis Consulting and has over 10 years experience in managing, designing and implementing Data Warehouse and Business Intelligence solutions. Nadine has been involved in multiple DW and BI projects in the UK and Australia, in Finance and Government.

Agile data warehouse test led development involves creating the test cases prior to development, and is a key part of the Agile development process, as it helps to clarify requirements, provide an accurate gauge of progress, and an early look at the data.

Here is the process for making it happen.Agile data warehouse test led development involves creating the test cases prior to development, and is a key part of the Agile development process, as it helps to clarify requirements, provide an accurate gauge of progress, and an early look at the data.

Here is the process for making it happen.

1. Identify Case Scenarios

Develop general business scenarios, for example, recipients who have left care and then returned after less than 28 days.

2. Extract Data that Matches the Scenario

Identify customers (or cases) that illustrate the identified scenarios, and extract all the information that is needed to determine the outcome. If the case doesn’t exist then copy and manipulate an existing one to meet the requirements.

Ideally, you already have the input data at this stage. By looking at the detail you experience data quality problems early. If you have difficulty finding scenarios that you expected to be common, that’s good to know.

3. Determine Expected Results

For each scenario, determine the expected results. It’s worthwhile to workshop the expected results with a few people, such as developers as well as people who understand the business rules.

By working through cases step by step, it clarifies what various terms mean. It’s better to discover this before coding, than during User Acceptance or Production.

4. Design and Code as Appropriate to pass as many tests as practical

This is the development part of Test-Led development. By the time we have reached this stage, we have an excellent understanding of both the high level (business terms and meaning) and the necessary detail (what to do in each case).

If there is a particularly complicated business rule that can be solved later, I recommend building code that attempts to solve all cases except that one. Once they have passed, you can tackle the outstanding business rule, and the other cases can be re- tested at the same time to ensure no bugs have been introduced.

5. Compare Results

This provides an accurate gauge of progress. It’s much better to have 90% cases complete, than 90% completion of only 50% of cases.

6. Repeat Step 4 if Required

If you find a problem, or you’re adding more functionality, you’ll need to return to Step 4. At this stage you would have passed all the tests, and there is increased confidence that the code is correct.

About the Author: Simon McAlister is a consultant with Altis Consulting and has experience in Dimensional Modelling and ETL (Extraction, Transformation and Load)