Maintaining data accurately

What are the best ways to store and organise your data?

You should store your data in a way that creates an efficient data networking and reporting system, enabling you to generate knowledge from the facts, figures, and statistics that you collect.

Using a master data management tool lets you consolidate information from multiple systems to create records that give you a complete and accurate view of your business.

Data – the backbone of civilisation

From humble beginnings on stone tablets to documents hand printed over and over in the Middle Ages, our societies have grown reliant on data. Brian Clegg called data "the backbone of our civilization." While we continue to amass data, we have been limited in our capacity to deal with it, analyse it, and make sense from it.

So many of our businesses are yet to fully see the complete picture that rich, combined and integrated data can paint for us.

As systems evolve and can combine and compare data at an unprecedented rate, we can only hope to harness the full power of data. For data processing to work properly, it needs access to data sets that are complete, clean and consistent. Yet, although they have systems to enable them to process data, many businesses are finding that the stored data they have is limiting them and letting them down.

The knowledge pyramid

Data at the base of the knowledge pyramid, with information sitting above it and the smallest portion being knowledge at the top. It is from data; the raw details, that we get information. From the organisation of data we can make knowledge, truisms we glean from the information.

The data could be sales-related; the raw facts and figures related to how many products were sold, to whom, and how much they spent.

Sitting above it, in a more refined, the narrower band is the information. Organising data and working with it gives us information. An example of turning data into information would be putting sales figures and times into a table. If you studied this table and found that people who shop after 7pm spend more than those who shop during the day, you will have used data and information to to form knowledge.

However, if the baseline data is not consistent, if we are missing a 'time of sale' record and in some records and the total spend amount in other data records, then it is simply too hard to turn data into information. The correlation between indicators becomes difficult to process. Knowledge can propel you forward, open doors and create possibilities.

Why data matters today

For many companies, data is the top priority and requires the most significant investment of any asset. As data systems become more complex, the amount of data that is being generated is rapidly increasing. Data offers great insight into opportunities and pathways for survival in today's tough economic climate.

Ensuring accurate data is an essential part of the digital transformation that so many enterprises are going through. With new technologies like machine learning, sophisticated analytics and market segmentation, artificial intelligence, and a high demand for personalisation, data is the tool that will help enterprises survive and thrive.

How to ensure accuracy of data collection

If you are using inaccurate data to make business decisions, there is a real risk you are compromising your earning potential. Having high-quality data gives you consistent information about how you are performing at any given moment and what trends and opportunities for the future might be.

Up to date and accurate data can inform your decision-making and improve your relationships with your customers, with precise insights into their thoughts and preferences. Accurate data also enables you to manage products, services and distribution effectively.

There are always risks involved with the acquisition and use of data; no data set is 100% correct. To reduce the risk of data coming into your systems with errors, you can use automation where possible to automatically add data consistently.

How to improve data quality

There are also ways of checking and monitoring data once it is entered into your system, and steps you can take at any time to improve data quality. Consider:

Possession and control - who has access to what data by defining use and management. If more than one system is used , this also means creating hierarchy to ensure that truth can be maintained from within a master source
Integrity - which means information should be created and stored with its intended use in mind
Authenticity - ensuring that the data is genuine and cannot be or has not been manipulated
Availability - providing access to data in a timely way, when it is required for use
Utility - how easy the data is to use, interpret and construct by the individual user

Importance of data accuracy

To ensure your data is effective and can be used for the best benefit of users and the business, you should ensure it is maintained, secured, and useable as an integrated data set. To maintain accuracy, currency and usability, records must be collected once and according to an agreed standard and reused to meet requirements.

Consistency is enabled through data classification standards, schemes, and definitions. Where base data is held in a different system, connectivity is enabled through a hierarchy that identifies which data should be considered the master or referenced set.

How to maintain data accurately

To maintain data accurately, you need to engage the people accessing and using the data to be involved in curating and caring for the data. Establishing a data quality program will send the standard for how you expect data to be maintained through clear processes, procedures, data naming, storing, and organisation conventions.

Data Leadership

There is also a relationship between data quality and leadership. Ensure a high degree of data integrity is a core competency for a leader, who should value the careful acquisition, storage, and use of data. Leaders need to take the lead in expressing a commitment to quality data practices, leading by example and ensuring they have suitable baseline knowledge of what data is used for around the organisation and why.

This is important for increasing a meaningful approach in line with the corporate vision. Strategic data leadership must be supported and underpinned with tools and frameworks to enable all data users to participate in best practice data use.

Empowering others to use data well

In the data context, empowering employees means equipping them with the knowledge and skills they need to make decisions based on data and delegated responsibility to take care of data systems and structures.

You can enable empowerment through data through four stages:

Meaning - when employees understand what data is being acquired, what it will be used for and why it is important to the overall process and practices of the business
Competence - when employees know what is expected, know the data rules that have been established and feel confident in their capacity to handle the data within the structure and systems
Self-determination - when an employee feels they have choices in relation to their actions or how to address a particular issue or problem that they have noticed
Impact - when an employee feels that their contributions make a difference and could validly have an effect on what the business experiences
To ensure data is used to its best benefit, captured, managed and transferred consistently. This consistency will enable it to be deployed seamlessly wherever it is needed, either as records and information available to a customer (such as prices, product availability or personalisation options) or internally (such as in reports and documents related to sales, trends, stock control). Standards should be established to govern how data is held and maintained.

Don't go overboard on data

You should maintain data to the standards required by the business for use. It doesn't need to be maintained or created at a higher standard than is useful for the business.

Although there is a tension between trying to work out what data might be needed or used in the future, there is also a real risk of adding unnecessary expense or time delay when records are created in more detail than what is useful.

Data standards should be meaningful and practical and not create unnecessary work. Data should not be over-processed and handled if it is not warranted for the business or customer needs.

Master data management

When you have data spread across multiple systems, a master data system (MDM) can help you define and improve data quality and assess the data for overall completeness and accuracy. Systems like Pimcore, which has an extensive MDM module, allow you to:

Use data validation techniques
Run data quality reports
Monitor data completeness
Connect business intelligence applications for more detailed analysis
Filter and exporting to give you customised insights

To maintain data accurately, you need both systems and people on your side. You will need to show leadership, clearly state expectations, and create a culture that values data. You will also need a robust data management system that can check with much greater reliability and with a sophisticated level of detail just how your data is performing.