What Is Big Data? Challenges and Solutions

Irish businesses believe that Big Data, analytics and cloud technologies will deliver the most value over the next two years.

EY’s Tech Horizon Report, which explores how technology and transformation can address key global challenges facing senior technology leaders, found that 50% of Irish respondents believe cloud technologies will deliver the most value to their business over the next 24 months.

In addition, 46% of respondents thought data and analytics would add the most significant value.

Real-time access to data has already transformed Irish businesses by empowering them to understand customers better and anticipate their needs.

The Bank of Ireland states that data is the currency of the digital economy and Irish businesses should be quick to capitalise on this to become more competitive and attractive to customers, employees and partners.

The report adds that SMEs may have an advantage in this area because of their size.

But what is Big Data, and why is it important?

Big Data refers to large, complex data sets.

These data sets are too voluminous for traditional data processing software to manage.

The benefit of Big Data is its ability to address business problems you might not previously have been able to tackle.

The three Vs of Big Data

Industry experts may refer to the three Vs of Big Data and these can be useful in understanding what Big Data is:

• Volume – Around 2.5 quintillion bytes of information is created each day. It is not uncommon for large companies to have Terabytes (roughly one trillion bytes) and even Petabytes (1,000 Terabytes) of data.
• Velocity – Refers to the speed data is received and acted upon. Some internet-enabled products operate in real-time or near real-time and may require real-time evaluation and action.
• Variety – Refers to the many types of data available. Once taking the shape of database files – such as Excel – data now comes in non-traditional forms, such as video, text, pdf, and graphics, and via tech, such as wearable devices. Unstructured and semi-structured, data including text, audio and video, require additional processing.

While Big Data and cloud computing are two distinctly different ideas, they are interwoven and almost inseparable. Cloud computing provides resources and services on demand, with significant Big Data stored in the cloud.

However, there are a number of Big Data issues and challenges that organisations may need to tackle to tackle advantage of the potential of Big Data.

1. There is so much data to make sense of

One of the three Vs of Big Data presents one of its biggest challenges – volume – and this is a key challenge to address when understanding what Big Data is and how it works.

Data is everywhere. Dell Technologies reported that 70% of Irish Businesses say they are gathering more data than they can use. Over half had increased the volume of data they were collecting, but the majority said they couldn’t analyse all the data they collected.

Businesses typically have data for customers’ interests, such as how many people are visiting their website, subscribing to a newsletter, active on social media channels, and financial information.

It’s a potential data goldmine, so how do you mine it?

Solutions for analysing Big Data

A good data strategy explores data collection, integration and management.

Start by looking at how you’re collecting data. For example, are the systems you use to collect customer information accurate? Such as, are there checks on your website to ensure that customers enter the correct number of mobile phone digits? Or are you checking addresses against a Postal Address Finder?

Secondly, to get the most out of your data, integrate databases. Cloud computing provides a valuable platform for storing Big Data in a single place but it is essential to consider data security.

When analysing Big Data, Artificial Intelligence (AI) is rapidly becoming as intertwined with big data as the cloud.

Many technology providers offer cloud-based AI solutions to help make sense of data.

2. Data protection and data security

Gathering and storing large amounts of data raises the stakes for security breaches, especially if data is less well organised. Organisations may have Big Data security concerns, especially related to Big Data and cyber security.

Learn more about ISO 27017 Cloud Data Protection certification used by organisations that require cloud services and by cloud service providers.

Moving Big Data to the cloud means companies often entrust Big Data or customer data to a third-party provider.

Potential security and data threats may include:

• Fake data – Collecting data from multiple sources increases the risk of collecting the wrong, incorrect or contradictory information.
• Unprotected stored data – Storing data without safeguards such as a firewall or encryption can leave it at risk of a data leak.
• Non-compliance with data privacy laws – The EU’s General Data Protection Regulation (GDPR) is the world’s strictest privacy and security law. GDPR applies if you process the personal data of EU citizens or residents, or offer goods or services to anyone resident in the EU – even if your organisation is located outside the EU. Fines for violating the GDPR are high. Companies that lack a proper strategy for big data have a higher risk of exposure from failing to gather the appropriate consent.

Learn more about how our Cyber Essentials certification can demonstrate your organisation’s commitment to cyber security, and learn more about what Cyber Essentials is and how it can help your business.

Read our guide on how to protect your business from cybercrime.

Solutions for securing Big Data

You can follow a few rules to implement data security, including knowing your data, having tighter controls over your data stores and backup, and safeguarding your network against unauthorised access.

ISO 27001 certification, for example, provides a framework to help organisations of all sizes protect their information systematically and cost-effectively.

3. Storing data in silos across an organisation

Data silos are a common problem when dealing with Big Data.

What is a data silo?

A data silo is a store of data that a single department or business unit controls and doesn’t share. Typically, siloed data sits in a standalone system and may be incompatible with other data sets, so it’s hard for others within the organisation to access and use it.

Any organisation that lacks a data management strategy can create data silos.

Negative consequences of data silos include:

• incomplete data sets
• inconsistent data
• duplicate data and processes
• less collaboration and data sharing
• data security and compliance issues

Solutions for dealing with Big Data silos

There are various means to break down data silos, including:

• Consolidate data into a single application.
• Deploy data warehouses and data lakes to create structured ways of holding large volumes of Big Data.
• Introduce a comprehensive data management strategy that aligns data management processes with business operations.
• Review your organisation’s culture and consider a data governance initiative.

Learn more about the security issues in cloud computing with our guide.

4. Your company lacks the skills needed for managing Big Data

Many organisations find recruiting the right people to organise, manage and analyse big data challenging.

Invest in Big Data training for your staff

Organisations implementing big data strategies and solutions should assess their skill requirements regularly to identify potential skill gaps. Filling gaps can be addressed by extra training/cross-training existing resources, hiring new resources, and leveraging consulting firms.

Read our cyber security checklist and essential controls for SMEs.

5. Data access control

In breaking down silos and creating a central store for your Big Data, it’s crucial to have rigorous policies around who can view and edit the data. Cloud-based systems such as Microsoft Teams, for example, are designed to enable collaboration.

How to control access to Big Data

A shift from on-premises to cloud-based solutions means it is easier to control who has access to data by implementing Identity Access Management (IAM).

Technologies include the use of multifactor authentication, including 3D authentication. It requires those accessing Big Data to prove who they are by combining something they know (such as a password) and something they are (such as a fingerprint) with something they have (a linked mobile phone number that receives a One Time Passcode).

Following relevant ISO standards is a good starting place to ensure organisations meet the best IAM practices.

Learn more about how ISO 27001 can help your business manage and secure Big Data.