The Migration of Big Data Processing To The Cloud
Empiric Empiric

The Migration of Big Data Processing To The Cloud

Published 10/09/2018

The Migration of Big Data Processing To The Cloud

The question of whether to allocate data storage and processing to owned and operated physical servers or to the Cloud might seem like a technical one – however, Cloud computing has the potential not just to cut costs but to revolutionise a business’s operations. Indeed, Cloud is arguably the central building block around which digital transformation is built – and when polled by SolarWinds, 95% of respondents named cloud and hybrid IT as being among the most critical technologies going for their business’s digital strategy.

Meanwhile, more and more organisations are exploring the opportunities offered by Big Data – and data processing and storage are increasingly moving to the Cloud (with data storage being the most common application of Public Cloud, according to CAPSiDE’s 2018 survey on The Maturity Of Public Cloud Adoption).

Better, Faster, Cheaper

First and foremost among the benefits are the flexibility that Cloud affords. The companies of the first dot-com boom had to run their own physical servers and maintain capacity to serve peak demand which was rarely actually required. Cloud computing, on the other hand, allows businesses to minimise their cost structure and to scale on demand – rapidly increasing server capacity and processing resources, as required. At the same time as being vastly more flexible, Cloud also presents the opportunity to significantly simplify infrastructure requirements.

“The great thing about Cloud applications is always that you only pay for what resources you use and when you use them,” says Empiric’s Big Data expert, Sebastian Rodriguez. “Running resource-heavy applications (with most Big Data applications being resource intensive) requires a big setup. Deploying real time Big Data applications on the cloud or running analytics on cloud resources means that companies that are not using these resources constantly can spin them up when they need to, run the processes they need and then shut them down.”

Alex Moore, Delivery Director UK & Tech Evangelist for CAPSiDE, also highlights Cloud’s capacity to incorporate automation tooling into infrastructure, working “in a systematic, reproducible way” and thereby saving time and effort.

There is also the advantage of connecting systems together. “Once you’ve got that large data platform, you want to be able to exploit the data at scale. While that might be running some kind of analytics or reporting, or whatever action it needs to be, it might also flow through any number of other systems,” says Alex. “By having them available in the Cloud environment you can naturally connect it into some of those wider ecosystems... Compared to legacy infrastructure, the flexibility, the scale, the immediacy makes it a no-brainer, I think.”

“When companies are going to Cloud they need to consider costs for certain – although applications should be cheaper – they need to be well written so they don’t use too many resources to return the desired results,” notes Sebastian. “Costs can quickly rise if applications are not well written.”

There are also an increasing range of Infrastructure as a Service and Platform as a Service options available – with Microsoft and Google both tackling Amazon’s status as industry incumbents.

“Microsoft and Google have tried to combat AWS’s dominance by offering their cloud services fairly cheaply or, in the case of Microsoft, for almost free as a bolt on to their enterprise software,” says Sebastian. “AWS still offers the largest range of services, with nearly 100 across compute, storage, database, analytics, networking, mobile, developer tools, management tools, IoT, security and enterprise applications. And it has been around the longest.”

There’s a good case to be made, though, that while AWS is highly competitive on pricing, Microsoft Azure may be a good bet for organisations that are highly invested in the Microsoft ecosystem.

Of course, there are also many other providers. “[Things open up] when you start looking beyond the top three,” notes Alex. “You’ve got people like IBM Cloud and Oracle Cloud and potentially Alibaba – and if you’re doing anything in China on Cloud, you’ll probably end up moving towards Alibaba.”


There are also other considerations, such as the fact that organisations may need to maintain some physical servers (and hence pursue a hybrid cloud structure) or may need multiple cloud installations (also known as multi-cloud).

“Businesses need to use the right cloud resource, which has the capability to integrate with their in-house systems,” says Sebastian. “In terms of security – encryption is a massive deal these days for most clients – data needs to be encrypted before it goes into a data warehouse and if you have built it already without encryption, it is extremely complex and expensive to encrypt it after.”

And security considerations are a common cause of apprehension among those wary of moving to the cloud (with 68% of respondents to CAPSiDE’s 2018 survey citing fears about security issues, making it the most reported concern). After all, migration means putting data and processing in the hands of a third-party organisation and, of course, making it remotely accessible. In a survey of over 1,900 cyber security professionals by Crowd Research Partners, the three top security challenges were identified as being “protecting against data loss and leakage (67%), threats to data privacy (61%), and breaches of confidentiality (53%)”. However, the biggest threat to cloud security is in fact the misconfiguration of cloud platforms (62%).

Most platforms are able to track processes at an extremely granular level, and to subsequently flag anomalies, however, if the system is set up incorrectly or if security has simply been ported over from a physical server infrastructure (via “lift and shift”), then critical vulnerabilities may remain in place.

“It can be quite easy to try and carry over a traditional data centre or hosting mentality into a cloud environment,” says Alex. “In general that’s not the best approach. It needs a new set of eyes, and I would take time to get some training and education around the platform directly or engage a third party.”

And that’s particularly true when it comes to managing automation, he notes. “It will take time to realise the value those platforms are going to provide you, rather than just treating it like a data centre that just happens to be run somewhere else,” he adds. “They are completely different.”

“What’s happened in some cases is that the Cloud is still seen as this new thing and, for many organisations, it’s left at arm’s length,” says Alex. “There is a whole development team messing around with the Cloud in the corner of the business, but actually they’re having great success with it and they’re building business-critical systems there, but they haven’t necessarily looked at governance.”

Everything is Connected

Taking in the bigger picture, emerging technologies are becoming ever more interconnected. Analytics and IoT devices amass previously unprecedented data sets; Big Data is assessed by AI to analyse the real world and provide market forecasts; and smart devices then relay information and recommendations back to users, while IoT machinery sets industrial processes in motion. Meanwhile, all of this is held together by communication with the Cloud.

This change is perhaps only natural given growing acceptance of DevOps working practices and the agreement that processes and units cannot function in isolation – but that change must be coordinated across the organisation.

About Empiric

Empiric is a multi-award winning business and one of the fastest growing technology and transformation recruitment agency's specialising in data, digital, cloud and security. We supply technology and change recruitment services to businesses looking for both contract and permanent professionals.

Read more (pdf download)

Empiric are committed to changing the gender and diversity imbalance within the technology sector. In addition to Next Tech Girls we proactively target skilled professionals from minority groups which in turn can help you meet your own diversity commitments. Our active investment within the tech community allows us to engage with specific talent pools and deliver a short list of relevant and diverse candidates.

For more information contact 02036757777

To view our latest job opportunities click here.


Login to your Empiric account.

Forgot password?


Don't have an account yet?

Create an account now and get access to our online features.


This website uses cookies to ensure you get the best experience on our website