It is unanimously agreed that cloud has played a major part in the digital transformation of businesses and institutions especially since the 2010s. The cloud has now taken over a major chunk of responsibilities pertaining to infrastructure, data centre, hosting, SaaS and architectural applications, platforms, networking and security functions thus freeing up the IT and business teams for more strategic tasks related to operations, customers, R&D, supply chain and others.
The focus areas during early days of the cloud journey in the 2010s included evaluating which aspects of the business to “cloudify”, examining public cloud providers while deciding on private vs public or single vs hybrid/ multi-cloud strategy and finally embarking on the cloud bandwagon through the “lift and shift” or “repurchase” approach. In mid-2019, this research paper by Gartner brought out some interesting developments: mentioning that 80% of the surveyed companies were actively pursuing a multi-cloud strategy with at least 2 or more public cloud providers and have been leveraging auto scaling and dynamic load balancing to improve performance and latency while focussing on cost optimisation as well.
The second half of the 2010s witnessed the emergence of cloud-native approach vis-à-vis the traditional lift and shift approach, utilizing the basic cloud computing to build and run these scalable applications by leveraging containers, microservices, serverless architectures, and declarative code. Gartner’s Hype Cycle here for cloud computing mentioned cloud-native as the next logical step in the cloud computing journey. Enterprise architects popularised cloud native composable architecture to now quickly develop, assemble and deploy independent, pluggable and replaceable best- in-breed systems to significantly reduced turn around times and costs, future proof investments and eliminate vendor lock-ins while providing the best choice of services and personalised offerings to customers on an ongoing basis.
This research by McKinsey estimates that the cloud has the potential to add USD 1 trillion of Economic Value across the Fortune 500 band of companies by 2030. It is hence critical to leverage automation extensively for provisioning, orchestrating, auto-scaling, monitoring and other aspects, thus improving productivity, resilience, security and enhancing cloud developers’ experience.
What’s the importance of automation in cloud computing?
Considering the rapid adoption and migration to cloud as well as rising popularity of cloud native and multi/ hybrid clouds, it is important to reduce manual efforts and improve accuracy/ eliminate errors in processes such as provisioning, configuring, sizing and auto-scaling, asset tagging, clustering and load balancing performance monitoring, deploying, performance management and others. Besides this, it is also critical to eliminate security vulnerabilities and minimise trouble shooting thus enhancing availability. Automation and orchestration are hence of paramount importance and also help enterprises in additionally adhering to compliance and governance parameters as well. It is to be remembered that automation also enables efficient and effective complementary cloud orchestration which is so essential in the dynamic and uncertain business conditions of today.
This McKinsey research, highlights that companies that have adopted end to end automation in their cloud platforms and initiatives report a 20-40% increase in speed of releasing new capabilities to market. A similar report by Deloitte, mentions that intelligent automation in the cloud enables scale in just 4-12 months, compared to the earlier 6-24 months period, through streamlined development and deployment processes.
CIOs are enabling automation for auto-provisioning the infrastructure and cloud resources, auto-scaling and shutting down unutilised instances, backups, workflow version control, and establishing Infrastructure as Code (IAC). This is hence, immensely value-adding to robust cloud native architecture by enhancing containerisation, clustering, network configuration, storage connectivity, load balancing and managing the workload lifecycle, besides highlighting vulnerabilities and risks. Enterprises pursuing hybrid or multi cloud strategies are similarly driving automation in private clouds as well as integrating with public clouds by creating automation assets that perform resource codification across all private and public clouds and offer a single API.
From a cloud developer’s perspective and especially during the excessive workloads brought about by the pandemic, cloud automation is in some ways a lifesaver by enabling rapid deployment, scaling and testing for the popular and ubiquitous agile based development methods such as DevOps and CI/ CD (Continuous Integration and Continuous Delivery/ Deployment)
What has been the impact of the pandemic on automation in cloud computing?
Even before the pandemic, emerging technologies such as artificial intelligence and machine learning, RPA, big data, extended reality, digital twins, internet of things, chatbots, and blockchain coupled with great advances in telecom network/ speeds, mobility and 5G and edge computing, helped advance enterprises towards cloud and digital transformation. The importance of data driven enterprises also ensured data life cycle management including data fabric, data lakes and data warehouses to also use cloud extensively. The pandemic accelerated these trends, especially on account of social distancing, travel restrictions, lockdowns, market uncertainties, hybrid working, and online education thus resulting in an accelerated adoption of digital transformation.
With continuing business uncertainty and pressure on IT budgets, CIOs and Leaders had turned even more towards automation to deliver elasticity especially during demand surges, shutdowns & restarts, shorter payback periods and reducing project cycles and gestation periods for their cloud initiatives. This article by Economic Times article in early 2020 highlights the importance of the Cloud Native architecture enabling companies to effectively use automation, distributed computing and Artificial Intelligence across their rationalised, distributed remote workforce. A similar research by Deloitte highlights the advantages of cloud-native platforms, which through automation have the ability for auto scaling and cost optimisation, all complying to privacy and data guidelines. Automation is an important lever behind the popularity of cloud-native and container platforms, which as stated by this Gartner research shall spur the cloud first strategy of more than 85% of the enterprises by 2025.
With more and more organisations seriously looking at the multi-cloud approach especially on account of their business drivers, operations and compliances, as well as technology stacks and IT roadmap, Hybrid Cloud Management Platforms (HCMP) and Cloud Service Brokers (CSBs) are also leveraging automation in their cross-platform tools to provide ongoing ROI and savings.
What is the cloud automation market size? Any recent examples of organisations in Asia adopting automation in cloud computing?
According to this report by Verified Market Research, the size of the cloud automation market, which was around USD 53 billion in 2021 will exceed USD 414 billion by 2030, growing at a CAGR exceeding 26% during this forecast period. 2022 has seen some clear indicators on the importance and scale of the cloud automation ecosystem, especially the Broadcom-VMWare acquisition and the awarding of a USD $10 billion NSA cloud contract to Amazon.
The rising popularity of conversational AI is based on the backbone of automation on the cloud besides the Data Scientists AI/ ML platform with their prebuilt ML models. This Deloitte research attributes the cloud’s elasticity, computational capability and availability of storage tasks to be a great enabler of easier and faster execution of the conversational AI end to end task cycle also known as Machine Learning Operations (ML Ops).
Cloud automation has helped several leading banks, insurance technology start-ups and many organisations in Singapore, Malaysia, Philippines and elsewhere across Asia accelerate their digital transformation initiatives immensely during the pandemic and throughout 2022.
From the internal organisational perspective, it is predicted that automation will fuel the next wave of platform-as-a-service cloud native solutions which had already been popularised on account of remote working. In India, penetration of PaaS services such as automated cloud orchestration and serverless computing will have dependence on evolution of automation and productivity tools, as this EY article indicates.
What are the challenges in adopting automation in cloud computing? What are some futuristic trends?
In the midst of the plethora of cloud automation tools and platforms, skillsets continue to remain a challenge. This Gartner research highlights that almost 50% of the organisations will face skill gaps in their hybrid and multi-cloud infrastructure initiatives by 2025. Investments in tools and technologies haven’t quite matched the corresponding resources- across developers, architects and solution designer skill sets, leading to gaps in skills and talent pool.
Along with Automation and IAC, it is also imperative to apply cybersecurity principles correspondingly. This McKinsey article highlights the importance of Security as Code (SaC) and the fact that it has dependencies on architecture and the right automation capabilities. And this is increasing in importance especially due to even more stringent privacy, compliance, regulatory and governance guidelines.
In this article, Gartner predicts that by 2025, more than 50% of cloud data centres shall deploy robots with AI capabilities especially across monitoring, server maintenance and upgrade, security, and cloud operations processes and thus automate repetitive tasks, minimise errors and free up manpower for more value-added tasks.
How has automation contributed to data storage?
According to this IDC report, in 2020, over 64 zettabytes of data was created/ replicated across the globe. During the period 2020-2025, it is envisaged that the global data creation and replication function shall grow at a CAGR of 23% during the period 2020-2025 and this growth shall be greater than twice the amount of data created since the invention of digital storage!
On the other hand, the global storage capacity was 6.7 zettabytes in 2020, and the growth of storage shall be less than that of data creation/ replication. CIOs and business leaders are expected to store more and more of data to leverage data analytics and business insights for resilience, launching new products and services as well as monitor employee, customer and supply chain engagement, satisfaction and reputation indices. Furthermore, cloud is playing an increasing role in storage as well as consumption of data, besides the edge, social media and IoT devices.
With the explosion of data in volume, variety and velocity across so many diverse sources, including but not limited data centres, cloud, on-premise applications and infrastructure, IoT, edge, social media and others, it is of paramount importance to manage, monitor and leverage corresponding storage. Automation hence, contributes in eliminating manual and tedious structured, unstructured and archival data storage tasks and resulting errors, streamlines provisioning and allocation of requisite storage, provides faster and optimised deployment across storage devices and ensures an end-to-end holistic visibility, control, insights and monitoring of data storage in a cost-effective manner.
Not dissimilar to cloud automation, data storage automation and orchestration also ensure prioritisation of processes, tasks and resources to balance speed, efficiency, usage and cost. Especially tasks such as provisioning, capacity management, workflows and data migration, resource optimisation basis retrieval demand, and data protection and disaster recovery policies. Forecasting of capacities and associated bandwidth, configuration management and software updates/ upgrades can also be automated for immense benefits.
What are the trends of data storage automation in 2022 and beyond?
CIOs are leveraging intelligent automation in both network-attached storage (NAS) and storage area network (SAN) solutions for the above-mentioned benefits, as well as automated data lifecycle management, cloud deployment, workload consolidation, hybrid storage and integration with Artificial Intelligence systems. IT personnel across enterprises and SMBs are deploying diverse storage architectures to make use of the cloud, besides the regular hardware and software defined storage systems for data and objects.
Increasing application of AI analytics in the data storage automation function provides forecasts on capacity requirements, performances, bottlenecks and other KPIs in the future, with the view of always optimising and minimising requirements of human resources for this function. AI is becoming even more important especially for fine tuning data storage across hybrid and multi-cloud environments.
Storage as a Service (STaaS) is growing in popularity and managed services around STaaS is expected to replace more than 40% of all on premises IT storage administration and support costs by 2025, up from 10% in 2021, as envisaged by Gartner here.
Whilst the benefits and use cases of automation across customer and supply chain businesses and functions are well known and expanding, automation plays an equally key area in the back-end infrastructure and operations, especially around cloud computing and storage. Organisations leveraging automation intelligently balancing market conditions, dynamics, IT roadmaps, technology and skill sets shall have significant competitive advantages.