File and Object Storage

Meeting the Storage Demands of Artificial Intelligence and Deep Learning

By Jacklyn Allgayer posted Mon August 13, 2018 11:00 PM

Decisions about storage are a key element to the success of any artificial intelligence (AI) or deep learning (DL) project, as demands on storage keep growing. For the past few years there has been a huge growth of data – the “Big Data” phenomenon. Now, there are new requirements to support artificial intelligence applications, notably in the field of deep learning. Storage administrators need to plan for, and be ready to support, these applications.

Is Artificial Intelligence for Real This Time Around?

If you have been around the computer industry for a while, or have an appreciation of its history, you will know that the current excitement around artificial intelligence, machine learning and deep learning is not a unique occurrence. There have been many periods of intense interest in artificial intelligence, dating as far back as the 1950s, interspersed with what have been called “AI winters” when expectations have not been met, hopes have been dashed, and pessimism has prevailed. Is today’s resurgence any different?

There is good reason to believe that things are different today and that there is a real future for artificial intelligence as a strategic imperative for organizations. Four main factors are combining to create a “perfect storm” for AI:

• Faster processors – GPUs provide the kind of firepower needed to process the data.
• Algorithmic improvements – in recent years a lot of work has been put into understanding and improving algorithms, notably building on early work in neural networks to develop the sophisticated and powerful deep learning algorithms that are available today.
• Data growth – vast stores of data offer a significantly better basis for analysis and action than has been seen in the past. Deep learning is ideally suited to mine this treasure trove of data. Conversely, you need a large pool of data to train a deep learning system and training data is available today in volumes never seen before.
• AI is mainstream – businesses are now seeing a real competitive advantage from deploying AI. It is no longer an esoteric and academic pursuit, and real-world results are emerging, further driving adoption.

Optimizing Storage for Deep Learning

So it is reasonable to think this time it is for real and will provide competitive advantage to businesses that adopt AI and DL. What, then, are the implications for data storage and management?

There are multiple stages in deploying a deep learning system. They can be summarized as:
• Data acquisition
• Data preparation
• Model training
• Deployment

The demands on data vary across these stages. At one end of the spectrum, data acquisition and preparation, there are huge volumes of data, so the underlying data infrastructure needs the capacity to handle them efficiently and effectively. Conversely, when a system is deployed, there is a premium on speedy results – for example, time is of the essence in detection of a potentially fraudulent financial transaction in order to minimize potential losses, so the underlying storage needs to meet stringent performance goals.

IBM Elastic Storage Server (ESS) is a platform that can meet both the capacity and performance requirements for deep learning applications. For example, ESS provides the storage for the world’s smartest and most powerful supercomputer, the Department of Energy Summit system, which places the ultimate demand for capacity coupled with performance. For more modest applications, ESS comes in a range of models tailored to address customers’ storage requirements. This includes recently introduced hybrid models that offer both SSD and hard drive storage within a single chassis, offering a blend of performance and capacity. The IBM Systems AI Infrastructure Reference Architecture shows how ESS can provide the storage layer for an organization’s artificial intelligence applications.

Another recent development that maintains the ESS track record of continuous improvement to meet and exceed customers’ expectation is the introduction of an improved and easier process for buying and installing ESS protocol nodes. And continuing ESS leadership in big data analytics, ESS has extended its partnership with Hortonworks with certification for Hortonworks Data Platform (HDP) version 3.0.

Across the entire portfolio, IBM is focused on providing industry-leading storage innovation to enable more efficient and agile multicloud solutions for our worldwide customer and partner base. Learn more about how your IT team can leverage the new multicloud capabilities to reduce costs, improve agility, and accelerate the success of your enterprise at our announcement webinar or at VMworld, booth 1312.