Apache® Spark™ has become a vital technology for development teams looking to leverage an ultrafast in-memory data engine for big data analytics. Spark is a flexible open-source platform, letting developers write applications in Java, Scala, Python or R. With Spark, development teams can accelerate analytics applications by orders of magnitude.

The rapid growth of Spark has not been without challenges. Most organizations have relied on sprawling deployments of the Hadoop Distributed File System (HDFS), with racks of spinning disks to meet the capacity and performance demands of data-intensive applications. That is about to change, however.