Spark RDD map()

In this Spark Tutorial, we shall learn to map one RDD to another. Mapping is transforming each RDD element using a function and returning a new RDD. Simple example would be calculating logarithmic value of each RDD element (RDD<Integer>) and creating a new RDD with the returned elements.

Spark RDD Map
  • Syntax
  • Java Examples
  • Python Examples

Syntax

RDD.map(<function>)

where<function>  is the transformation function for each of the element of source RDD.

Examples

Java Example 1 – Spark RDD Map Example

In this example, we will an RDD with some integers. We shall then call map() function on this RDD to map integer items to their logarithmic values The item in RDD is of type Integer, and the output for each item would be Double. So we are mapping an RDD<Integer> to RDD<Double>.

RDDmapExample2.java 

import java.util.Arrays;  import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext;  public class RDDmapExample2 {  	public static void main(String[] args) { 		// configure spark 		SparkConf sparkConf = new SparkConf().setAppName("Read Text to RDD") 										.setMaster("local[2]").set("spark.executor.memory","2g"); 		// start a spark context 		JavaSparkContext sc = new JavaSparkContext(sparkConf); 		 		// initialize an integer RDD 		JavaRDD<Integer> numbers = sc.parallelize(Arrays.asList(14,21,88,99,455)); 		 		// map each line to number of words in the line 		JavaRDD<Double> log_values = numbers.map(x -> Math.log(x));  		 		// collect RDD for printing 		for(double value:log_values.collect()){ 			System.out.println(value); 		} 	} }

Run this Spark Application and you would get the following output in the console.

17/11/28 16:31:11 INFO DAGScheduler: ResultStage 0 (collect at RDDmapExample2.java:23) finished in 0.373 s 17/11/28 16:31:11 INFO DAGScheduler: Job 0 finished: collect at RDDmapExample2.java:23, took 1.067919 s 2.6390573296152584 3.044522437723423 4.477336814478207 4.59511985013459 6.12029741895095 17/11/28 16:31:11 INFO SparkContext: Invoking stop() from shutdown hook

Java Example 2 – Spark RDD.map()

In this example, we will map an RDD of Strings to an RDD of Integers with each element in the mapped RDD representing the number of words in the input RDD. The final mapping would be RDD<String> -> RDD<Integer>.

RDDmapExample.java 

import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext;  public class RDDmapExample {  	public static void main(String[] args) { 		// configure spark 		SparkConf sparkConf = new SparkConf().setAppName("Read Text to RDD") 										.setMaster("local[2]").set("spark.executor.memory","2g"); 		// start a spark context 		JavaSparkContext sc = new JavaSparkContext(sparkConf); 		 		// provide path to input text file 		String path = "data/rdd/input/sample.txt"; 		 		// read text file to RDD 		JavaRDD<String> lines = sc.textFile(path); 		 		// map each line to number of words in the line 		JavaRDD<Integer> n_words = lines.map(x -> x.split(" ").length);  		 		// collect RDD for printing 		for(int n:n_words.collect()){ 			System.out.println(n); 		} 	} }

Following is the input text file we used.

data/rdd/input/sample.txt 

Welcome to TutorialKart Learn Apache Spark Learn to work with RDD

Run the above Java Example, and you would get the following output in console.

17/11/28 16:25:22 INFO DAGScheduler: ResultStage 0 (collect at RDDmapExample.java:24) finished in 0.568 s 17/11/28 16:25:22 INFO DAGScheduler: Job 0 finished: collect at RDDmapExample.java:24, took 0.852748 s 3 3 5 17/11/28 16:25:22 INFO SparkContext: Invoking stop() from shutdown hook

We have successfully created a new RDD with strings transformed to number of words in it.

Python Example 1 – Spark RDD.map()

In this example, we will map integers in RDD to their logarithmic values using Python.

spark-rdd-map-example-2.py 

import sys, math  from pyspark import SparkContext, SparkConf  if __name__ == "__main__":    # create Spark context with Spark configuration   conf = SparkConf().setAppName("Map Numbers to their Log Values - Python")   sc = SparkContext(conf=conf)    # read input text file to RDD   numbers = sc.parallelize([14,21,88,99,455])    # map lines to n_words   log_values = numbers.map(lambda n : math.log10(n))    # collect the RDD to a list   llist = log_values.collect()    # print the list   for line in llist:     print line

Run the following command to submit this Python program to run as Spark Application.

$ spark-submit spark-rdd-map-example-2.py

Following is the output of this Python Application in console.

17/11/28 19:40:42 INFO DAGScheduler: ResultStage 0 (collect at /home/arjun/workspace/spark/spark-rdd-map-example-2.py:18) finished in 1.253 s 17/11/28 19:40:42 INFO DAGScheduler: Job 0 finished: collect at /home/arjun/workspace/spark/spark-rdd-map-example-2.py:18, took 1.945158 s 1.14612803568 1.3.6.0929473 1.94448267215 1.9956351946 2.65801139666 17/11/28 19:40:42 INFO SparkContext: Invoking stop() from shutdown hook

Python Example 2 – Spark RDD.map()

In this example, we will map sentences to number of words in the sentence.

spark-rdd-map-example.py 

import sys  from pyspark import SparkContext, SparkConf  if __name__ == "__main__":    # create Spark context with Spark configuration   conf = SparkConf().setAppName("Read Text to RDD - Python")   sc = SparkContext(conf=conf)    # read input text file to RDD   lines = sc.textFile("/home/arjun/workspace/spark/sample.txt")    # map lines to n_words   n_words = lines.map(lambda line : len(line.split()))    # collect the RDD to a list   llist = n_words.collect()    # print the list   for line in llist:     print line

Run the above python program using following spark-submit command.

$ spark-submit spark-rdd-map-example.py
17/11/28 19:31:44 INFO DAGScheduler: ResultStage 0 (collect at /home/arjun/workspace/spark/spark-rdd-map-example.py:18) finished in 0.716 s 17/11/28 19:31:44 INFO DAGScheduler: Job 0 finished: collect at /home/arjun/workspace/spark/spark-rdd-map-example.py:18, took 0.866953 s 3 3 5 17/11/28 19:31:44 INFO SparkContext: Invoking stop() from shutdown hook

Conclusion

In this Spark Tutorial, we learned the syntax and examples for RDD.map() method.