Apache Spark: What is RDD?

Key Takeaways

• An RDD is an immutable, partitioned collection that Spark can process in parallel.
• Transformations such as map() and filter() are lazy; actions such as count() trigger computation.
• RDDs aren't cached automatically. Persistence is explicit, and storage levels can use memory, disk, or both.
• Start with DataFrames for structured data. Use RDDs when you need lower-level control; typed Datasets are another option in Scala and Java.

An RDD in Spark, short for Resilient Distributed Dataset, is an immutable collection divided into partitions for parallel processing. RDDs remain useful for understanding Spark and implementing some low-level workloads, but DataFrames are generally the better starting point for structured data because Spark can optimize their operations.

If you're new to the engine itself, start with our introduction to Apache Spark. Here, the important distinction is that an RDD describes distributed data and how to compute it—not an automatically populated in-memory cache.

What Does Resilient Distributed Dataset Mean?

Resilient: Recover Lost Partitions Through Lineage

Spark records an RDD's lineage: the parent datasets and transformations needed to compute it. If an executor loses a partition, Spark can recompute the missing data from available parents or source data rather than requiring you to restart the entire application.

The original RDD paper explains this recovery model. It isn't a guarantee against every failure: recovery still needs the required inputs, executable transformations, and a functioning application. Keep inputs available and transformations reproducible; lineage isn't a backup of deleted source data.

Distributed: Divide Work Into Partitions

A partition is a portion of an RDD's records. During a stage of execution, a task processes one partition, and Spark can run tasks concurrently across executors—the processes that execute work and retain persisted data.

Distribution enables parallelism, not automatic balance. Too few partitions can leave resources idle, while uneven partitions can leave one task processing far more data than the others.

Dataset: A Collection, Not Necessarily a Table

An RDD can contain strings, numbers, key-value pairs, or custom objects. It doesn't require named columns or a tabular schema. The word “Dataset” in RDD's name shouldn't be confused with Spark's separate typed Dataset API.

How RDDs Work: Lazy Transformations and Actions

The RDD programming guide separates operations into two categories:

• Transformations create another RDD without immediately processing its records. map() applies a function to each element; filter() retains matching elements.
• Actions request a result or output and trigger the necessary computation. count() returns the number of elements; collect() brings every element to the driver, the process coordinating the application.

Building a transformation chain records dependencies. Spark executes the required work when an action needs its result.

Build lineage:  source RDD --filter--> filtered RDD --map--> result RDD
Request work:   result RDD.count()
Execute tasks:  partitions --> filter + map + local counts
Return result:  driver combines counts

A Runnable PySpark RDD Example

This example uses classic PySpark locally. First, install compatible Python and Java runtimes and configure JAVA_HOME according to the PySpark installation requirements. Save the following as rdd_example.py:

from pyspark import SparkConf, SparkContext, StorageLevel


def main() -> None:
    conf = SparkConf().setAppName("RDD basics").setMaster("local[2]")
    sc = SparkContext(conf=conf)
    try:
        numbers = sc.parallelize([1, 2, 3, 4, 5, 6], numSlices=2)

        # Lazy transformations: neither processes the records yet.
        squares = numbers.filter(lambda n: n % 2 == 0).map(lambda n: n * n)

        # Request persistence; an action still needs to compute the data.
        squares.persist(StorageLevel.MEMORY_AND_DISK)

        print(squares.count())              # Action: 3
        print(sorted(squares.collect()))    # Action: [4, 16, 36]
        squares.unpersist()
    finally:
        sc.stop()


if __name__ == "__main__":
    main()

Install PySpark in your Python environment and run the file:

python3 -m pip install pyspark
python3 rdd_example.py

numSlices=2 creates two partitions; local[2] runs Spark locally with two worker threads. The first action computes and persists the squares. The second can reuse retained partitions instead of repeating the transformations. These numbers are deliberately small enough to collect safely.

The Language API Affects Execution

Scala uses RDD[T]; Java exposes JavaRDD<T> and JavaPairRDD<K, V>; Python uses pyspark.RDD. Scala and Java RDD functions operate on JVM objects, while PySpark RDD functions execute in Python worker processes. Python execution and serialization between Python and the JVM add costs you should account for.

Persistence and Checkpointing Are Different Tools

Without explicit persistence, an RDD's transformations may run again for another action. persist() requests that computed partitions be retained for reuse; it doesn't compute them immediately.

• cache() uses the RDD default storage level, MEMORY_ONLY. Partitions that don't fit in memory aren't retained and may need recomputation.
• persist(StorageLevel.MEMORY_AND_DISK) allows partitions that don't fit in memory to be stored on disk.
• unpersist() releases persisted blocks when you no longer need them.

Persist an expensive intermediate result when multiple actions reuse it. Caching everything consumes resources and can be less useful than recomputing cheap transformations.

Checkpointing addresses a different problem: long lineage. After a reliable checkpoint is written, Spark removes the RDD's parent references and can recover from the checkpoint instead.

Configure a durable, cluster-accessible checkpoint directory with SparkContext.setCheckpointDir(), call rdd.checkpoint() before executing a job on that RDD, then run an action to materialize it. Persisting first can avoid recomputing the RDD just to write the checkpoint.

Partitions and Shuffles Determine the Cost of an RDD

Transformations such as map() and filter() have narrow dependencies: each output partition needs one corresponding parent partition, so Spark can pipeline the operations. A shuffle redistributes records between partitions, usually across executors, and creates a stage boundary.

Shuffles involve serialization, network traffic, and disk I/O. Keyed grouping, aggregation, and joins can require them; adding machines doesn't make those costs disappear.

For aggregation, prefer reduceByKey() over grouping all values and then summing them. It combines values locally before transferring partial results. Its reduction function must be associative and commutative, meaning regrouping or reordering inputs doesn't change the result.

# Fragment: insert inside the earlier example's try block.
pairs = sc.parallelize([("spark", 1), ("rdd", 1), ("spark", 2)], 2)
totals = pairs.reduceByKey(lambda left, right: left + right)
print(sorted(totals.collect()))  # [('rdd', 1), ('spark', 3)]

Use groupByKey() when you genuinely need each key's individual values, not merely a total. Also watch for skew—uneven data distribution, such as one key dominating the workload. The Spark tuning guide covers partition parallelism and task memory pressure.

Are RDDs Better Than Hadoop MapReduce?

Not categorically. In a pipeline of classic Hadoop MapReduce jobs, one job commonly writes its output to a filesystem for the next job to read. The Hadoop MapReduce tutorial documents this file-based output model.

Spark can pipeline compatible transformations without writing each transformation's complete result to disk. Explicit persistence also lets repeated computations reuse intermediate data, which is especially useful for iterative workloads.

That doesn't make Spark disk-free or guarantee a speedup. Inputs, outputs, shuffles, spills, and disk-backed persistence still involve I/O. For the broader comparison, read our MapReduce quick explanation and Hadoop MapReduce vs Spark.

RDD vs DataFrame vs Dataset: Which Should You Choose?

Spark SQL's structured APIs expose the data's structure and the requested operations to Spark's optimizer. An arbitrary RDD function doesn't expose the same relational information.

PySpark provides DataFrames, not the typed Dataset API available in Scala and Java. For new structured pipelines, start with a DataFrame. Choose an RDD when its lower-level control solves a concrete problem—not simply because the input arrives as text or lacks a predefined schema.

Common Mistakes

• Collecting production-sized results. collect() loads the complete result into driver memory. Keep collected results small.
• Assuming persistence guarantees residency. Missing or evicted cached partitions may still need recomputation.
• Putting irreversible side effects inside transformations. Tasks can run again during recovery. Prefer reproducible transformations and make necessary external writes safe to repeat.

FAQ

What is an RDD in Spark?

An RDD, or Resilient Distributed Dataset, is an immutable collection divided into partitions that Spark can process in parallel. It supports transformations and actions, with lineage recording how its data can be recomputed.

Are RDDs still supported in Spark?

Yes, RDDs remain available through Spark's classic APIs; DataFrames and Datasets haven't removed them. Spark Connect doesn't support RDDs.

When should I use an RDD instead of a DataFrame?

Use an RDD when you need custom low-level transformations or partition-oriented processing that the structured APIs don't express well. Prefer a DataFrame for structured filtering, joins, aggregations, and SQL-style work.

How do Spark RDDs provide fault tolerance?

Spark tracks the parent datasets and transformations needed to compute each RDD. It can recompute lost partitions when the required inputs and transformations remain available; lineage doesn't protect against every application or storage failure.

Are RDDs cached in memory by default?

No. Request persistence explicitly with cache() or persist(); an action then computes the required partitions. RDD cache() uses MEMORY_ONLY, while other storage levels can retain data on disk or in both memory and disk.

Further Reading

• Spark RDD programming guide — creation, transformations, actions, shuffles, and persistence.
• Spark SQL, DataFrames, and Datasets guide — structured execution and language support.
• PySpark installation guide — Python, Java, and package requirements.
• PySpark reduceByKey() reference — local combining and reduction requirements.
• PySpark persist() reference — explicit retention across operations.
• PySpark checkpoint() reference — checkpoint timing and lineage truncation.
• Spark Connect overview — remote execution and unsupported APIs.
• The original Resilient Distributed Datasets paper — lineage, dependencies, and the design's motivation.