RangeExec¶

Overview¶

RangeExec is a leaf physical operator that generates a sequence of 64-bit long integers within a specified range. It produces rows containing sequential numbers from a start value to an end value with a configurable step size, distributing the generation across multiple partitions for parallel processing.

When Used¶

The query planner chooses RangeExec when executing SQL range() function calls or DataFrame range operations. It's selected when the logical plan contains a Range node that specifies start, end, step, and optional partition count parameters.

Input Requirements¶

Expected input partitioning: None (leaf operator)
Expected input ordering: None (leaf operator)
Number of children: 0 (leaf node)

Output Properties¶

Output partitioning:
SinglePartition when numSlices == 1
RangePartitioning with natural ordering when numSlices > 1
UnknownPartitioning(0) for empty ranges
Output ordering: Natural ascending/descending order based on step sign
Output schema: Single column of LongType representing the generated range values

Algorithm¶

Calculate if the range is empty using start == end || (start < end ^ 0 < step)
Distribute range elements across numSlices partitions based on session.leafNodeDefaultParallelism
Each partition calculates its start/end boundaries using BigInt arithmetic to handle overflow
Generate numbers in batches of 1000 elements for better memory management and metrics updating
Use iterator-based approach in doExecute() with overflow detection via step direction checking
Support both whole-stage code generation and interpreted execution paths
Handle Long.MAX_VALUE/Long.MIN_VALUE boundaries using getSafeMargin() function

Memory Usage¶

Does not spill to disk (generates values on-demand)
Minimal memory requirements - only stores current batch state and iterator position
Uses UnsafeRow format with pre-calculated row size for memory efficiency
Batching limits memory usage to ~1000 elements worth of state per partition

Partitioning Behavior¶

Creates exactly numSlices partitions (defaults to spark.sql.leafNodeDefaultParallelism)
No shuffle required as data is generated locally in each partition
Each partition generates a contiguous sub-range of the total range
Partition boundaries calculated using (i * numElements) / numSlices formula

Supported Join/Aggregation Types¶

Not applicable - this is a data generation operator, not a join or aggregation operator.

Metrics¶

numOutputRows: Total number of rows generated across all partitions
Also increments standard input metrics (inputMetrics.incRecordsRead) for consistency with other operators

Code Generation¶

Yes, implements CodegenSupport interface with doProduce() method. Generates optimized Java code with: - Inline mutable state for task context and metrics - Batched processing with configurable batch size (1000) - Overflow-safe BigInteger calculations for partition boundaries - Optimized inner loops with optional stop checks

Configuration Options¶

spark.sql.leafNodeDefaultParallelism: Controls default number of partitions when not explicitly specified
Batch size is hard-coded to 1000 elements but could be made configurable
Task interruption checking controlled by parent operator's needStopCheck property

Edge Cases¶

Empty ranges (start == end or invalid step direction) return EmptyRDD
Long overflow handled using BigInteger arithmetic and getSafeMargin() clamping
Step direction validation prevents infinite loops
Partition boundary calculations handle edge cases where ranges don't divide evenly
TaskContext interruption support prevents runaway tasks

Examples¶

== Physical Plan ==
*(1) Range (1, 1000000, step=1, splits=4)
   +- InMemoryTableScan [id#0L]

-- Generated by query:
SELECT * FROM range(1, 1000000, 1, 4)