SparkVersion¶

Overview¶

The SparkVersion expression returns the version of the currently running Apache Spark instance. This is a leaf expression that provides runtime information about the Spark version being used, which can be useful for compatibility checks and debugging purposes.

Syntax¶

SELECT version();

import org.apache.spark.sql.functions._
df.select(expr("version()"))

Arguments¶

Argument	Type	Description
None	-	This expression takes no arguments

Return Type¶

Returns StringType - a string representation of the Spark version.

Supported Data Types¶

This expression does not accept input data types as it is a parameterless leaf expression that returns static information.

Algorithm¶

Extends LeafExpression meaning it has no child expressions to evaluate
Implements RuntimeReplaceable, so the actual implementation is delegated to another expression
Uses StaticInvoke to call ExpressionImplUtils.getSparkVersion() method at runtime
The replacement expression is created lazily and calls the static method with returnNullable = false
Extends DefaultStringProducingExpression for consistent string output formatting

Partitioning Behavior¶

This expression has no impact on partitioning behavior:

Preserves existing partitioning since it's a constant value across all partitions
Does not require shuffle operations
Can be evaluated independently on each partition

Edge Cases¶

Never returns null values (returnNullable = false)
Always returns the same constant string value within a single Spark session
No special input validation required since it accepts no parameters
Thread-safe as it returns static version information

Code Generation¶

This expression supports code generation through the StaticInvoke replacement mechanism. The StaticInvoke expression generates optimized code that directly calls the static method ExpressionImplUtils.getSparkVersion() rather than using interpreted evaluation.

Examples¶

-- Get the current Spark version
SELECT version();
-- Output: "3.1.0 a6d6ea3efedbad14d99c24143834cd4e2e52fb40"

-- Use in a query for debugging
SELECT version() as spark_version, count(*) as row_count FROM my_table;

// Example DataFrame API usage
import org.apache.spark.sql.functions._

val df = spark.range(1)
df.select(expr("version()").as("spark_version")).show()

// Using in a more complex query
val result = df.select(
  col("id"),
  expr("version()").as("spark_version")
)