Since the Washington and Jefferson have null or empty values in array and map, the following snippet out does not contain these. This will ignore elements that have null or empty.

And when the input column is a map, posexplode function creates 3 columns “pos” to hold the position of the map element, “key” and “value” columns. Posexplode(e: Column) creates a row for each element in the array and creates two columns “pos’ to hold the position of the array element and the ‘col’ to hold the actual array value. Posexplode – explode array or map elements to rows from the above example, Washington and Jefferson have null or empty values in array and map, hence the following snippet out does not contain these rows.ĭf.select($'name',explode_outer($'properties')) When a map is passed, it creates two new columns one for key and one for value and each element in map split into the row. When an array is passed to this function, it creates a new default column “col1” and it contains all array elements. Spark function explode(e: Column) is used to explode or create array or map columns to rows. Val df = spark.createDataFrame((arrayData),arraySchema)Įxplode – spark explode array or map column to rows

add('properties', MapType(StringType,StringType))

add('knownLanguages', ArrayType(StringType))

Use intellij key map for python