SPARK SQL throws AssertionError: assertion failed: Found duplicate rewrite attributes (Spark 3.0.2)

2k Views Asked by At

Executing the above in Spark 3.0.2 produces Exception in thread "main" java.lang.AssertionError: assertion failed: Found duplicate rewrite attributes. It was working in Spark 2.4.3.

SELECT 
    COALESCE(view_1_alias.name, view_2.name) AS name, 
    COALESCE(view_1_alias.id, view_2.id) AS id, 
    COALESCE(view_1_alias.second_id, view_2.second_id) AS second_id, 
    COALESCE(view_1_alias.local_timestamp, view_2.local_timestamp) AS local_timestamp, 
    COALESCE(view_1_alias.utc_timestamp, view_2.utc_timestamp) AS utc_timestamp, 
    view_1_alias.alias_1_column_1, 
    view_1_alias.alias_1_column_2, 
    view_1_alias.alias_1_column_3, 
    view_1_alias.alias_1_column_4, 
    view_1_alias.alias_1_column_5, 
    view_1_alias.alias_1_column_6, 
    view_1_alias.alias_1_column_7, 
    view_2.alias_2_coumn_1 
FROM 
    view_1 view_1_alias FULL OUTER JOIN view_2 
ON 
    view_1_alias.name = view_2.name AND 
    view_1_alias.id = view_2.id AND 
    view_1_alias.second_id = view_2.second_id AND
    view_1_alias.local_timestamp = view_2.local_timestamp;

The header for view_1:

| name |id|second_id|local_timestamp|utc_timestamp|alias_1_column_1|alias_1_column_2 | alias_1_column_3|alias_1_column_4|alias_1_column_5|alias_1_column_6|alias_1_column_7 |

The header for the view_2:

| name | id |second_id |local_timestamp |utc_timestamp|alias_2_coumn_1|

Intrestingly we have several of these queries, but only one specifically gives issues. I saw this question but not sure how to translate it to spark sql.

0

There are 0 best solutions below