Question

How can i add use a SparkDFDataset to my Great Expectations' validator?

score 191 · Answer 1 · 2023-11-22T11:07:22.090000

191

Views

How can i add use a SparkDFDataset to my Great Expectations' validator?

Published on 22 November 2023 at 11:07

score 81 · Answer 2 · 2023-11-17T14:00:00.157000

81

Views

Multiple CSV Scan in spark structured streaming for same file in s3

Published on 17 November 2023 at 14:00

score 68 · Answer 3 · 2022-03-12T11:38:11.090000

68

Views

How force spark to move record nonnull fields as _corrupt_record?

Published on 12 March 2022 at 11:38

score 1.3k · Answer 4 · 2022-05-12T07:51:45.170000

1.3k

Views

How to get week of month in Spark 3.0+?

Published on 12 May 2022 at 07:51

score 21.5k · Answer 5 · 2017-05-15T12:34:36.183000

21.5k

Views

Does Spark support the WITH clause like SQL?

Published on 15 May 2017 at 12:34

score 97 · Answer 6 · 2023-07-09T08:01:24.150000

97

Views

Spark 3.3.1 picking up current date automatically in data frame if date is missing from given timestamp and not marking it as _corrupt record

Published on 09 July 2023 at 08:01

score 4.3k · Answer 7 · 2021-11-30T02:19:51.330000

4.3k

Views

ServiceConfigurationError running spark 3.2

Published on 30 November 2021 at 02:19

score 323 · Answer 8 · 2021-12-01T17:34:18.870000

323

Views

Apache Phoenix - Count query returns more than 100k rows, but SELECT query does not return any row

Published on 01 December 2021 at 17:34

score 339 · Answer 9 · 2023-02-01T20:59:53.390000

339

Views

Configure Spark 3 thrift server with Apache Ranger

Published on 01 February 2023 at 20:59

score 1.9k · Answer 10 · 2022-01-23T20:36:50.833000

1.9k

Views

How to round timestamp to 10 minutes in Spark 3.0?

Published on 23 January 2022 at 20:36

score 1.3k · Answer 11 · 2022-08-04T10:37:30.887000

1.3k

Views

How to set driver python path in cluster mode (pyspark)

Published on 04 August 2022 at 10:37

score 175 · Answer 12 · 2022-08-15T16:47:05.110000

175

Views

Issues defining an Aggregator with case class input

Published on 15 August 2022 at 16:47

score 422 · Answer 13 · 2022-06-27T09:56:03.767000

422

Views

Spark can't connect to DB with built-in connection providers

Published on 27 June 2022 at 09:56

score 559 · Answer 14 · 2022-07-12T07:54:18.623000

559

Views

ImportError: Pandas >= 0.23.2 must be installed; however, it was not found. / pyspark/pandas are not properly imported in Apache Spark 3.2.1

Published on 12 July 2022 at 07:54

score 56 · Answer 15 · 2024-01-03T12:33:07.757000

56

Views

Parquet Size Difference

Published on 03 January 2024 at 12:33

score 27 · Answer 16 · 2024-02-28T20:45:35.470000

27

Views

Spark2 to Spark3 Upgrade -> DataFrame foreachPartition method does not work

Published on 28 February 2024 at 20:45

score 843 · Answer 17 · 2022-10-03T10:12:54.773000

843

Views

Aggregate function with Expr in PySpark 3.0.3

Published on 03 October 2022 at 10:12

score 33 · Answer 18 · 2022-10-24T05:23:59.340000

33

Views

Apache spark: asc not working as expected

Published on 24 October 2022 at 05:23

score 83 · Answer 19 · 2021-10-24T19:52:35.233000

83

Views

Why Apache Spark does some checks and raises those exceptions during the job runtime, but has never thrown them during Unit test?

Published on 24 October 2021 at 19:52

score 82 · Answer 20 · 2022-09-23T04:08:12.237000

82

Views

How to suppres INFO spark logs?

Published on 23 September 2022 at 04:08

List Question

How can i add use a SparkDFDataset to my Great Expectations' validator?

Multiple CSV Scan in spark structured streaming for same file in s3

How force spark to move record nonnull fields as _corrupt_record?

How to get week of month in Spark 3.0+?

Does Spark support the WITH clause like SQL?

Spark 3.3.1 picking up current date automatically in data frame if date is missing from given timestamp and not marking it as _corrupt record

ServiceConfigurationError running spark 3.2

Apache Phoenix - Count query returns more than 100k rows, but SELECT query does not return any row

Configure Spark 3 thrift server with Apache Ranger

How to round timestamp to 10 minutes in Spark 3.0?

How to set driver python path in cluster mode (pyspark)

Issues defining an Aggregator with case class input

Spark can't connect to DB with built-in connection providers

ImportError: Pandas >= 0.23.2 must be installed; however, it was not found. / pyspark/pandas are not properly imported in Apache Spark 3.2.1

Parquet Size Difference

Spark2 to Spark3 Upgrade -> DataFrame foreachPartition method does not work

Aggregate function with Expr in PySpark 3.0.3

Apache spark: asc not working as expected

Why Apache Spark does some checks and raises those exceptions during the job runtime, but has never thrown them during Unit test?

How to suppres INFO spark logs?

Trending Questions

Popular # Hahtags

Popular Questions