i had the error Py4JJavaError: An error occurred while calling o65.showString in pyspark

232 Views Asked by At

i am trying to implement this code using:

python 3.9

spark-3.3.1-bin-hadoop3 included pyspark

java 1.8.0_171

the paths is alright and i am running other codes on jupyter but i didn't find any answer related to the error Py4JJavaError: An error occurred while calling o65.showString

note: my spark contains spark-sql_2.12-3.3.1 and graphframes-0.8.2-spark3.2-s_2.12 jar files and thats mean the same version of scala 2.12

from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark import SparkContext


sc = SparkContext()
sc.addPyFile('C:\Program Files\spark-3.3.1-bin-hadoop3\jars\graphframes-0.8.2-spark3.2-s_2.12.jar')
sqlc = SQLContext(sc)
# Create a Vertex DataFrame with unique ID column "id"
v = sqlc.createDataFrame([
  ("a", "Alice", 34),
  ("b", "Bob", 36),
  ("c", "Charlie", 30),
], ["id", "name", "age"])
# Create an Edge DataFrame with "src" and "dst" columns
e = sqlc.createDataFrame([
  ("a", "b", "friend"),
  ("b", "c", "follow"),
  ("c", "b", "follow"),
], ["src", "dst", "relationship"])
# Create a GraphFrame
from graphframes import *
g = GraphFrame(v, e)

# Query: Get in-degree of each vertex.
g.inDegrees.show()

# Query: Count the number of "follow" connections in the graph.
g.edges.filter("relationship = 'follow'").count()

# Run PageRank algorithm, and show results.
results = g.pageRank(resetProbability=0.01, maxIter=20)
results.vertices.select("id", "pagerank").show()

i hope some one help to fix the error

0

There are 0 best solutions below