Error accessing Spark thrift Server

Question

Error accessing Spark thrift Server

787 Views Asked by Mercury Trivival At 16 April 2025 at 22:50

Spark version: 1.6.3

I running Spark thrift server as proxy. But it not running as long as I expected. It always stop when get high load.

This is Error when I access.

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /jobs/. Reason:
<pre>    Server Error</pre></p><h3>Caused by:</h3><pre>java.lang.OutOfMemoryError: Java heap space
 at java.util.Arrays.copyOfRange(Arrays.java:3664)
 at java.lang.String.<init>(String.java:207)
 at java.lang.StringBuilder.toString(StringBuilder.java:407)
 at scala.collection.mutable.StringBuilder.toString(StringBuilder.scala:427)
 at scala.xml.Node.buildString(Node.scala:161)
 at scala.xml.Node.toString(Node.scala:166)
 at org.apache.spark.ui.JettyUtils$$anonfun$htmlResponderToServlet$1.apply(JettyUtils.scala:55)
 at org.apache.spark.ui.JettyUtils$$anonfun$htmlResponderToServlet$1.apply(JettyUtils.scala:55)
 at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:83)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
 at org.spark-project.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
 at org.spark-project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507)
 at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:179)
 at org.spark-project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
 at org.spark-project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
 at org.spark-project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
 at org.spark-project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
 at org.spark-project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
 at org.spark-project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
 at org.spark-project.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
 at org.spark-project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
 at org.spark-project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
 at org.spark-project.jetty.server.Server.handle(Server.java:370)
 at org.spark-project.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
 at org.spark-project.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
 at org.spark-project.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
 at org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:641)
 at org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
 at org.spark-project.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
 at org.spark-project.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
 at org.spark-project.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
</pre>
<hr /><i><small>Powered by Jetty://</small></i><br/>                                                
</body>
</html>

I see error is java.lang.OutOfMemoryError: Java heap space

But I don't know what memory I need increase:

Memory of server running SPARK
Memory of executor configure with SPARK
other Memory configure...

Update: my SPARK configure

my SPARK configure

Original Q&A

There are 2 best solutions below

**OneCricketeer** · Answer 1

The thrift server runs in it's own memory space. You can specify that via the executor memory property.

For example

./sbin/start-thriftserver.sh --executor-memory 512m

I think the default is 1g, as the spark-env.sh has.

**Saurav Bhowmick** · Answer 2

Based on the number of nodes in the cluster the Spark thrift Server can be brought up for( e.g., on AWS EMR) you can use with 1 small r3.2xlarge node

sudo /usr/lib/spark/sbin/start-thriftserver.sh --driver-memory 10g --verbose --master yarn-client --executor-memory 15g --num-executors 6 --executor-cores 2

Also ensure monitoring or cleaning up the Spark History folder - hdfs dfs -ls /var/log/spark/apps/ -- on EMR

Queries will be stuck if multiple jobs are in .inprogress state in the folder

there is a way to set up automated cleanup of Spark history folder

Error accessing Spark thrift Server

There are 2 best solutions below

Related Questions in HADOOP

Related Questions in APACHE-SPARK

Related Questions in APACHE-SPARK-1.6

Related Questions in SPARK-THRIFTSERVER

Trending Questions

Popular # Hahtags

Popular Questions