How to set vcores for sqoop job

688 Views Asked by At

How does sqoop aquires vcores for execution. I submit a sqoop job with 4 mappers and the application starts with 1 vcore even when there are around 50 vcores available. So there is no parelllization. This is an import sqoop job into text file. These are the configuration paramaters in yarn with respect to vcrore allocation. How does a sqoop job acquires vcores for execution?

<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>4</value>
<source>yarn-site.xml</source>
</property>

<property>
<name>yarn.app.mapreduce.am.resource.cpu-vcores</name>
<value>1</value>
<source>mapred-default.xml</source>
</property>

<property>
<name>mapreduce.reduce.cpu.vcores</name>
<value>1</value>
<source>mapred-default.xml</source>
</property>

<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>8</value>
<source>yarn-default.xml</source>
</property>

<property>
<name>mapreduce.map.cpu.vcores</name>
<value>1</value>
<source>mapred-default.xml</source>
</property>

<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
<source>yarn-site.xml</source>
</property>

<property>
<name>yarn.scheduler.increment-allocation-vcores</name>
<value>1</value>
<source>yarn-site.xml</source>
</property>
1

There are 1 best solutions below

1
tk421 On

From https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_controlling_parallelism

Sqoop imports data in parallel from most database sources. You can specify the number of map tasks (parallel processes) to use to perform the import by using the -m or --num-mappers argument.

Map tasks are the same as to vcores.