I'm working with the Giraph-based application that makes heavy use of memory in a NUMA system. It frequently writes and reads to the memory and has multiple threads. Assuming I schedule 4 workers with as many cores as there are cores per chip would that mean they would be pinned to specific CPU so that they can only use local NUMA memory?
Will YARN working on NUMA respect node memory locality?
123 Views Asked by kboom At
1
There are 1 best solutions below
Related Questions in HADOOP-YARN
- Impala Resource Estimation for queries with Group by
- mapreduce job not setting compression codec correctly
- What does namespace and block pool mean in MapReduce 2.0 YARN?
- nmap does not show all open ports
- Spark resources not fully allocated on Amazon EMR
- Not able to format namenode in hadoop-2.6.0 multi node installation
- MapReduce job fails with ExitCodeException exitCode=255
- Spark GraphX memory out of error SparkListenerBus (java.lang.OutOfMemoryError: Java heap space)
- Yarn autodetect slaves failure
- how to ignore key-value pair in Map-Reduce if values are blank?
- What happens to orphaned Yarn Child processes?
- Setting mapreduce.job.jar with a jar in path containing space
- Hive index mapreduce memory errors
- YARN log aggregation on a per job basis
- Importtsv command gives : Container exited with a non-zero exit code 1 error
Related Questions in NUMA
- NUMA support on which CPU? What are the current server configuration of this kind of CPU?
- Is there some way to allocate specific memory for data in NUMA?
- OpenMP performance of different data types on NUMA architecture
- Numa balancer in Linux
- In C, Migrate memory to a NUMA node
- Link to static library by cmake
- how to config linux/CPU better for large scale software running (NUMA)
- Given a page pointer in a linux kernel module, how to get the NUMA node id on which it is assigned?
- Assigning tasks to threads in Cilk and assigning threads to NUMA nodes
- Intel Ultra Path Interconnect (UPI) Bandwidth
- There is a way to pass --cpus-cpuset and --cpuset-mems to docker swarm
- Powershell: Find out if NUMA is configured and how many CPUs are assigned to each NUMA node?
- performance issues with parallel MATLAB on a NUMA machine
- Windows SetThreadAffinityMask has no effect
- In NUMA, does each CPU also has local I/O controller similar to local RAM?
Related Questions in GIRAPH
- How to give multiple input paths to gora avrostore in giraph (or) how to make giraph read multiple input files
- Error in building Apache Giraph Core
- Conversion of graph data from Edge Input Format to Vertex Input Format
- Whats the difference between .ppk and .pem . Where .pem is stored in amazons ec2 cluster?
- Run giraph on Hadoop yarn 2.6.0
- Giraph: Class not Found Exception on custom Job
- Giraph tutorial ShortestPath example job failing
- Pregel for Dynamic Graph Processing and Graph Streams?
- Giraph Shortest Paths Example ClassNotFoundException
- Where are Apache Giraph logs (with log4j) located?
- Compilation Error when building Giraph
- Job failed as tasks failed. failedMaps:1 failedReduces:0 exception while using hadoop and giraph
- compilation error with apache giraph (building from the trunk)
- Eclipse setup for Giraph
- Compiling a project that uses old-versioned Giraph with Maven
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
It depends on what version of YARN are you using. Since YARN 3.1.0, you can enable NUMA scheduling in config. Patch can be found here:
https://issues.apache.org/jira/browse/YARN-5764