I am trying to setup Hadoop on my laptop (just to learn it and do some basics, and if I have the time a couple advanced things).
I am on Windows 10, using Intellij, I also installed Hadoop 3.2.1 following the steps here.
Hadoop seems to work fine, I mean I was able to do basic interaction with HDFS, copying simple text files, creating folders and so on.
Then I tried running start-dfs.cmd, which seems to work fine as well. When trying start-yarn.cmd however, the RessourceManager seems to start without any notable error, but the following problem occurs with the Yarn NodeManager:
2023-11-11 11:39:34,167 ERROR util.SysInfoWindows: ExitCodeException exitCode=1: PdhAddCounter \Network Interface(*)\Bytes Received/Sec failed with 0xc0000bb8.
Error in GetDiskAndNetwork. Err:1
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008)
at org.apache.hadoop.util.Shell.run(Shell.java:901)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)
at org.apache.hadoop.util.SysInfoWindows.getSystemInfoInfoFromShell(SysInfoWindows.java:86)
at org.apache.hadoop.util.SysInfoWindows.refreshIfNeeded(SysInfoWindows.java:101)
at org.apache.hadoop.util.SysInfoWindows.getPhysicalMemorySize(SysInfoWindows.java:153)
at org.apache.hadoop.yarn.util.ResourceCalculatorPlugin.getPhysicalMemorySize(ResourceCalculatorPlugin.java:64)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl.serviceInit(ContainersMonitorImpl.java:228)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:323)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:516)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:974)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1054)
2023-11-11 11:39:34,167 WARN monitor.ContainersMonitorImpl: NodeManager's totalPmem could not be calculated. Setting it to -1
2023-11-11 11:39:34,178 INFO containermanager.ContainerManagerImpl: Not a recoverable state store. Nothing to recover.
2023-11-11 11:39:34,201 INFO conf.Configuration: resource-types.xml not found
2023-11-11 11:39:34,202 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2023-11-11 11:39:34,209 INFO conf.Configuration: node-resources.xml not found
2023-11-11 11:39:34,209 INFO resource.ResourceUtils: Unable to find 'node-resources.xml'.
The same error happens over and over again until I force the process to stop (close window or Ctrl+C). I tried a lot of things to debug (mainly using ChatGPT, I didn't find many links on Google) but I was unsuccessful. These things were mainly around checking setup of env variables and several hadoop configuration files, for example once ChatGPT suggested I add some property block code into the file yarn-site.xml.
My goal is to run a simple MapReduce WordCount in Hadoop.