Initialise datanode in your hadoop worker machine

Question

When I do start-all.cmd ,all the daemons are starting in master node,but the datanodes don't start automatically in worker machines. I need to connect to my master machine by starting the datanode in my worker machine. Both of my nodes can connect each other , and also hold same copy of config files , then why is datanode not initiated automatically in worker machine ? what shall I do to initialize datanode in worker machines automatically when I run start-all.cmd in master Machine?

Is there any config I am missing?

I am using windows machines. And I have provided hostnames in workers file.

Answer 1

start-all is deprecated and calls start-dfs and start-yarn

Look at the source code of start-dfs.cmd and it runs only hadoop namenode and hadoop datanode. It doesn't loop through any files or configurations to communicate to other nodes.

start-dfs.sh (for Unix environments) on the other hand does much more, and is what the documentation refers to when mentioning the "workers file".

If you want to manage a cluster from Windows, you can use solutions like Chef (via Apache Bigtop), Puppet, or Ansible.

Also worth pointing out that Hadoop on Windows isn't really recommended.