Installing a hadoop cluster between VMs in different hardware machines

Multi tool use
I am responsible for teaching Hadoop to a group of people (let's say 5 people), but without any hardware available.
Each of them has a laptop, with a quite good amount of memory and processors.
I would like to make them create a Hadoop cluster between their own laptops, which will be connected to the same network.
So far, what I think about is to:
- create a VM image with ubuntu 16 preconfigured (ubuntu 16 is my choice) to be ready for being a cluster node
- ask each of them to run the VM on their computer
- creating a cluster on top of this network of VMs
However, I have some locks:
1/ is it possible to create a private network of VMs located on their different machines, so that the hadoop cluster is isolated from the network that links the physical machines?
2/ What could be wrong with this approach?
3/ Is there a better way for handling this need of setting a Hadoop cluster between different personal machine?
By the way, I am pretty ok with Hadoop installation and so on...
Thanks in advance for your help, suggestions, ...
hadoop virtual-machine virtualbox hadoop2
add a comment |
I am responsible for teaching Hadoop to a group of people (let's say 5 people), but without any hardware available.
Each of them has a laptop, with a quite good amount of memory and processors.
I would like to make them create a Hadoop cluster between their own laptops, which will be connected to the same network.
So far, what I think about is to:
- create a VM image with ubuntu 16 preconfigured (ubuntu 16 is my choice) to be ready for being a cluster node
- ask each of them to run the VM on their computer
- creating a cluster on top of this network of VMs
However, I have some locks:
1/ is it possible to create a private network of VMs located on their different machines, so that the hadoop cluster is isolated from the network that links the physical machines?
2/ What could be wrong with this approach?
3/ Is there a better way for handling this need of setting a Hadoop cluster between different personal machine?
By the way, I am pretty ok with Hadoop installation and so on...
Thanks in advance for your help, suggestions, ...
hadoop virtual-machine virtualbox hadoop2
add a comment |
I am responsible for teaching Hadoop to a group of people (let's say 5 people), but without any hardware available.
Each of them has a laptop, with a quite good amount of memory and processors.
I would like to make them create a Hadoop cluster between their own laptops, which will be connected to the same network.
So far, what I think about is to:
- create a VM image with ubuntu 16 preconfigured (ubuntu 16 is my choice) to be ready for being a cluster node
- ask each of them to run the VM on their computer
- creating a cluster on top of this network of VMs
However, I have some locks:
1/ is it possible to create a private network of VMs located on their different machines, so that the hadoop cluster is isolated from the network that links the physical machines?
2/ What could be wrong with this approach?
3/ Is there a better way for handling this need of setting a Hadoop cluster between different personal machine?
By the way, I am pretty ok with Hadoop installation and so on...
Thanks in advance for your help, suggestions, ...
hadoop virtual-machine virtualbox hadoop2
I am responsible for teaching Hadoop to a group of people (let's say 5 people), but without any hardware available.
Each of them has a laptop, with a quite good amount of memory and processors.
I would like to make them create a Hadoop cluster between their own laptops, which will be connected to the same network.
So far, what I think about is to:
- create a VM image with ubuntu 16 preconfigured (ubuntu 16 is my choice) to be ready for being a cluster node
- ask each of them to run the VM on their computer
- creating a cluster on top of this network of VMs
However, I have some locks:
1/ is it possible to create a private network of VMs located on their different machines, so that the hadoop cluster is isolated from the network that links the physical machines?
2/ What could be wrong with this approach?
3/ Is there a better way for handling this need of setting a Hadoop cluster between different personal machine?
By the way, I am pretty ok with Hadoop installation and so on...
Thanks in advance for your help, suggestions, ...
hadoop virtual-machine virtualbox hadoop2
hadoop virtual-machine virtualbox hadoop2
asked yesterday
M Jed
1
1
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
is it possible to create a private network of VMs located on their different machines
Yes, companies do this all the time with clusters of VMs. Granted, these companies have people with years of experience doing networking setups like this, and have some deep knowledge of firewalls and routing tables
so that the hadoop cluster is isolated from the network that links the physical machines?
Not without a specific subnet for connecting all the machines. I'm guessing each laptop is sharing the same router, though, and each device has one network interface shared between the host and the VM, so creating this may prove difficult.
What could be wrong with this approach?
You need to designate at least one machine as the "master" - the namenode, and the ResourceManager. Without this machine, nothing will work. A better approach uses HA deployments, but then you're reliant on "two people"
Is there a better way for handling this need of setting a Hadoop cluster
Use a free tier/credit of AWS, Azure, or GCP for setting up a cluster. It can start with 2-3 nodes, not 5
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53944034%2finstalling-a-hadoop-cluster-between-vms-in-different-hardware-machines%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
is it possible to create a private network of VMs located on their different machines
Yes, companies do this all the time with clusters of VMs. Granted, these companies have people with years of experience doing networking setups like this, and have some deep knowledge of firewalls and routing tables
so that the hadoop cluster is isolated from the network that links the physical machines?
Not without a specific subnet for connecting all the machines. I'm guessing each laptop is sharing the same router, though, and each device has one network interface shared between the host and the VM, so creating this may prove difficult.
What could be wrong with this approach?
You need to designate at least one machine as the "master" - the namenode, and the ResourceManager. Without this machine, nothing will work. A better approach uses HA deployments, but then you're reliant on "two people"
Is there a better way for handling this need of setting a Hadoop cluster
Use a free tier/credit of AWS, Azure, or GCP for setting up a cluster. It can start with 2-3 nodes, not 5
add a comment |
is it possible to create a private network of VMs located on their different machines
Yes, companies do this all the time with clusters of VMs. Granted, these companies have people with years of experience doing networking setups like this, and have some deep knowledge of firewalls and routing tables
so that the hadoop cluster is isolated from the network that links the physical machines?
Not without a specific subnet for connecting all the machines. I'm guessing each laptop is sharing the same router, though, and each device has one network interface shared between the host and the VM, so creating this may prove difficult.
What could be wrong with this approach?
You need to designate at least one machine as the "master" - the namenode, and the ResourceManager. Without this machine, nothing will work. A better approach uses HA deployments, but then you're reliant on "two people"
Is there a better way for handling this need of setting a Hadoop cluster
Use a free tier/credit of AWS, Azure, or GCP for setting up a cluster. It can start with 2-3 nodes, not 5
add a comment |
is it possible to create a private network of VMs located on their different machines
Yes, companies do this all the time with clusters of VMs. Granted, these companies have people with years of experience doing networking setups like this, and have some deep knowledge of firewalls and routing tables
so that the hadoop cluster is isolated from the network that links the physical machines?
Not without a specific subnet for connecting all the machines. I'm guessing each laptop is sharing the same router, though, and each device has one network interface shared between the host and the VM, so creating this may prove difficult.
What could be wrong with this approach?
You need to designate at least one machine as the "master" - the namenode, and the ResourceManager. Without this machine, nothing will work. A better approach uses HA deployments, but then you're reliant on "two people"
Is there a better way for handling this need of setting a Hadoop cluster
Use a free tier/credit of AWS, Azure, or GCP for setting up a cluster. It can start with 2-3 nodes, not 5
is it possible to create a private network of VMs located on their different machines
Yes, companies do this all the time with clusters of VMs. Granted, these companies have people with years of experience doing networking setups like this, and have some deep knowledge of firewalls and routing tables
so that the hadoop cluster is isolated from the network that links the physical machines?
Not without a specific subnet for connecting all the machines. I'm guessing each laptop is sharing the same router, though, and each device has one network interface shared between the host and the VM, so creating this may prove difficult.
What could be wrong with this approach?
You need to designate at least one machine as the "master" - the namenode, and the ResourceManager. Without this machine, nothing will work. A better approach uses HA deployments, but then you're reliant on "two people"
Is there a better way for handling this need of setting a Hadoop cluster
Use a free tier/credit of AWS, Azure, or GCP for setting up a cluster. It can start with 2-3 nodes, not 5
answered 19 hours ago
cricket_007
79.2k1142109
79.2k1142109
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53944034%2finstalling-a-hadoop-cluster-between-vms-in-different-hardware-machines%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
ikYUS7G TBMIwmN,cQGkQDYHoGcKwjU14od FjJ1gy5hvi,RSx