Installing a hadoop cluster between VMs in different hardware machines

Multi tool use
Multi tool use












0














I am responsible for teaching Hadoop to a group of people (let's say 5 people), but without any hardware available.



Each of them has a laptop, with a quite good amount of memory and processors.



I would like to make them create a Hadoop cluster between their own laptops, which will be connected to the same network.



So far, what I think about is to:




  • create a VM image with ubuntu 16 preconfigured (ubuntu 16 is my choice) to be ready for being a cluster node

  • ask each of them to run the VM on their computer

  • creating a cluster on top of this network of VMs


However, I have some locks:



1/ is it possible to create a private network of VMs located on their different machines, so that the hadoop cluster is isolated from the network that links the physical machines?



2/ What could be wrong with this approach?



3/ Is there a better way for handling this need of setting a Hadoop cluster between different personal machine?



By the way, I am pretty ok with Hadoop installation and so on...



Thanks in advance for your help, suggestions, ...










share|improve this question



























    0














    I am responsible for teaching Hadoop to a group of people (let's say 5 people), but without any hardware available.



    Each of them has a laptop, with a quite good amount of memory and processors.



    I would like to make them create a Hadoop cluster between their own laptops, which will be connected to the same network.



    So far, what I think about is to:




    • create a VM image with ubuntu 16 preconfigured (ubuntu 16 is my choice) to be ready for being a cluster node

    • ask each of them to run the VM on their computer

    • creating a cluster on top of this network of VMs


    However, I have some locks:



    1/ is it possible to create a private network of VMs located on their different machines, so that the hadoop cluster is isolated from the network that links the physical machines?



    2/ What could be wrong with this approach?



    3/ Is there a better way for handling this need of setting a Hadoop cluster between different personal machine?



    By the way, I am pretty ok with Hadoop installation and so on...



    Thanks in advance for your help, suggestions, ...










    share|improve this question

























      0












      0








      0







      I am responsible for teaching Hadoop to a group of people (let's say 5 people), but without any hardware available.



      Each of them has a laptop, with a quite good amount of memory and processors.



      I would like to make them create a Hadoop cluster between their own laptops, which will be connected to the same network.



      So far, what I think about is to:




      • create a VM image with ubuntu 16 preconfigured (ubuntu 16 is my choice) to be ready for being a cluster node

      • ask each of them to run the VM on their computer

      • creating a cluster on top of this network of VMs


      However, I have some locks:



      1/ is it possible to create a private network of VMs located on their different machines, so that the hadoop cluster is isolated from the network that links the physical machines?



      2/ What could be wrong with this approach?



      3/ Is there a better way for handling this need of setting a Hadoop cluster between different personal machine?



      By the way, I am pretty ok with Hadoop installation and so on...



      Thanks in advance for your help, suggestions, ...










      share|improve this question













      I am responsible for teaching Hadoop to a group of people (let's say 5 people), but without any hardware available.



      Each of them has a laptop, with a quite good amount of memory and processors.



      I would like to make them create a Hadoop cluster between their own laptops, which will be connected to the same network.



      So far, what I think about is to:




      • create a VM image with ubuntu 16 preconfigured (ubuntu 16 is my choice) to be ready for being a cluster node

      • ask each of them to run the VM on their computer

      • creating a cluster on top of this network of VMs


      However, I have some locks:



      1/ is it possible to create a private network of VMs located on their different machines, so that the hadoop cluster is isolated from the network that links the physical machines?



      2/ What could be wrong with this approach?



      3/ Is there a better way for handling this need of setting a Hadoop cluster between different personal machine?



      By the way, I am pretty ok with Hadoop installation and so on...



      Thanks in advance for your help, suggestions, ...







      hadoop virtual-machine virtualbox hadoop2






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked yesterday









      M Jed

      1




      1
























          1 Answer
          1






          active

          oldest

          votes


















          0















          is it possible to create a private network of VMs located on their different machines




          Yes, companies do this all the time with clusters of VMs. Granted, these companies have people with years of experience doing networking setups like this, and have some deep knowledge of firewalls and routing tables




          so that the hadoop cluster is isolated from the network that links the physical machines?




          Not without a specific subnet for connecting all the machines. I'm guessing each laptop is sharing the same router, though, and each device has one network interface shared between the host and the VM, so creating this may prove difficult.




          What could be wrong with this approach?




          You need to designate at least one machine as the "master" - the namenode, and the ResourceManager. Without this machine, nothing will work. A better approach uses HA deployments, but then you're reliant on "two people"




          Is there a better way for handling this need of setting a Hadoop cluster




          Use a free tier/credit of AWS, Azure, or GCP for setting up a cluster. It can start with 2-3 nodes, not 5






          share|improve this answer





















            Your Answer






            StackExchange.ifUsing("editor", function () {
            StackExchange.using("externalEditor", function () {
            StackExchange.using("snippets", function () {
            StackExchange.snippets.init();
            });
            });
            }, "code-snippets");

            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "1"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53944034%2finstalling-a-hadoop-cluster-between-vms-in-different-hardware-machines%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            0















            is it possible to create a private network of VMs located on their different machines




            Yes, companies do this all the time with clusters of VMs. Granted, these companies have people with years of experience doing networking setups like this, and have some deep knowledge of firewalls and routing tables




            so that the hadoop cluster is isolated from the network that links the physical machines?




            Not without a specific subnet for connecting all the machines. I'm guessing each laptop is sharing the same router, though, and each device has one network interface shared between the host and the VM, so creating this may prove difficult.




            What could be wrong with this approach?




            You need to designate at least one machine as the "master" - the namenode, and the ResourceManager. Without this machine, nothing will work. A better approach uses HA deployments, but then you're reliant on "two people"




            Is there a better way for handling this need of setting a Hadoop cluster




            Use a free tier/credit of AWS, Azure, or GCP for setting up a cluster. It can start with 2-3 nodes, not 5






            share|improve this answer


























              0















              is it possible to create a private network of VMs located on their different machines




              Yes, companies do this all the time with clusters of VMs. Granted, these companies have people with years of experience doing networking setups like this, and have some deep knowledge of firewalls and routing tables




              so that the hadoop cluster is isolated from the network that links the physical machines?




              Not without a specific subnet for connecting all the machines. I'm guessing each laptop is sharing the same router, though, and each device has one network interface shared between the host and the VM, so creating this may prove difficult.




              What could be wrong with this approach?




              You need to designate at least one machine as the "master" - the namenode, and the ResourceManager. Without this machine, nothing will work. A better approach uses HA deployments, but then you're reliant on "two people"




              Is there a better way for handling this need of setting a Hadoop cluster




              Use a free tier/credit of AWS, Azure, or GCP for setting up a cluster. It can start with 2-3 nodes, not 5






              share|improve this answer
























                0












                0








                0







                is it possible to create a private network of VMs located on their different machines




                Yes, companies do this all the time with clusters of VMs. Granted, these companies have people with years of experience doing networking setups like this, and have some deep knowledge of firewalls and routing tables




                so that the hadoop cluster is isolated from the network that links the physical machines?




                Not without a specific subnet for connecting all the machines. I'm guessing each laptop is sharing the same router, though, and each device has one network interface shared between the host and the VM, so creating this may prove difficult.




                What could be wrong with this approach?




                You need to designate at least one machine as the "master" - the namenode, and the ResourceManager. Without this machine, nothing will work. A better approach uses HA deployments, but then you're reliant on "two people"




                Is there a better way for handling this need of setting a Hadoop cluster




                Use a free tier/credit of AWS, Azure, or GCP for setting up a cluster. It can start with 2-3 nodes, not 5






                share|improve this answer













                is it possible to create a private network of VMs located on their different machines




                Yes, companies do this all the time with clusters of VMs. Granted, these companies have people with years of experience doing networking setups like this, and have some deep knowledge of firewalls and routing tables




                so that the hadoop cluster is isolated from the network that links the physical machines?




                Not without a specific subnet for connecting all the machines. I'm guessing each laptop is sharing the same router, though, and each device has one network interface shared between the host and the VM, so creating this may prove difficult.




                What could be wrong with this approach?




                You need to designate at least one machine as the "master" - the namenode, and the ResourceManager. Without this machine, nothing will work. A better approach uses HA deployments, but then you're reliant on "two people"




                Is there a better way for handling this need of setting a Hadoop cluster




                Use a free tier/credit of AWS, Azure, or GCP for setting up a cluster. It can start with 2-3 nodes, not 5







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered 19 hours ago









                cricket_007

                79.2k1142109




                79.2k1142109






























                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Stack Overflow!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.





                    Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


                    Please pay close attention to the following guidance:


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53944034%2finstalling-a-hadoop-cluster-between-vms-in-different-hardware-machines%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    ikYUS7G TBMIwmN,cQGkQDYHoGcKwjU14od FjJ1gy5hvi,RSx
                    Xoa hVtSKHS49TpbO22e,S,nm0IEcS,X3pzDq rXGsnWept,c2kJ hi uNhgr,OWRUfBUjWaeeiYf4SXU9N3r7JnECPwCVZf,t87BEiU,Gk,1

                    Popular posts from this blog

                    Monofisismo

                    Angular Downloading a file using contenturl with Basic Authentication

                    Olmecas