How To Fix: HtmlUnit GetElementById Returns Null

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}

I am writing a web scraper and am trying to type in a search word into a search box. However, it looks like I am getting null when I try to access the search box by ID. I am just learning HtmlUnit so I could be missing something very obvious but I have not been able to identify this myself yet.

Here is the website's code:

<html xmlns="http://www.w3.org/1999/xhtml" xml:1ang="en" class="no-touch">

    <head>-</head>

    <body lang="en" class="garageBrand" emailcookiename="grgemailca" loyaltycookiename="grgloyaltyca">

        <div id="fb-root" class="fb_reset">-</div>

        <noscript>...</noscript>

        <script>...</script>

        <div id="container">

            <div id="avsDialog" sty1e="disp1ay: none; position: absolute; top: 0; right: 0;"></div>

            <input type="hidden" value="en" id="displayLanguage">

            <input type="hidden" value="garageSiteCA" id="currSiteId">

            <input type="hidden" value="en_CA" id="currLocale">

            <div id="contentarea">

                <div id="header" class="nonHeaderScroll">

                <div id="topnav">...</div>

                <div class="socialSearch">

                <div id="searchMenu">

                    <form action="//www.garageclothing.com/ca/search/search.jsp" method="GET">

                        <input type="hidden" name="N" value="0">

                        <input type="hidden" name="Dy" value="1">

                        <input type="hidden" name="Nty" value="1">

                        <input type="hidden" name="Ntk" value="All">

                        <input type="hidden" name="Ntx" value="mode matchall">

                        <input id="searchText" maxlength="40" type="text" name="Ntt" class="textInput" placeholder="Search..." autocomplete="off">

                        <input class="mainSearchButton" type="image" src="//images.gdicdn.com/img/magnifying-glass.png?version=375" name="search">

                    </form>

                </div>

Here is my code:

import com.gargoylesoftware.htmlunit.WebClient;

import com.gargoylesoftware.htmlunit.html.HtmlPage;

import com.gargoylesoftware.htmlunit.html.HtmlSubmitInput;

import com.gargoylesoftware.htmlunit.html.HtmlInput;



import java.io.IOException;





public class Main {



public static void main(String args) {



    WebClient client = new WebClient();

    client.getOptions().setJavaScriptEnabled(true);

    client.getOptions().setCssEnabled(false);

    client.getOptions().setUseInsecureSSL(true);



    try {

        HtmlPage page = client.getPage("https://www.garageclothing.com/ca");



        // Check for popup.

        if(page.getElementById("cboxClose") != null) {

            page = page.getElementById("cboxClose").click();

        }



        // Debugging line that returns null:

        System.out.println(page.getElementById("searchText"));

        // What I would like to do:

      /*HtmlInput searchInput = (HtmlInput) page.getElementById("searchText");

        searchInput.setValueAttribute("red scarf");

        HtmlSubmitInput submitBtn = page.getElementByName("search");

        page = submitBtn.click();



        System.out.println(page.asXml());*/



    } catch (IOException e) {

        e.printStackTrace();

    }

}



}

edited Jan 10 at 18:58

asked Jan 3 at 20:53

cosmicluna

69110

add a comment |

Here is the website's code:

<html xmlns="http://www.w3.org/1999/xhtml" xml:1ang="en" class="no-touch">

    <head>-</head>

    <body lang="en" class="garageBrand" emailcookiename="grgemailca" loyaltycookiename="grgloyaltyca">

        <div id="fb-root" class="fb_reset">-</div>

        <noscript>...</noscript>

        <script>...</script>

        <div id="container">

            <div id="avsDialog" sty1e="disp1ay: none; position: absolute; top: 0; right: 0;"></div>

            <input type="hidden" value="en" id="displayLanguage">

            <input type="hidden" value="garageSiteCA" id="currSiteId">

            <input type="hidden" value="en_CA" id="currLocale">

            <div id="contentarea">

                <div id="header" class="nonHeaderScroll">

                <div id="topnav">...</div>

                <div class="socialSearch">

                <div id="searchMenu">

                    <form action="//www.garageclothing.com/ca/search/search.jsp" method="GET">

                        <input type="hidden" name="N" value="0">

                        <input type="hidden" name="Dy" value="1">

                        <input type="hidden" name="Nty" value="1">

                        <input type="hidden" name="Ntk" value="All">

                        <input type="hidden" name="Ntx" value="mode matchall">

                        <input id="searchText" maxlength="40" type="text" name="Ntt" class="textInput" placeholder="Search..." autocomplete="off">

                        <input class="mainSearchButton" type="image" src="//images.gdicdn.com/img/magnifying-glass.png?version=375" name="search">

                    </form>

                </div>

Here is my code:

import com.gargoylesoftware.htmlunit.WebClient;

import com.gargoylesoftware.htmlunit.html.HtmlPage;

import com.gargoylesoftware.htmlunit.html.HtmlSubmitInput;

import com.gargoylesoftware.htmlunit.html.HtmlInput;



import java.io.IOException;





public class Main {



public static void main(String args) {



    WebClient client = new WebClient();

    client.getOptions().setJavaScriptEnabled(true);

    client.getOptions().setCssEnabled(false);

    client.getOptions().setUseInsecureSSL(true);



    try {

        HtmlPage page = client.getPage("https://www.garageclothing.com/ca");



        // Check for popup.

        if(page.getElementById("cboxClose") != null) {

            page = page.getElementById("cboxClose").click();

        }



        // Debugging line that returns null:

        System.out.println(page.getElementById("searchText"));

        // What I would like to do:

      /*HtmlInput searchInput = (HtmlInput) page.getElementById("searchText");

        searchInput.setValueAttribute("red scarf");

        HtmlSubmitInput submitBtn = page.getElementByName("search");

        page = submitBtn.click();



        System.out.println(page.asXml());*/



    } catch (IOException e) {

        e.printStackTrace();

    }

}



}

edited Jan 10 at 18:58

asked Jan 3 at 20:53

cosmicluna

69110

add a comment |

Here is the website's code:

<html xmlns="http://www.w3.org/1999/xhtml" xml:1ang="en" class="no-touch">

    <head>-</head>

    <body lang="en" class="garageBrand" emailcookiename="grgemailca" loyaltycookiename="grgloyaltyca">

        <div id="fb-root" class="fb_reset">-</div>

        <noscript>...</noscript>

        <script>...</script>

        <div id="container">

            <div id="avsDialog" sty1e="disp1ay: none; position: absolute; top: 0; right: 0;"></div>

            <input type="hidden" value="en" id="displayLanguage">

            <input type="hidden" value="garageSiteCA" id="currSiteId">

            <input type="hidden" value="en_CA" id="currLocale">

            <div id="contentarea">

                <div id="header" class="nonHeaderScroll">

                <div id="topnav">...</div>

                <div class="socialSearch">

                <div id="searchMenu">

                    <form action="//www.garageclothing.com/ca/search/search.jsp" method="GET">

                        <input type="hidden" name="N" value="0">

                        <input type="hidden" name="Dy" value="1">

                        <input type="hidden" name="Nty" value="1">

                        <input type="hidden" name="Ntk" value="All">

                        <input type="hidden" name="Ntx" value="mode matchall">

                        <input id="searchText" maxlength="40" type="text" name="Ntt" class="textInput" placeholder="Search..." autocomplete="off">

                        <input class="mainSearchButton" type="image" src="//images.gdicdn.com/img/magnifying-glass.png?version=375" name="search">

                    </form>

                </div>

Here is my code:

import com.gargoylesoftware.htmlunit.WebClient;

import com.gargoylesoftware.htmlunit.html.HtmlPage;

import com.gargoylesoftware.htmlunit.html.HtmlSubmitInput;

import com.gargoylesoftware.htmlunit.html.HtmlInput;



import java.io.IOException;





public class Main {



public static void main(String args) {



    WebClient client = new WebClient();

    client.getOptions().setJavaScriptEnabled(true);

    client.getOptions().setCssEnabled(false);

    client.getOptions().setUseInsecureSSL(true);



    try {

        HtmlPage page = client.getPage("https://www.garageclothing.com/ca");



        // Check for popup.

        if(page.getElementById("cboxClose") != null) {

            page = page.getElementById("cboxClose").click();

        }



        // Debugging line that returns null:

        System.out.println(page.getElementById("searchText"));

        // What I would like to do:

      /*HtmlInput searchInput = (HtmlInput) page.getElementById("searchText");

        searchInput.setValueAttribute("red scarf");

        HtmlSubmitInput submitBtn = page.getElementByName("search");

        page = submitBtn.click();



        System.out.println(page.asXml());*/



    } catch (IOException e) {

        e.printStackTrace();

    }

}



}

edited Jan 10 at 18:58

asked Jan 3 at 20:53

cosmicluna

69110

Here is the website's code:

<html xmlns="http://www.w3.org/1999/xhtml" xml:1ang="en" class="no-touch">

    <head>-</head>

    <body lang="en" class="garageBrand" emailcookiename="grgemailca" loyaltycookiename="grgloyaltyca">

        <div id="fb-root" class="fb_reset">-</div>

        <noscript>...</noscript>

        <script>...</script>

        <div id="container">

            <div id="avsDialog" sty1e="disp1ay: none; position: absolute; top: 0; right: 0;"></div>

            <input type="hidden" value="en" id="displayLanguage">

            <input type="hidden" value="garageSiteCA" id="currSiteId">

            <input type="hidden" value="en_CA" id="currLocale">

            <div id="contentarea">

                <div id="header" class="nonHeaderScroll">

                <div id="topnav">...</div>

                <div class="socialSearch">

                <div id="searchMenu">

                    <form action="//www.garageclothing.com/ca/search/search.jsp" method="GET">

                        <input type="hidden" name="N" value="0">

                        <input type="hidden" name="Dy" value="1">

                        <input type="hidden" name="Nty" value="1">

                        <input type="hidden" name="Ntk" value="All">

                        <input type="hidden" name="Ntx" value="mode matchall">

                        <input id="searchText" maxlength="40" type="text" name="Ntt" class="textInput" placeholder="Search..." autocomplete="off">

                        <input class="mainSearchButton" type="image" src="//images.gdicdn.com/img/magnifying-glass.png?version=375" name="search">

                    </form>

                </div>

Here is my code:

import com.gargoylesoftware.htmlunit.WebClient;

import com.gargoylesoftware.htmlunit.html.HtmlPage;

import com.gargoylesoftware.htmlunit.html.HtmlSubmitInput;

import com.gargoylesoftware.htmlunit.html.HtmlInput;



import java.io.IOException;





public class Main {



public static void main(String args) {



    WebClient client = new WebClient();

    client.getOptions().setJavaScriptEnabled(true);

    client.getOptions().setCssEnabled(false);

    client.getOptions().setUseInsecureSSL(true);



    try {

        HtmlPage page = client.getPage("https://www.garageclothing.com/ca");



        // Check for popup.

        if(page.getElementById("cboxClose") != null) {

            page = page.getElementById("cboxClose").click();

        }



        // Debugging line that returns null:

        System.out.println(page.getElementById("searchText"));

        // What I would like to do:

      /*HtmlInput searchInput = (HtmlInput) page.getElementById("searchText");

        searchInput.setValueAttribute("red scarf");

        HtmlSubmitInput submitBtn = page.getElementByName("search");

        page = submitBtn.click();



        System.out.println(page.asXml());*/



    } catch (IOException e) {

        e.printStackTrace();

    }

}



}

java web-scraping htmlunit

edited Jan 10 at 18:58

asked Jan 3 at 20:53

cosmicluna

69110

edited Jan 10 at 18:58

asked Jan 3 at 20:53

cosmicluna

69110

edited Jan 10 at 18:58

asked Jan 3 at 20:53

cosmicluna

69110

asked Jan 3 at 20:53

cosmicluna

69110

asked Jan 3 at 20:53

cosmicluna

69110

add a comment |

2 Answers
2

active

oldest

votes

Even if the page looks simple, this page is (lake many shopping portals) really complicated and based on tons of javascript (not only for the page itself, but also for all this nasty trackers to observe the users). If you like to learn more about this page i suggest to use a web proxy like Charles to capture the whole traffic.

Now back to your problem...
Because HtmlUnit javascript support (based on Rhino) is not perfect, you face some javascript errors. To not stop at js errors, you have to configure the client

webClient.getOptions().setThrowExceptionOnScriptError(false);

The next step is to get the page. This is also not that simple because of all the js stuff. It looks like the js stuff also replaces the page initially returned by getting the url. Because of this you have to do three steps

get the page

wait some time to let the js do some work

get the current page from the current window

Now you are able to find the search field; type some search into it and finally press the search button. Then you have to do again the three steps to get the current content.

Hope that helps....

public static void main(String args) throws IOException {

    String url = "https://www.garageclothing.com/ca";



    try (final WebClient webClient = new WebClient()) {

        // do not stop at js errors

        webClient.getOptions().setThrowExceptionOnScriptError(false);



        webClient.getPage(url);

        webClient.waitForBackgroundJavaScript(10000);



        HtmlPage page = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();

        HtmlInput searchInput = (HtmlInput) page.getElementById("searchText");

        searchInput.type("red scarf");



        HtmlElement submitBtn = (HtmlElement) page.getElementByName("search");

        submitBtn.click();

        webClient.waitForBackgroundJavaScript(10000);



        page = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();

        // System.out.println("------------------------------------------------");

        // System.out.println(page.asXml());



        System.out.println("------------------------------------------------");

        final DomNodeList<DomNode> divs = page.querySelectorAll(".divProdPriceSale");

        for (DomNode div : divs) {

            System.out.println(div.asText());

        }

    }

}

answered Jan 14 at 19:49

RBRi

1,3662610

add a comment |

You should check the URL you are passing to the WebClient is the one you are viewing in the web browser you are using.

I went to the link you use in your code (https://www.garageclothing.com) and the page I got is not the one you are expecting. It asked me to pick a country (USA or Canada) and after I clicked in any of the options, it then took me to the page you are expecting.

Try changing the URL to "https://www.garageclothing.com/us/" or "https://www.garageclothing.com/ca/"

answered Jan 9 at 19:43

DSantiagoBC

35019

Thanks for the response! Sorry, I actually hadn't updated my answer, as this was something I realized and am currently using. Though it still does not work with the updated URL. (I am using the /ca version)

– cosmicluna
Jan 10 at 18:59

@cosmicluna, looks like the content of the page is loaded by some Javascripts executed at the initial page load. While debugging I noticed the page throws some Javascript errors that "real browsers" apparently can handle. I think you should open an HtmlUnit issue (Github) about this, so the api devs can help you

– DSantiagoBC
Jan 10 at 20:18

ok awesome I will do that! If you don't mind me asking, how are you debugging this? I haven't found any effective ways to visualize what is actually going on behind the scenes of HtmlUnit.

– cosmicluna
Jan 10 at 21:05

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54029644%2fhow-to-fix-htmlunit-getelementbyid-returns-null%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

2 Answers
2

active

oldest

votes

2 Answers
2

active

oldest

votes

Now back to your problem...
Because HtmlUnit javascript support (based on Rhino) is not perfect, you face some javascript errors. To not stop at js errors, you have to configure the client

webClient.getOptions().setThrowExceptionOnScriptError(false);

get the page

wait some time to let the js do some work

get the current page from the current window

Now you are able to find the search field; type some search into it and finally press the search button. Then you have to do again the three steps to get the current content.

Hope that helps....

public static void main(String args) throws IOException {

    String url = "https://www.garageclothing.com/ca";



    try (final WebClient webClient = new WebClient()) {

        // do not stop at js errors

        webClient.getOptions().setThrowExceptionOnScriptError(false);



        webClient.getPage(url);

        webClient.waitForBackgroundJavaScript(10000);



        HtmlPage page = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();

        HtmlInput searchInput = (HtmlInput) page.getElementById("searchText");

        searchInput.type("red scarf");



        HtmlElement submitBtn = (HtmlElement) page.getElementByName("search");

        submitBtn.click();

        webClient.waitForBackgroundJavaScript(10000);



        page = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();

        // System.out.println("------------------------------------------------");

        // System.out.println(page.asXml());



        System.out.println("------------------------------------------------");

        final DomNodeList<DomNode> divs = page.querySelectorAll(".divProdPriceSale");

        for (DomNode div : divs) {

            System.out.println(div.asText());

        }

    }

}

answered Jan 14 at 19:49

RBRi

1,3662610

add a comment |

Now back to your problem...
Because HtmlUnit javascript support (based on Rhino) is not perfect, you face some javascript errors. To not stop at js errors, you have to configure the client

webClient.getOptions().setThrowExceptionOnScriptError(false);

get the page

wait some time to let the js do some work

get the current page from the current window

Now you are able to find the search field; type some search into it and finally press the search button. Then you have to do again the three steps to get the current content.

Hope that helps....

public static void main(String args) throws IOException {

    String url = "https://www.garageclothing.com/ca";



    try (final WebClient webClient = new WebClient()) {

        // do not stop at js errors

        webClient.getOptions().setThrowExceptionOnScriptError(false);



        webClient.getPage(url);

        webClient.waitForBackgroundJavaScript(10000);



        HtmlPage page = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();

        HtmlInput searchInput = (HtmlInput) page.getElementById("searchText");

        searchInput.type("red scarf");



        HtmlElement submitBtn = (HtmlElement) page.getElementByName("search");

        submitBtn.click();

        webClient.waitForBackgroundJavaScript(10000);



        page = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();

        // System.out.println("------------------------------------------------");

        // System.out.println(page.asXml());



        System.out.println("------------------------------------------------");

        final DomNodeList<DomNode> divs = page.querySelectorAll(".divProdPriceSale");

        for (DomNode div : divs) {

            System.out.println(div.asText());

        }

    }

}

answered Jan 14 at 19:49

RBRi

1,3662610

add a comment |

Now back to your problem...
Because HtmlUnit javascript support (based on Rhino) is not perfect, you face some javascript errors. To not stop at js errors, you have to configure the client

webClient.getOptions().setThrowExceptionOnScriptError(false);

get the page

wait some time to let the js do some work

get the current page from the current window

Now you are able to find the search field; type some search into it and finally press the search button. Then you have to do again the three steps to get the current content.

Hope that helps....

public static void main(String args) throws IOException {

    String url = "https://www.garageclothing.com/ca";



    try (final WebClient webClient = new WebClient()) {

        // do not stop at js errors

        webClient.getOptions().setThrowExceptionOnScriptError(false);



        webClient.getPage(url);

        webClient.waitForBackgroundJavaScript(10000);



        HtmlPage page = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();

        HtmlInput searchInput = (HtmlInput) page.getElementById("searchText");

        searchInput.type("red scarf");



        HtmlElement submitBtn = (HtmlElement) page.getElementByName("search");

        submitBtn.click();

        webClient.waitForBackgroundJavaScript(10000);



        page = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();

        // System.out.println("------------------------------------------------");

        // System.out.println(page.asXml());



        System.out.println("------------------------------------------------");

        final DomNodeList<DomNode> divs = page.querySelectorAll(".divProdPriceSale");

        for (DomNode div : divs) {

            System.out.println(div.asText());

        }

    }

}

answered Jan 14 at 19:49

RBRi

1,3662610

Now back to your problem...
Because HtmlUnit javascript support (based on Rhino) is not perfect, you face some javascript errors. To not stop at js errors, you have to configure the client

webClient.getOptions().setThrowExceptionOnScriptError(false);

get the page

wait some time to let the js do some work

get the current page from the current window

Now you are able to find the search field; type some search into it and finally press the search button. Then you have to do again the three steps to get the current content.

Hope that helps....

public static void main(String args) throws IOException {

    String url = "https://www.garageclothing.com/ca";



    try (final WebClient webClient = new WebClient()) {

        // do not stop at js errors

        webClient.getOptions().setThrowExceptionOnScriptError(false);



        webClient.getPage(url);

        webClient.waitForBackgroundJavaScript(10000);



        HtmlPage page = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();

        HtmlInput searchInput = (HtmlInput) page.getElementById("searchText");

        searchInput.type("red scarf");



        HtmlElement submitBtn = (HtmlElement) page.getElementByName("search");

        submitBtn.click();

        webClient.waitForBackgroundJavaScript(10000);



        page = (HtmlPage) webClient.getCurrentWindow().getEnclosedPage();

        // System.out.println("------------------------------------------------");

        // System.out.println(page.asXml());



        System.out.println("------------------------------------------------");

        final DomNodeList<DomNode> divs = page.querySelectorAll(".divProdPriceSale");

        for (DomNode div : divs) {

            System.out.println(div.asText());

        }

    }

}

answered Jan 14 at 19:49

RBRi

1,3662610

answered Jan 14 at 19:49

RBRi

1,3662610

answered Jan 14 at 19:49

RBRi

1,3662610

answered Jan 14 at 19:49

RBRi

1,3662610

add a comment |

You should check the URL you are passing to the WebClient is the one you are viewing in the web browser you are using.

Try changing the URL to "https://www.garageclothing.com/us/" or "https://www.garageclothing.com/ca/"

answered Jan 9 at 19:43

DSantiagoBC

35019

Thanks for the response! Sorry, I actually hadn't updated my answer, as this was something I realized and am currently using. Though it still does not work with the updated URL. (I am using the /ca version)

– cosmicluna
Jan 10 at 18:59

@cosmicluna, looks like the content of the page is loaded by some Javascripts executed at the initial page load. While debugging I noticed the page throws some Javascript errors that "real browsers" apparently can handle. I think you should open an HtmlUnit issue (Github) about this, so the api devs can help you

– DSantiagoBC
Jan 10 at 20:18

ok awesome I will do that! If you don't mind me asking, how are you debugging this? I haven't found any effective ways to visualize what is actually going on behind the scenes of HtmlUnit.

– cosmicluna
Jan 10 at 21:05

add a comment |

You should check the URL you are passing to the WebClient is the one you are viewing in the web browser you are using.

Try changing the URL to "https://www.garageclothing.com/us/" or "https://www.garageclothing.com/ca/"

answered Jan 9 at 19:43

DSantiagoBC

35019

Thanks for the response! Sorry, I actually hadn't updated my answer, as this was something I realized and am currently using. Though it still does not work with the updated URL. (I am using the /ca version)

– cosmicluna
Jan 10 at 18:59

@cosmicluna, looks like the content of the page is loaded by some Javascripts executed at the initial page load. While debugging I noticed the page throws some Javascript errors that "real browsers" apparently can handle. I think you should open an HtmlUnit issue (Github) about this, so the api devs can help you

– DSantiagoBC
Jan 10 at 20:18

ok awesome I will do that! If you don't mind me asking, how are you debugging this? I haven't found any effective ways to visualize what is actually going on behind the scenes of HtmlUnit.

– cosmicluna
Jan 10 at 21:05

add a comment |

You should check the URL you are passing to the WebClient is the one you are viewing in the web browser you are using.

Try changing the URL to "https://www.garageclothing.com/us/" or "https://www.garageclothing.com/ca/"

answered Jan 9 at 19:43

DSantiagoBC

35019

You should check the URL you are passing to the WebClient is the one you are viewing in the web browser you are using.

Try changing the URL to "https://www.garageclothing.com/us/" or "https://www.garageclothing.com/ca/"

answered Jan 9 at 19:43

DSantiagoBC

35019

answered Jan 9 at 19:43

DSantiagoBC

35019

answered Jan 9 at 19:43

DSantiagoBC

35019

answered Jan 9 at 19:43

DSantiagoBC

35019

Thanks for the response! Sorry, I actually hadn't updated my answer, as this was something I realized and am currently using. Though it still does not work with the updated URL. (I am using the /ca version)

– cosmicluna
Jan 10 at 18:59

@cosmicluna, looks like the content of the page is loaded by some Javascripts executed at the initial page load. While debugging I noticed the page throws some Javascript errors that "real browsers" apparently can handle. I think you should open an HtmlUnit issue (Github) about this, so the api devs can help you

– DSantiagoBC
Jan 10 at 20:18

ok awesome I will do that! If you don't mind me asking, how are you debugging this? I haven't found any effective ways to visualize what is actually going on behind the scenes of HtmlUnit.

– cosmicluna
Jan 10 at 21:05

add a comment |

Thanks for the response! Sorry, I actually hadn't updated my answer, as this was something I realized and am currently using. Though it still does not work with the updated URL. (I am using the /ca version)

– cosmicluna
Jan 10 at 18:59

@cosmicluna, looks like the content of the page is loaded by some Javascripts executed at the initial page load. While debugging I noticed the page throws some Javascript errors that "real browsers" apparently can handle. I think you should open an HtmlUnit issue (Github) about this, so the api devs can help you

– DSantiagoBC
Jan 10 at 20:18

ok awesome I will do that! If you don't mind me asking, how are you debugging this? I haven't found any effective ways to visualize what is actually going on behind the scenes of HtmlUnit.

– cosmicluna
Jan 10 at 21:05

Thanks for the response! Sorry, I actually hadn't updated my answer, as this was something I realized and am currently using. Though it still does not work with the updated URL. (I am using the /ca version)

– cosmicluna
Jan 10 at 18:59

@cosmicluna, looks like the content of the page is loaded by some Javascripts executed at the initial page load. While debugging I noticed the page throws some Javascript errors that "real browsers" apparently can handle. I think you should open an HtmlUnit issue (Github) about this, so the api devs can help you

– DSantiagoBC
Jan 10 at 20:18

ok awesome I will do that! If you don't mind me asking, how are you debugging this? I haven't found any effective ways to visualize what is actually going on behind the scenes of HtmlUnit.

– cosmicluna
Jan 10 at 21:05

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Bdtjtk