Bdtjtk

I am trying to scrape flyers from flipp.com/weekly_ads using Scrapy. Before I can scrape the flyers, I need to input my area code, and search for local flyers (on the site, this is done by clicking a button).

I am trying to input a value, and simualate "clicking a button" using Scrapy.

Initially, I thought that I would be able to use a FormRequest.from_response to search for the form, and input my area code as a value. However, the button is written in javascript, meaning that the form cannot be found.

So, I tried to find the HTTP call via Inspect Element > Developer Tools > Network > XHR to see if any of the calls would load the equivalent flipp page with the new, inputted area code (my area code).

Now, I am very new to Scrapy, and HTTP requests/responses, so I am unsure if the link I found is the correct one (as in, the response with the new area code), or not.

This is the request I found:

https://gateflipp.flippback.com/bf/flipp/data?locale=en-us&postal_code=90210&sid=10775773055673477

I used an arbitrary postal code for the request (90210).

I suspect this is the incorrect request, but in the case that I am wrong, and this is correct:

How do I navigate to - flipp.com/weekly_ads/groceries from this request, while maintaining the new area code?

If this is incorrect:

How do I input a value for a javascript button, and get the result using Scrapy?

import scrapy

import requests

import json



class flippSpider(scrapy.Spider):

    name = "flippSpider"

    postal_code = "M1T2R8"

    start_urls = ["https://flipp.com/weekly_ads"]



    def parse(self, response): #Input value and simulate button click

        return Request() #Find http call to simulate button click with correct field/value parameters



    def parse_formrequest(self, response):

        yield scrapy.Request("https://flipp.com/weekly_ads/groceries", callback= self.parse_groceries)



def parse_groceries(self, response):

        flyers = 

        flyer_names = response.css("class.flyer-name").extract()

        for flyer_name in flyer_names:

            flyer = FlippspiderItem()

            flyer["name"] = flyer_name

            flyers.append(flyer)

            self.log(flyer["name"])

            print(flyer_name)

        return flyers

I expected to find the actual javascript button request within the XHR links but the one I found seems to be incorrect.

Edit: I do not want to use Selenium, it's slow, and I do not want a browser to pop up during execution of the spider.

I suspect this is the incorrect request, but in the case that I am wrong, and this is correct:

That is the correct URL to get the data powering that website; the things you see on screen when you go to flipp.com/weekly_ads/groceries is just packaging that data in HTML

How do I navigate to - flipp.com/weekly_ads/groceries from this request, while maintaining the new area code?

I am pretty sure you are asking the wrong question. You don't need to -- and in fact navigating to flipp.com/weekly_ads/groceries will 100% not do what you want anyway. You can observe that when you click on "Groceries", the content changes but the browser does not navigate to any new page, nor does it make a new XHR request. Thus, everything that you need is in that JSON. What is happening is they are using the flyers.*.categories that contains "Groceries" to narrow down the 129 flyers that are returned to just those related to Groceries.

As for "maintaining the new area code," it's a similar "wrong question" because every piece of data that is returned by that XHR is scoped to the postal code in question. Thus, you don't need to re-submit anything, and nor would I expect any data that comes back from your postal_code=90210 request to contain 30309 (or whatever) data.

Believe it or not, you're actually in a great place: you don't need to deal with complicated CSS or XPath queries to liberate the data from its HTML prison: they are kind enough to provide you with an API to their data. You just need to deal with unpacking the content from their structure into your own.