Scraping URL or SKU dynamically with JS

Ask the community
,
1

I’m trying to integrate consumer generated content (reviews and ratings linking to other websites). To do this I want to add JS coding to our product description pages. Adding coding in to an iFrame won’t work because this creates a whole new section within the body of the page, it seems impossible to add script that has access to parts of the page that live outside this iFrame. I want to somehow scrape the product ID or SKU dynamically from the page using javascript, so that we can populate two injection divs with linked data. Is this possible using Wix code?

Any tips? Thanks for helping out!

2

You can use the fetch in Javascript to grab the page and then ordinary string manipulations to grab the stuff you want inside that string that will be returned from the fetch promise. Just make sure you are allowed to scrape the content so you won’t find yourself in trouble. I have done that several times and found myself in trouble scraping content that was copyrighted of the sites owner :slight_smile:

3

Hi Andreas,

Can you elaborate more on “use the fetch in Javascript”, please? I’m working on the same project as Charles, and I want to make clear that:

  1. He is using a store app to manage their online product catalog, and this also affects the product page template that we’re looking to add this code to.
  2. He will be scraping his OWN site - No other way to pass the URL or product SKU into dynamic code.
  3. In attempts to put code inside an iFrame or inside the $w.onReady function, we either get an error when trying to use document.getElementByID (for the SKU) or we get a bogus URL (probably the iFrame’s URL) instead of the main product page URL when trying to grab that data instead. Essentially, we need to be able to use Javascript to grab the SKU OR the page URL, parse the value and then insert it into a new attribute on an existing div.
4

Hello Charles and Gem,

As Andreas mentioned Wix has a a fetch() api that allows you to scrape data from other websites/api’s
You can read more about how to use fetch here - link
Here is a tutorial of an example of one way to use fetch - link

Hope this helps,
Majd

5

Hi Majd,

Thank you for your response, but the page we need to scrape is the same page where the script is going to run. Using the Wix Code Reference, I figured out how to grab the last portion of URL path.

Now we’re stuck on this problem:
We have no way to add a an div with a specific ID using the Wix editor, and no way to use Javascript to dynamically insert or modify an attribute on said div (attribute needs to be called “data-bv-productid”).

Here’s the code we have so far:

import wixLocation from ‘wix-location’;

$w.onReady( function () {
// This block works fine
let path = wixLocation.path;
let product_id = path[path.length - 1];
console.log("BVTEST - Path: ", path);
console.log("BVTEST - Product ID: ", product_id);

// get the summary div by id and inject the appropriate BV-specific attributes including product id
let bv_summary_section = $w(“#bvSummarySection”);
//The rest of this block fails b/c we can’t use setAttribute!
bv_summary_section.setAttribute(“data-bv-show”, “rating_summary”);
bv_summary_section.setAttribute(“data-bv-productid”, product_id);

// get the reviews div by id and inject the appropriate BV-specific attributes including product id
let bv_reviews_section = $w(“#bvReviewsSection”);
//The rest of this block fails b/c we can’t use setAttribute!
bv_reviews_section.setAttribute(“data-bv-show”, “reviews”);
bv_reviews_section.setAttribute(“data-bv-productid”, product_id);
});

6

Hey
I think you should not try to do things in Wix Code that won’t work. make it easy for you, make the HTML / Javascript whatever you need in a standalone HTML document. Insert that into a HTML Component in your Wix Code Page and then communicate between these two so you can send stuff to that and receive stuff from that. Then you have the best of two worlds.

7

Hi Andreas, can you elaborate on your scraping explanation? I am very familiar with Fetch when using third party API’s, but not so much on just grabbing an ordinary web page then the string manipulations?

Let’s just say I want to get the name of all the projects on this goFundme page: “https://www.gofundme.com/discover”. I assume I cannot return it as a JSON, so text?. Is it just the following, then it returns something that I can loop through and extract?

 
$w("#fetch").onClick( () => {
        fetch("https://www.gofundme.com/discover", {method: 'get'})
  .then(httpResponse => httpResponse.text())
  .then(text => console.log(text));
    
});

Thanks!