Scrape nested page puppeteer

I would like to know how to scrape data located in nested pages. Here's an example I tried to build however couldn't make it work. The idea is to go to https://dev.to/, click the question and grab its title. Then go back and redo the process for the next question.

const puppeteer = require("puppeteer");    (async () => {    const browser = await puppeteer.launch();    const page = await browser.newPage();    await page.goto("https://dev.to/");      try {      const selectors = await page.$$(".crayons-story > a");        for (const post of selectors) {        await Promise.all([          page.waitForNavigation(),          post.click(),          page.goBack(),        ]);      }    } catch (error) {      console.log(error);    } finally {      browser.close();    }  })();  

When I run this code, I get Error: Node is either not visible or not an HTMLElement

Edit: The code is missing a piece where grabs the title, but is enough for the purpose.

https://stackoverflow.com/questions/66738589/scrape-nested-page-puppeteer March 22, 2021 at 07:21AM