how to Scrap data from a webPage when the next Button doesn’t work?

Web scraping is a powerful technique that enables extracting data from websites for various purposes, such as market research, competitive analysis, and data-driven decision-making. UiPath, a leading Robotic Process Automation (RPA) tool, offers a comprehensive platform for automating web scraping tasks. When dealing with websites that employ lazy loading, where content is loaded dynamically as the user scrolls or interacts with the page, UiPath’s capabilities can be harnessed to navigate and extract data efficiently.

In this article, we will explore the world of web scraping in UiPath and delve into how it can be seamlessly integrated with lazy-loading websites

In this article, we discuss one type of lazy-loading website. How to load all pages and scrap the data.

if does not work next button continues scraping the data. does not have any endpoint.

An example problem is (https://datafit.ai/cat/email) website extracts the data in different pages how do load all the pages

This problem will solve by using Uipath using various activities like Sendhot keys, anchor base, etc.

this problem, solving the anchor base using will complete the trial.

solution:

• step 1:

1. open the uipath studio and create a new sequence
2. drag and drop the open browser activity and enter the URL and other properties given.
3. open browser activity properties and select the browser type.

Step 2:

1. By using the anchor base activity we find how many pages are available On this website
2. we drag and drop the anchor base activity.in this anchor base activity, we use the find Element and get attribute activity

1. In this activity, we first find the right arrow symbol using the find element activity.
2. after that, we use the get attribute activity(aaname) to find how many pages are available on the website.
3. find the page number store the new variable to save the int type.

Step 3:

1. assign activity using assign a new variable give the default value is 1.

step 4:

we use the DO while Activity enters every page and scrapes the data.

1. we are using the dynamic variable and click activity to enter the page and scrape the data.
2. the dynamic variable is an increment by one in the body loop.
3. here give the condition dynamic variable <= max pages.

Step 5:

in this body block to data scraping activity use extract the required data.

web scraping in UiPath, combined with the utilization of lazy loading techniques, opens up a world of opportunities for efficient and effective data extraction from dynamic websites

Navigating the complexities of web pages with never-ending next buttons in the context of data extraction can be a daunting task. However, armed with the right strategies and tools, such as UiPath, it is possible to conquer this challenge and effectively extract data from these pages.