This blog is a comprehensive guide on deep scraping using Byteline’s managed web scraper.  If you're looking to perform deep scraping efficiently, Byteline is your go-to tool. Our managed web scraper service is designed to simplify the deep scraping process by configuring and maintaining the scraper for you. The deep scraping is performed by simply picking the categories and the fields for which you want the data.

This article will walk you through the concept of categories, the scraping process, and how to utilize the data.

Understanding the Concept of Categories

Byteline’s managed web scraper works on the concept of categories. Categories are essentially groupings of similar items on a website. For instance, if you want to scrape data about AI companies from Aixploria’s AI categories, each category, shown below, represents a collection of companies.

Clicking on a category reveals a list of companies, and our goal is to scrape each company’s name, link, and description.

Most websites have similar structures; for example, cheffsupplies.ca has collections that function as categories.

How to Perform Deep Scraping with Byteline Managed Scraper

Step 1: Select Categories for Deep Scraping

The first step in deep scraping is selecting the categories you wish to scrape. But before that, you need to request a site to be added, which is typically processed within hours. Once your request is completed, you can add a scraper for that site and choose the desired categories.

Step 2: Choose Fields to Scrape

After selecting the categories, the next step is to pick the specific fields you want to scrape, such as the company's name, link, and description.

Step 3: Verify Data Quality with Test Run

The Next button on the above screen takes you to the test run results to verify the data quality. This ensures the data meets your requirements before proceeding with the entire scrape.

Downloading the Scraped Data as a CSV

To download the CSV of the entire data, you can click on the “Export entire scrape” link on the test run results page. This takes you to the scraper dashboard page from where you can download the entire dataset as a CSV file, once the deep scraping process is complete.

Automating the Deep Scraping Process

For advanced usage, such as scheduling regular scraping tasks, you can leverage our Workflow Automation product. Our managed scraper integrates natively with our automation platform.

You can easily create an automation flow by either using the “Automate” button from the above screen or the “Use in Automation” button from the test run results.

You can further customize the automation flow by modifying the trigger nodes and adding more action nodes, enabling functionalities like sending notifications via Slack.

Requesting a New Site for Deep Scraping

To start scraping a new site, you need to request its addition to Byteline’s managed scraper. This can be easily done from the site request section on the scraper dashboard.

Support

If you have any questions or need assistance, feel free to use the chat tool on the bottom right of our site.

‍