Using Scrapfly Proxy

PriceFetchWP supports Scrapfly as an optional proxy and anti-bot provider.

Scrapfly can be useful when an external website blocks a normal Direct Fetch, requires stronger anti-bot handling, or needs browser-style rendering before the price becomes available.

This guide explains how to connect Scrapfly to PriceFetchWP, assign it to a URL Entry, and configure the available options.

– What Is Scrapfly?

Scrapfly is a third-party web scraping and proxy service.

Instead of PriceFetchWP connecting directly from your WordPress instance to the external website, the request is sent through Scrapfly:

Your WordPress Instance
        ↓
Scrapfly
        ↓
External Website

Scrapfly makes the final request using its own infrastructure and returns the page content to PriceFetchWP.

This can help with websites that:

• block normal server-based requests
• use anti-bot protection
• restrict certain server IP addresses
• require browser-style requests
• load price information using JavaScript

Scrapfly is optional. If a website works reliably using Direct Fetch, there is generally no need to use a proxy for that URL Entry.

– Getting a Scrapfly API Key

To use Scrapfly with PriceFetchWP, you will first need a Scrapfly account and API key.

Create or sign in to your Scrapfly account and locate your API key.

Once you have the key, return to:

PriceFetchWP → Settings → Proxy Integration

Select the Scrapfly tab.

– Connect Scrapfly to PriceFetchWP

Enter your Scrapfly API key into the API key field.

Click: Verify

PriceFetchWP will make a test request to confirm that the API key can be used.

If verification is successful, click: Quick Save

Your API key and Scrapfly settings will now be stored in PriceFetchWP.

Tip: Quick Save saves the Proxy Integration settings without requiring the entire settings page to refresh.

– Assign Scrapfly to a URL Entry

Connecting your Scrapfly account does not automatically send every PriceFetchWP request through Scrapfly.

Proxy providers are selected per URL Entry.

Find the URL Entry you want to use with Scrapfly and open: Advanced Options

Enable the proxy option and select: Scrapfly

Save your settings.

PriceFetchWP will now use Scrapfly when fetching prices for that URL Entry. A Proxy: Scrapfly indicator will now show on this entry.

This allows websites that work normally through Direct Fetch to remain direct while Scrapfly is only used where needed.

Affiliate and Proxy modes cannot be used on the same URL Entry. If Affiliate mode is active, Scrapfly cannot be selected.

– Anti-Scraping Protection (ASP)

Anti-Scraping Protection (ASP) is one of Scrapfly’s primary features for websites that restrict automated requests.

When ASP is enabled, Scrapfly can automatically adjust how the request is made based on the website being fetched.

Depending on the target, Scrapfly may change or improve things such as:

• proxy routing
• browser usage
• request headers
• cookies
• request fingerprinting
• other anti-bot handling

This makes ASP useful when a normal request is returning a blocked response such as a 403, challenge page, or other access-related result.

When should I use ASP?

Consider enabling ASP when:

• Direct Fetch is blocked
• a basic Scrapfly request is still blocked
• the website uses stronger anti-bot protection
• Test Entry shows that the expected product page is not being returned

ASP can require additional Scrapfly resources, which means the credit cost of a request may increase.

For this reason, it is generally best to use Scrapfly and ASP only on URL Entries that need them.

– JavaScript Rendering

JavaScript Rendering tells Scrapfly to load the page using a cloud browser and execute the page’s JavaScript before returning the content to PriceFetchWP.

This can help when the price is not present in the initial HTML and is instead inserted or changed after the page begins loading.

When should I enable JavaScript Rendering?

Consider enabling it when:

• the page fetch succeeds
• the selector appears correct
• the price is visible in a normal browser
• but PriceFetchWP cannot find the price in the fetched content

JavaScript Rendering uses additional Scrapfly resources and can make requests slower and more expensive.

It should normally be enabled only when a website actually requires it.

Note: When ASP is enabled, Scrapfly may automatically use browser resources when needed. Because of this, manually enabling JavaScript Rendering is not always necessary for every ASP request.

– Understanding Scrapfly Credit Costs

Scrapfly uses a credit-based billing system.

Not every request necessarily costs the same number of credits.

A simple request may use a standard datacenter proxy and cost very little, while a more difficult website may require additional resources.

For example, Scrapfly’s current pricing model includes:

• Datacenter request: 1 credit
• Residential proxy request: 25 credits
• JavaScript Rendering: adds 5 credits

ASP can automatically change the resources required for a particular website. For example, it may determine that a residential proxy or browser is necessary to successfully retrieve the page.

This means two PriceFetchWP URL Entries using Scrapfly can have very different request costs.

A website that is easy to retrieve may cost:

1 credit

while another website with stronger protection could require:

25 credits

or more depending on the resources Scrapfly uses.

This does not mean PriceFetchWP necessarily made 25 separate requests. It can represent the cost of the resources Scrapfly used for a single fetch.

– Viewing Request Cost

When Scrapfly reports the credit cost of a request, PriceFetchWP can display that information in the Test Entry results.

This makes it easier to see whether a particular external website is inexpensive or more resource-intensive to retrieve.

You can also review detailed request usage and credit information from your Scrapfly account.

Because ASP can adapt to changes in an external website’s anti-bot protection, request costs may change over time.

– Testing a URL with Scrapfly

After assigning Scrapfly to a URL Entry, use: Test Entry

Review the result for:

• fetch status
• fetch mode
• Scrapfly result
• credit cost (this feature is being tested)
• selector matches
• regular price
• sale price
• reported errors
• raw HTML

If Scrapfly successfully retrieves the page but PriceFetchWP still cannot find the expected price, review the price selector.

– Direct Fetch Works — Should I Still Use Scrapfly?

Usually, no.

If PriceFetchWP can reliably retrieve the correct price using Direct Fetch, continuing to use Direct Fetch is the best option.

Scrapfly is most useful for URL Entries where direct requests are blocked or additional rendering is required.

⚠️ Common Issues

API key does not verify

Check that:

• the API key was copied correctly
• there are no extra spaces
• your Scrapfly account is active
• your account has available usage

Then try Verify again.

Scrapfly returns the page but the price is not found

A successful request does not necessarily mean the configured selector matches the expected price.

Check:

• XPath, ID, Class, or Script selector
• whether the expected price exists in the returned content
• whether JavaScript Rendering is needed
• whether the external website recently changed its page structure

Run Test Entry again after making changes.

Direct Fetch fails but Scrapfly works

This can happen when an external website blocks requests from your WordPress hosting server or requires additional anti-bot handling.

If Scrapfly consistently retrieves the expected page and price, you can leave Scrapfly assigned to that URL Entry.

ASP is enabled but the request still fails

ASP can improve compatibility with many protected websites, but it does not guarantee that every website can be successfully retrieved.

External websites may change their security systems at any time, and some pages may depend on additional cookies, sessions, location, interaction, or other site-specific behavior.

One website uses many more credits than another

Unfortunately, this is normal.

Scrapfly may need different proxy or browser resources for different websites.

Use the reported credit cost and your Scrapfly monitoring dashboard to understand the usage for each target.

– Scrapfly Usage

Scrapfly is a third-party service and has its own plans, credit limits, usage policies, and pricing.

PriceFetchWP does not control Scrapfly billing or account limits.

Because Scrapfly’s Anti-Scraping Protection can dynamically change the resources used for a request, the credit cost for a particular website may also change over time.

For current billing, credit, and account information, refer to your Scrapfly account and official Scrapfly documentation.