DEV Community

Devil Scrapes profile picture

Devil Scrapes

View all public scrapers @ https://apify.com/devilscrapes

Joined Joined on 
Bayt Jobs Scraper: the proxy country pin that was the block
devil_scrapes profile

Bayt Jobs Scraper: the proxy country pin that was the block

#webscraping #python #apify #proxy
Comments
3 min read
InfoJobs Spain Scraper: the proxy default that 407'd every run
devil_scrapes profile

InfoJobs Spain Scraper: the proxy default that 407'd every run

#webscraping #python #apify #proxy
Comments
3 min read
Fotocasa Scraper: the null field that hid every discounted listing
devil_scrapes profile

Fotocasa Scraper: the null field that hid every discounted listing

#webscraping #python #apify #realestate
Comments
3 min read
XING's job data ships as a JS object, not JSON. Here's how we parse it without breaking on the first undefined.
devil_scrapes profile

XING's job data ships as a JS object, not JSON. Here's how we parse it without breaking on the first undefined.

#python #webscraping #graphql #apify
Comments
3 min read
PropertyFinder.ae's category parameter wasn't what the spec assumed. We tested it before shipping.
devil_scrapes profile

PropertyFinder.ae's category parameter wasn't what the spec assumed. We tested it before shipping.

#python #webscraping #apify #realestate
Comments
3 min read
A green QA run hid an untested pagination path. Here's the check that caught it before we shipped.
devil_scrapes profile

A green QA run hid an untested pagination path. Here's the check that caught it before we shipped.

#python #webscraping #qa #apify
Comments
3 min read
A probe and a production scraper saw two different pages for the same missing Snapchat user
devil_scrapes profile

A probe and a production scraper saw two different pages for the same missing Snapchat user

#python #webscraping #apify #nextjs
Comments
4 min read
Per-profile or per-link: the billing unit on a link-in-bio scraper differs by 10x
devil_scrapes profile

Per-profile or per-link: the billing unit on a link-in-bio scraper differs by 10x

#python #webscraping #apify #api
Comments
4 min read
We tested the 'always pin the proxy country' rule instead of trusting it. Same data came back from two exit countries.
devil_scrapes profile

We tested the 'always pin the proxy country' rule instead of trusting it. Same data came back from two exit countries.

#python #webscraping #apify #proxy
Comments
4 min read
nbHits said 6,360. The API only let us page through 1,000 of them.
devil_scrapes profile

nbHits said 6,360. The API only let us page through 1,000 of them.

#python #webscraping #apify #algolia
Comments
4 min read
Filter the CFTC COT report for WHEAT and you get three different markets
devil_scrapes profile

Filter the CFTC COT report for WHEAT and you get three different markets

#python #opendata #finance #api
1
Comments 1
5 min read
Our two-state UCC run returned 200 rows, 127 filings, and zero from Colorado
devil_scrapes profile

Our two-state UCC run returned 200 rows, 127 filings, and zero from Colorado

#python #opendata #webscraping #api
Comments
5 min read
We doubled the memory, cut runtime 33%, and raised the cost 41%
devil_scrapes profile

We doubled the memory, cut runtime 33%, and raised the cost 41%

#python #devops #apify #webscraping
Comments
3 min read
We swept 288 scrapers for columns that are empty in every row. 35 had one.
devil_scrapes profile

We swept 288 scrapers for columns that are empty in every row. 35 had one.

#webscraping #python #testing #apify
Comments
4 min read
A note said the run scraped 0 rows. The platform's own record said 6.
devil_scrapes profile

A note said the run scraped 0 rows. The platform's own record said 6.

#python #api #fintech #debugging
Comments
4 min read
Our scraper returned 50 rows and 39 distinct IDs — count identities, not rows
devil_scrapes profile

Our scraper returned 50 rows and 39 distinct IDs — count identities, not rows

#python #webscraping #api #apify
Comments
4 min read
Our QA run passed with rows in the dataset and still proved nothing about pagination
devil_scrapes profile

Our QA run passed with rows in the dataset and still proved nothing about pagination

#python #webscraping #testing #apify
Comments
4 min read
Our Actor got 403'd on every page and still reported SUCCEEDED with zero rows
devil_scrapes profile

Our Actor got 403'd on every page and still reported SUCCEEDED with zero rows

#python #webscraping #debugging #apify
Comments
4 min read
A CONNECT 407 looked like a proxy-entitlement problem. It was a country pin with no group name.
devil_scrapes profile

A CONNECT 407 looked like a proxy-entitlement problem. It was a country pin with no group name.

#python #webscraping #apify #proxy
Comments
4 min read
Bing's ad-library search box accepts any query and returns the exact same ads
devil_scrapes profile

Bing's ad-library search box accepts any query and returns the exact same ads

#python #api #webscraping #marketing
Comments
4 min read
44 rows of Apple's 'Icon + Asset Ad' format, and not one had an asset. Bug or null?
devil_scrapes profile

44 rows of Apple's 'Icon + Asset Ad' format, and not one had an asset. Bug or null?

#python #webscraping #api #debugging
Comments
4 min read
We queried Wallapop for zzznonsensequery9999xyz. Zero results is what proved the search actually works.
devil_scrapes profile

We queried Wallapop for zzznonsensequery9999xyz. Zero results is what proved the search actually works.

#python #webscraping #testing #apify
Comments
4 min read
pump.fun's search parameter accepts any string and returns the same feed. We shipped without it.
devil_scrapes profile

pump.fun's search parameter accepts any string and returns the same feed. We shipped without it.

#python #api #crypto #webscraping
Comments
5 min read
OLX Brazil embeds a decoy listings array with the exact shape of the real one. We had to prove which was which.
devil_scrapes profile

OLX Brazil embeds a decoy listings array with the exact shape of the real one. We had to prove which was which.

#webscraping #python #brazil #classifieds
Comments
5 min read
We doubled Mercado Libre's scraper memory and clearance went from 50% to 90% - with zero code changes.
devil_scrapes profile

We doubled Mercado Libre's scraper memory and clearance went from 50% to 90% - with zero code changes.

#webscraping #python #ecommerce #mexico
Comments
6 min read
Kleinanzeigen has two page-2 URLs. One is a redirect trap. We verified the one we shipped actually advances.
devil_scrapes profile

Kleinanzeigen has two page-2 URLs. One is a redirect trap. We verified the one we shipped actually advances.

#webscraping #python #germany #classifieds
Comments
6 min read
We shelved a finished scraper because the site ignored its own search box. Now we test for it first.
devil_scrapes profile

We shelved a finished scraper because the site ignored its own search box. Now we test for it first.

#api #python #webscraping #crypto
Comments
5 min read
FurnishedFinder Scraper: caps every search at 50, and the fix isn't in the docs
devil_scrapes profile

FurnishedFinder Scraper: caps every search at 50, and the fix isn't in the docs

#webscraping #python #graphql #api
Comments
5 min read
Our QA harness certified this Actor at 4096 MB. It ships on 512.
devil_scrapes profile

Our QA harness certified this Actor at 4096 MB. It ships on 512.

#testing #python #webscraping #devops
Comments
3 min read
A run delivered 0 rows and still got billed $0.30 — the bug was ours, not Bayut's
devil_scrapes profile

A run delivered 0 rows and still got billed $0.30 — the bug was ours, not Bayut's

#webscraping #python #playwright #realestate
Comments
3 min read
The TikTok 'block' that was actually Camoufox running out of memory
devil_scrapes profile

The TikTok 'block' that was actually Camoufox running out of memory

#webscraping #python #api #tiktok
Comments
3 min read
A datacenter IP gets 0 listings; a residential one gets 12. Same build.
devil_scrapes profile

A datacenter IP gets 0 listings; a residential one gets 12. Same build.

#webdev #python #scraping #api
Comments 1
3 min read
Our scraper went 0-for-19 against Cloudflare. One boolean took it to 9-for-10.
devil_scrapes profile

Our scraper went 0-for-19 against Cloudflare. One boolean took it to 9-for-10.

#webscraping #python #devops #api
Comments
5 min read
We cleared PeoplePerHour 4/4 locally. The first cloud run got HTTP 202 from every URL.
devil_scrapes profile

We cleared PeoplePerHour 4/4 locally. The first cloud run got HTTP 202 from every URL.

#webscraping #python #devops #api
Comments
4 min read
A Dockerfile typo turned 33 real Depop rows into a silent, SUCCEEDED zero
devil_scrapes profile

A Dockerfile typo turned 33 real Depop rows into a silent, SUCCEEDED zero

#webscraping #python #docker #ecommerce
Comments
4 min read
An isinstance(str) check that was always False capped every Facebook Page at exactly 3 posts
devil_scrapes profile

An isinstance(str) check that was always False capped every Facebook Page at exactly 3 posts

#webscraping #python #api #debugging
Comments
4 min read
Alibaba renamed one JavaScript variable and our price field silently dropped to 0%
devil_scrapes profile

Alibaba renamed one JavaScript variable and our price field silently dropped to 0%

#webscraping #python #javascript #ecommerce
Comments
5 min read
A 586 KB HTTP 200 from YouTube that contains zero videos
devil_scrapes profile

A 586 KB HTTP 200 from YouTube that contains zero videos

#webscraping #python #api #youtube
Comments
4 min read
Our test asked for 4 branches, ran 1, and reported PASS
devil_scrapes profile

Our test asked for 4 branches, ran 1, and reported PASS

#testing #python #webscraping #devops
Comments
4 min read
LinkedIn has an /api/ endpoint that needs no login and returns HTML
devil_scrapes profile

LinkedIn has an /api/ endpoint that needs no login and returns HTML

#webscraping #python #api #linkedin
Comments
3 min read
Our release gate read a docstring and blocked the deploy over a word
devil_scrapes profile

Our release gate read a docstring and blocked the deploy over a word

#devops #python #testing #webscraping
Comments
4 min read
No introspection, no allowlist: reconstructing Whatnot's GraphQL queries from a compiled AST
devil_scrapes profile

No introspection, no allowlist: reconstructing Whatnot's GraphQL queries from a compiled AST

#graphql #python #webscraping #api
Comments
4 min read
Zero results, five rows: the Flippa search response that lies about matching nothing
devil_scrapes profile

Zero results, five rows: the Flippa search response that lies about matching nothing

#python #webscraping #api #webdev
Comments
4 min read
A 200 that isn't reachability: SpareRoom's soft-redirect and the empty run nobody notices
devil_scrapes profile

A 200 that isn't reachability: SpareRoom's soft-redirect and the empty run nobody notices

#python #webscraping #api #webdev
Comments
5 min read
A 404 is not a failure: what Discord's invite endpoint taught us about billing
devil_scrapes profile

A 404 is not a failure: what Discord's invite endpoint taught us about billing

#python #api #webscraping #discord
5
Comments
5 min read
A 25%-failing test is not flaky. It's a bug reporting itself honestly.
devil_scrapes profile

A 25%-failing test is not flaky. It's a bug reporting itself honestly.

#python #testing #webscraping #debugging
5
Comments
4 min read
A bare useApifyProxy: true resolves to datacenter. Patreon 403s every one of those.
devil_scrapes profile

A bare useApifyProxy: true resolves to datacenter. Patreon 403s every one of those.

#webscraping #python #api #cloudflare
Comments
4 min read
grep '"aggregate_rating":' finds zero matches on Zomato. The field is there, just double-escaped.
devil_scrapes profile

grep '"aggregate_rating":' finds zero matches on Zomato. The field is there, just double-escaped.

#webscraping #python #javascript #api
Comments
4 min read
?q=chairs on Chairish isn't a search - it's a redirect to a category page wearing a search URL.
devil_scrapes profile

?q=chairs on Chairish isn't a search - it's a redirect to a category page wearing a search URL.

#webscraping #python #api #ecommerce
Comments
5 min read
?q=chairs on Chairish isn't a search - it's a redirect to a category page wearing a search URL.
devil_scrapes profile

?q=chairs on Chairish isn't a search - it's a redirect to a category page wearing a search URL.

#webscraping #python #api #ecommerce
Comments
5 min read
?page=2 on Immowelt returns page 1, byte for byte. It's a button, not a route.
devil_scrapes profile

?page=2 on Immowelt returns page 1, byte for byte. It's a button, not a route.

#webscraping #python #api #realestate
Comments
5 min read
window.__NUXT__ is not JSON. It's a function you have to run first.
devil_scrapes profile

window.__NUXT__ is not JSON. It's a function you have to run first.

#webscraping #python #javascript #api
Comments
4 min read
Our build wave shipped four scrapers that scraped nothing
devil_scrapes profile

Our build wave shipped four scrapers that scraped nothing

#testing #python #webscraping #devops
Comments
4 min read
A one-step probe told us YouTube was blocked. It was asking the wrong endpoint.
devil_scrapes profile

A one-step probe told us YouTube was blocked. It was asking the wrong endpoint.

#webscraping #python #api #youtube
Comments
4 min read
The top-earning product on Indie Hackers reports $10,000,000,000,000,000 a month
devil_scrapes profile

The top-earning product on Indie Hackers reports $10,000,000,000,000,000 a month

#webscraping #python #saas #data
Comments
3 min read
Grailed's __NEXT_DATA__ is empty. The listings come from Algolia, and the key is on the page.
devil_scrapes profile

Grailed's __NEXT_DATA__ is empty. The listings come from Algolia, and the key is on the page.

#webscraping #python #algolia #nextjs
Comments
3 min read
Coursera ships two big JSON blobs on every search page. Neither has the results.
devil_scrapes profile

Coursera ships two big JSON blobs on every search page. Neither has the results.

#webscraping #python #html #api
Comments
4 min read
dict.get(key, [None])[0] is not safe, and our green test suite agreed with us
devil_scrapes profile

dict.get(key, [None])[0] is not safe, and our green test suite agreed with us

#python #testing #webscraping #debugging
Comments
4 min read
Pinterest's ad library endpoint was hiding behind one missing curl flag
devil_scrapes profile

Pinterest's ad library endpoint was hiding behind one missing curl flag

#webscraping #python #api #marketing
Comments
4 min read
Google News RSS has no cursor, so the query IS the cursor
devil_scrapes profile

Google News RSS has no cursor, so the query IS the cursor

#news #webscraping #python #api
Comments
4 min read
loading...