DEV Community

Devil Scrapes profile picture

Devil Scrapes

View all public scrapers @ https://apify.com/devilscrapes

Joined Joined on 
The Location header is relative more often than your link checker thinks
devil_scrapes profile

The Location header is relative more often than your link checker thinks

#webscraping #python #seo #http
Comments
4 min read
Cloudflare's /security.txt 404s with Content-Type: text/plain
devil_scrapes profile

Cloudflare's /security.txt 404s with Content-Type: text/plain

#security #webscraping #python #seo
Comments
4 min read
A domain that doesn't exist returns HTTP 200 OK
devil_scrapes profile

A domain that doesn't exist returns HTTP 200 OK

#dns #api #python #devops
Comments
4 min read
Wikimedia blocks python-requests and waves through an empty User-Agent
devil_scrapes profile

Wikimedia blocks python-requests and waves through an empty User-Agent

#python #api #webscraping #debugging
Comments
4 min read
Docker Hub's API 403s Chrome and 200s Firefox — same URL, same second
devil_scrapes profile

Docker Hub's API 403s Chrome and 200s Firefox — same URL, same second

#docker #webscraping #python #api
Comments
4 min read
OData's worst failure isn't the 400 — it's the query that quietly matches nothing
devil_scrapes profile

OData's worst failure isn't the 400 — it's the query that quietly matches nothing

#api #python #webscraping #data
Comments
3 min read
The FDIC's API returns data.data, and ASSET is in thousands
devil_scrapes profile

The FDIC's API returns data.data, and ASSET is in thousands

#api #python #fintech #webscraping
Comments
4 min read
Our container reported SUCCEEDED, ran nothing, and scored 100% on every dashboard
devil_scrapes profile

Our container reported SUCCEEDED, ran nothing, and scored 100% on every dashboard

#python #docker #webscraping #devops
Comments
4 min read
UN Comtrade's free tier: 500 rows, no pagination, and a silent 429
devil_scrapes profile

UN Comtrade's free tier: 500 rows, no pagination, and a silent 429

#python #api #webscraping #datascience
Comments
4 min read
EDGAR's total count lies to you, and it says so
devil_scrapes profile

EDGAR's total count lies to you, and it says so

#python #api #webscraping #fintech
Comments
3 min read
Most EDGAR tools stop at the filing. The trades are one layer deeper.
devil_scrapes profile

Most EDGAR tools stop at the filing. The trades are one layer deeper.

#python #api #webscraping #fintech
Comments
3 min read
NHTSA decodes any 17 characters you give it, valid or not
devil_scrapes profile

NHTSA decodes any 17 characters you give it, valid or not

#python #api #webscraping #automotive
Comments
3 min read
ORCID's search API has a ceiling it never tells you about
devil_scrapes profile

ORCID's search API has a ceiling it never tells you about

#python #api #webscraping #opensource
Comments
4 min read
The NWS API makes you resolve a grid before it tells you the weather
devil_scrapes profile

The NWS API makes you resolve a grid before it tells you the weather

#python #api #webscraping #datascience
Comments
4 min read
Our Chrome fingerprint was the thing getting us blocked
devil_scrapes profile

Our Chrome fingerprint was the thing getting us blocked

#python #api #webscraping #fintech
Comments
3 min read
Four levels of JSON to say the word Leasehold
devil_scrapes profile

Four levels of JSON to say the word Leasehold

#python #api #opendata #realestate
Comments
4 min read
Your demo input has an expiry date (and your tests can't see it)
devil_scrapes profile

Your demo input has an expiry date (and your tests can't see it)

#python #api #webscraping #testing
Comments
4 min read
openFDA's docs say +AND+. Send that and you get a 500.
devil_scrapes profile

openFDA's docs say +AND+. Send that and you get a 500.

#python #api #healthcare #webscraping
Comments
4 min read
Our browser impersonation made the API return HTML (and 29 tests missed it)
devil_scrapes profile

Our browser impersonation made the API return HTML (and 29 tests missed it)

#python #api #testing #webscraping
Comments
3 min read
Don't scrape data.gov. Speak CKAN.
devil_scrapes profile

Don't scrape data.gov. Speak CKAN.

#python #api #opendata #webscraping
Comments
4 min read
The offset parameter that silently does nothing (and would have billed for it)
devil_scrapes profile

The offset parameter that silently does nothing (and would have billed for it)

#python #webscraping #seo #api
Comments
4 min read
rdap.org answers 302, and a client that doesn't follow it sees nothing
devil_scrapes profile

rdap.org answers 302, and a client that doesn't follow it sees nothing

#python #webscraping #dns #api
Comments
4 min read
255 App Store scrapers and not one of them tracks keyword rank
devil_scrapes profile

255 App Store scrapers and not one of them tracks keyword rank

#python #webscraping #aso #api
Comments
4 min read
A SUCCEEDED run with zero log lines: the probe that never executed
devil_scrapes profile

A SUCCEEDED run with zero log lines: the probe that never executed

#python #webscraping #debugging #ecommerce
Comments
4 min read
One retry budget, two unrelated failures: how a scraper blamed the wrong wall
devil_scrapes profile

One retry budget, two unrelated failures: how a scraper blamed the wrong wall

#python #webscraping #debugging #leadgen
Comments
4 min read
Two green runs an hour apart is not reliability if the scraper is 33% reliable
devil_scrapes profile

Two green runs an hour apart is not reliability if the scraper is 33% reliable

#python #webscraping #testing #debugging
Comments
4 min read
Same API standard, 57 rivals, nobody consolidated: harvesting every Socrata portal at once
devil_scrapes profile

Same API standard, 57 rivals, nobody consolidated: harvesting every Socrata portal at once

#opendata #api #python #webscraping
Comments
4 min read
The 404 Was Our URL, Not Idealista's Wall
devil_scrapes profile

The 404 Was Our URL, Not Idealista's Wall

#python #webscraping #realestate #api
Comments
4 min read
We debugged the same bug for 3 days because it was never actually deployed
devil_scrapes profile

We debugged the same bug for 3 days because it was never actually deployed

#webscraping #python #devops #debugging
Comments
7 min read
A SUCCEEDED run with 3 rows, and every row was fabricated
devil_scrapes profile

A SUCCEEDED run with 3 rows, and every row was fabricated

#webscraping #python #testing #api
Comments
6 min read
Open Brewery DB Has Two Ways to Say 'Zero Results'
devil_scrapes profile

Open Brewery DB Has Two Ways to Say 'Zero Results'

#python #api #opendata #webscraping
Comments
4 min read
UK Police's Crime API Returns 404 When the Answer Is Zero
devil_scrapes profile

UK Police's Crime API Returns 404 When the Answer Is Zero

#python #api #opendata #govtech
Comments
4 min read
The proxy wall that was a leftover environment variable
devil_scrapes profile

The proxy wall that was a leftover environment variable

#webscraping #python #debugging #devops
Comments
4 min read
USAspending.gov Returns a Fake 500 to Block Scrapers — Here's the Fix
devil_scrapes profile

USAspending.gov Returns a Fake 500 to Block Scrapers — Here's the Fix

#python #api #govtech #webscraping
Comments
3 min read
openFDA Tells You the Outcome Was 5
devil_scrapes profile

openFDA Tells You the Outcome Was 5

#api #python #opendata #healthcare
Comments
3 min read
Our Scraper Said Every EU Tender Was Titled "F"
devil_scrapes profile

Our Scraper Said Every EU Tender Was Titled "F"

#api #python #testing #webscraping
Comments
3 min read
An API With No Pagination Is a Billing Problem
devil_scrapes profile

An API With No Pagination Is a Billing Problem

#api #python #opendata #webscraping
Comments
3 min read
The French Registry Says 10,000 Results. It Means the Cap.
devil_scrapes profile

The French Registry Says 10,000 Results. It Means the Cap.

#api #python #opendata #webscraping
Comments
3 min read
One Medicare API, Four Datasets, Three Renamed Fields
devil_scrapes profile

One Medicare API, Four Datasets, Three Renamed Fields

#api #python #opendata #webscraping
Comments
4 min read
EPA ECHO Reports a Rejected Search as HTTP 200
devil_scrapes profile

EPA ECHO Reports a Rejected Search as HTTP 200

#python #api #opendata #webscraping
Comments
4 min read
UK Contracts Finder: how to scrape public-sector tenders without losing rows to bad GUID parsing
devil_scrapes profile

UK Contracts Finder: how to scrape public-sector tenders without losing rows to bad GUID parsing

#webscraping #python #opendata #api
Comments
3 min read
Only OSM Nodes Carry Coordinates — Ways and Relations Need out center
devil_scrapes profile

Only OSM Nodes Carry Coordinates — Ways and Relations Need out center

#python #api #openstreetmap #webscraping
Comments
3 min read
NIH RePORTER Caps You at Offset 14,999 No Matter What Matches
devil_scrapes profile

NIH RePORTER Caps You at Offset 14,999 No Matter What Matches

#python #api #opendata #webscraping
Comments
3 min read
Finland's Company API Accepts Your Filter and Silently Ignores It
devil_scrapes profile

Finland's Company API Accepts Your Filter and Silently Ignores It

#python #api #webscraping #opendata
Comments
3 min read
NVD gives you 5 requests per 30 seconds, and CVSS in four schemas
devil_scrapes profile

NVD gives you 5 requests per 30 seconds, and CVSS in four schemas

#security #api #python #devops
Comments
2 min read
The Rebag URL everyone scrapes is the one robots.txt disallows
devil_scrapes profile

The Rebag URL everyone scrapes is the one robots.txt disallows

#webscraping #python #shopify #ecommerce
Comments
3 min read
Hipcamp Scraper: Three Listing Types That Never Share a Schema
devil_scrapes profile

Hipcamp Scraper: Three Listing Types That Never Share a Schema

#webscraping #python #api #datascience
Comments
4 min read
Fashionphile Scraper: Its Own Search Page Hides Real Prices Behind a Null Block
devil_scrapes profile

Fashionphile Scraper: Its Own Search Page Hides Real Prices Behind a Null Block

#webscraping #python #ecommerce #api
Comments
4 min read
Our build wave shipped four scrapers that scraped nothing
devil_scrapes profile

Our build wave shipped four scrapers that scraped nothing

#testing #python #webscraping #devops
Comments 1
4 min read
The Zenodo API Caps You At 25 Results — Above That It's Just 400
devil_scrapes profile

The Zenodo API Caps You At 25 Results — Above That It's Just 400

#webscraping #python #api #apify
Comments
3 min read
Mercari Sold Listings: When 200 OK And 403 Both Lie To You
devil_scrapes profile

Mercari Sold Listings: When 200 OK And 403 Both Lie To You

#webscraping #python #ecommerce #apify
Comments
4 min read
Goodreads Has No Book API — So Its Site Is One Big GraphQL Cache
devil_scrapes profile

Goodreads Has No Book API — So Its Site Is One Big GraphQL Cache

#webscraping #python #graphql #apify
Comments
4 min read
Gumroad has no public API, but every page inlines its own JSON
devil_scrapes profile

Gumroad has no public API, but every page inlines its own JSON

#webscraping #python #apify #api
Comments
4 min read
4 states, 1 open-data API, and the pagination bug that splits a company in half
devil_scrapes profile

4 states, 1 open-data API, and the pagination bug that splits a company in half

#webscraping #python #apify #data
Comments
4 min read
Scraping Lu.ma: one JSON blob answers list pages, event pages, and address privacy
devil_scrapes profile

Scraping Lu.ma: one JSON blob answers list pages, event pages, and address privacy

#webscraping #python #apify #data
Comments
3 min read
YouTube's innertube player API: 200 OK doesn't mean you got the data
devil_scrapes profile

YouTube's innertube player API: 200 OK doesn't mean you got the data

#webscraping #python #apify #youtube
Comments
4 min read
5 states, 2 working filters: scraping US childcare license registries
devil_scrapes profile

5 states, 2 working filters: scraping US childcare license registries

#python #api #opendata #leadgen
Comments
4 min read
Suumo.jp: one site, two completely different schemas for rent vs. buy listings
devil_scrapes profile

Suumo.jp: one site, two completely different schemas for rent vs. buy listings

#python #webscraping #realestate #japan
Comments
4 min read
Same API standard, four incompatible schemas: scraping state cosmetology license registries
devil_scrapes profile

Same API standard, four incompatible schemas: scraping state cosmetology license registries

#python #api #opendata #leadgen
Comments
4 min read
Google Trends API: the 200 OK that means you got soft-blocked
devil_scrapes profile

Google Trends API: the 200 OK that means you got soft-blocked

#python #api #webscraping #trends
Comments
4 min read
loading...