curl www.google.com
, Google has many ways to know that you are not a human (for example by looking at the headers). Headers are small pieces of information that go with every HTTP request that hits the servers. One of those pieces of information precisely describes the client making the request, This is the infamous “User-Agent” header. Just by looking at the “User-Agent” header, Google knows that you are using cURL. If you want to learn more about headers, the Wikipedia page is great. As an experiment, just go over here. This webpage simply displays the headers information of your request.requests
module doesn't support changing the TLS fingerprint out of the box. Here are a few resources to change your TLS version and cypher suite in your favorite language:Anti-bot technique | Counter measure | Supported by ScrapingBee |
---|---|---|
Browser Fingerprinting | Headless browsers | ✅ |
IP-rate limiting | Rotating proxies | ✅ |
Banning Data center IPs | Residential IPs | ✅ |
TLS Fingerprinting | Forge and rotate TLS fingerprints | ✅ |
Captchas on suspicious activity | All of the above | ✅ |
Systematic Captchas | Captchas-solving tools and services | ❌ |