Proxies for AI training data collection
Building a dataset from the public web usually means weeks of crawling in several languages. With the HiProxy unlimited residential plan, you fix the cost for that whole period up front and send requests from 60 countries.
Why is per GB billing a poor fit for dataset crawls?
When a crawl covers thousands of domains, you cannot control page sizes. On a per GB bill, a few image heavy sites can push the budget far off plan.
The unlimited plan is priced only by period, connection count and bandwidth. That lets you write down the exact cost of a three month crawl before the first request.
How do I collect each language separately?
Many sites serve a country's language version based on the visitor's IP. Put the country in the username, such as -cc-de for Germany or -cc-jp for Japan, and requests exit from there.
Give each language its own worker pool and username. Keep one crawler codebase and change only the username, and the language buckets stay apart.
What does a three month crawl cost?
At 200 connections and 100 Mbps, the Quarter (90 day) plan is $3.61 a day, $324.90 in total. That is 24% less than buying one day at a time ($4.75).
For a bigger crawl, the 1,000 connection, 200 Mbps tier on Month (30 days) is $8.78 a day, $263.40 in total. The dashboard also lets you set any period from 1~365 days, so you buy only the days your crawl runs.
What should I keep in mind when collecting data?
Collect only public content and follow robots.txt and site terms. Check the copyright and privacy rules that apply to training use as well. Filter out personal data you do not need before it enters the dataset.
Cap requests per domain even when the proxy has room for more. Sites that stay healthy are sites you can crawl again for the next dataset version.
Connect HiProxy for AI training data
1
Top up enough balance for the full crawl period. One top-up can be up to $5,000, and $100 or more earns bonus balance.
2
Buy an unlimited residential plan that matches your crawl days. Quarter (90 days) has the lowest daily rate.
3
Create one username per language, such as hpexample1-zone-unl-cc-fr, and attach each to a worker pool.
4
Connect the crawler to gate.hiproxy.shop:6100 and limit open connections to the tier you bought.
Common questions
Can one plan use several countries?
Yes. One unlimited plan covers all 60 countries, and you pick the country per request in the username.
Should a crawl use sticky sessions?
Usually a new IP per request spreads load more evenly. Use -sid only for sites where cookies need to carry across several pages.
How do I pay?
In crypto, through the payment window. The accepted coins and networks are listed there. The amount lands in your USD balance.
What happens to balance left after the crawl?
Balance does not expire. Use it for the next crawl or for any other product.
Top up $30 and get 78 GB of residential traffic
Pick a username and a password and you're in. Your balance never expires.