Grow Your Arms, cURL With Proxies

cURL was brought into the world to transfer data. When coupled with a proxy, it morphs into a powerful data mining tool.

This guide will introduce cURL and explain a few different ways you can seamlessly use it with proxies.

What’s a proxy?

A proxy is a computer that sits between your own computer and the internet, routing all your network traffic. You can use it for various purposes, such as caching web pages or anonymizing the connection.

A residential proxy connects you to the internet through another real server (usually in a different location), making it seem like you’re browsing from that place instead of your home.

Learn more about rotating residential proxies here.

What’s cURL?

If you’ve never heard of cURL, maybe you know it by older names – httpget or urlget. It’s been around for some time, since 1996. It’s a handy tool that allows you to transfer data from remote servers. You can use cURL with proxies to fetch web pages, download remote files, and do many more things without detection.

What’s the URL?

Uniform Resource Locator is the proper term for a web address like https://example.com, for example. (What? It’s a good example!)

The first part of the URL (HTTPS) is the protocol identifier. Sometimes the protocol is HTTP, and sometimes it’s HTTPS. The difference is that one (HTTPS) uses a secure protocol with encryption, and the other (HTTP) is unencrypted. There are a lot more protocols, but I’ll mostly focus on these two.

Which should you use? It depends on your risk tolerance. You can read more about encryption and risk here.

The second part of the URL (example.com) is the resource.

And the little ‘c’ in front of the URL?

It stands for client. (That’d be you)

Combine them, and you get cURL. You and URL, connecting and locating resources together. (Touching stuff!)

What does cURL do?

cURL operates on Linux, Windows, and Mac operating systems, as well as Android and iOS. It also provides options for customizing its features, making it a much-desired tool in the webmasters’ community.

It also supports authentication methods such as Basic and Digest and can handle user-level HTTP proxy configurations exceptionally well.

cURL has many configuration options that let you decide what it should do when it encounters certain conditions, such as an HTTP error code.

Bottom line: it does most things a browser does, but all in a tiny black window without pictures and all that fun stuff.

Quick cURL installation guide:

If you use Mac OSX or an Apple device (IOS), curl comes preinstalled, so just skip this part.

For Windows with Cygwin already installed, curl will be recognized as an internal command, so there’s no need to download it. If you’re a Windows user and don’t have Cygwin installed, cURL is most likely not in your c:\windows\system32 directory, so go ahead and grab it from here.

For example, if you’re using Ubuntu, cURL will be available in the default repositories, so just issue this command: Sudo apt-get install curl.

The essential parts of a cURL command line

If you’re not a computer astrophysicist, allow me to shed some light on the structure of a command line.

• Protocol – refers to network communication schemes like HTTP, HTTPS, SOCKS, etc.
• Proxy server – the address of the proxy server or proxy host you plan to use
• Port number – the port you will access through
• Resource address – the webpage you want to target
• Username and password – if the proxy address requires authentication
• Tags – are the command-line options that turn each command on and off
• Command-line arguments – allow you to pass on information regarding the command line option (tag)

Put them all together, and you have something like this:

How do you run cURL with proxies?

You can set up cURL with various kinds of proxies like residential proxies, private proxies, shared proxies, and datacenter proxies.

The most common way to set up cURL with proxies is with a proxy command.

Using proxy commands

A proxy command is a command-line option, or tag, that influences the proxy string (proxy details + port + user + pass). The influence is mostly just turning the proxy on or off.

Each tag has a short (-) and long (–) version. Kind of like morse code’s dits and dahs.

-x 

is the short option.

--proxy 

is the long option.

They both do the same thing, except that -x is more efficient (and perhaps more aesthetically pleasing).

To properly connect cURL to a proxy, use the -x or –proxy tag followed by the protocol, proxy server, port number, and whatever resource you’re after.

Here’s a bare-bones example of what the command line might look like:

-x, --proxy [protocol]://[proxyhost]:[port][resource address]

And a little more fleshed out:

-x, --proxy http://residential.ipb.server:7777 https://google.com

There are a few other ways to use a proxy with cURL.

The following sections will touch on the basic authentication scheme, using environment variables, an alias, the reconfig file, different protocols, and disabling proxies.

cURL with a username and password

The tag to use a proxy authentication scheme is -u or –proxy-user. Right between the proxy string and resource, throw in the tag, user name, and password.

 $ curl -x http://residential.ipb.server:7777 [-u username:password] http://google.com

Using an alias

An alias for a command, for a proxy, sounds like top secret triple-agent-type tactics.

It’s actually pretty simple.

You’re just making the curl command equal the curl command + the proxy details.

So when you enter:

$ curl https//google.com

It actually means:

$ curl -x [all your proxy details] https://google.com

How is this wizardry done? Simple (like I said):

$ alias curl=curl -x [enter all the proxy details here]

Using environment variables

What’s an environment variable? If you’re thinking something like slippery road conditions due to rain – close, but not quite.

In this case, it’s a shortcut.

Instead of typing out the proxy string all the time, you can set the proxy permanently.

Environment variables work for every type of protocol. All you need to do is enable it by setting the variable PROTOCOL_PROXY.

http_proxy, HTTPS_PROXY, FTP_PROXY

…and so on for whatever protocol_proxy combination you choose. Notice that HTTP is the only one in smaller case letters. That was on purpose. It’s the only one that has to be lower case, but I won’t get into the technical reason for it here.

You can also use:

ALL_PROXY

if your tasks aren’t protocol-specific.

For example, let’s make up a proxy string following this template:

$ curl -x [proxy server]:[port] [username:password] [resource]

(making shit up)

$ curl -x  http://180.1.0.0.1222:8080 -u jumpingjack:candlestick https://nimblejack.com

Using the proxy command option, you’d have to type this out all the time. Instead, set the variable with export, like this:

export http_proxy=http://180.1.0.0.1222:8080 -u jumpingjack:candlestick https://nimblejack.com

or

export WHATEVER OTHER PROTOCOL YOU WANT_PROXY:// [proxy server]:[port] [username:password]

Now, all you have to enter in the command line is:

$ curl https//nimblejack.com

And you’re good to go!

Note: If you want to use a different proxy after you set an environment variable, -x tags will override it.

Change the rc.file

Here you appeal to cURL’s inherent nature to use its default settings. When the program loads, it will search %APPDATA% or %USERPROFILE%\Application Data for the reconfig file.

It will look like rc.file on Mac and Linux, and _curlrc on Windows.

So all you need to do is open up that file and change the proxy details:

proxy=http://[proxy details]

cURL with a SOCKS proxy

cURL supports SOCKS version SOCKS4 and SOCKS5. 

If the proxy servers use socks protocols, the syntax remains the same: -socks4://, socks4a:// or socks5://.

Here are the command-line options for each specific SOCKS version:

SOCKS4:

$ curl -x socks4://180.1.0.0.1222:8080 http://ipburger.com

SOCKS4A:

$ curl -x socks4a://180.1.0.0.1222:8080 http://ipburger.com/bestresidentialproxies

SOCKS5:

$ curl -x socks5://180.1.0.0.1222:8080 http://ipburger.com/betterthanoxylabsandbrightdata

SOCKS5H:

$ curl -x socks5h://180.1.0.0.1222:8080 http://ipburger.com/decentreadingmaterialtoo

You may notice that I didn’t put a port number after the IP address.

(Maybe you were distracted…)

When using the SOCKS protocol, you don’t need to include a port number in the URL because it isn’t necessary.

Just so you know.

Disable proxies

We now know that cURL with a proxy can be set by all sorts of means. This makes running cURL easy and saves time typing. However, you may need to reset your current proxy settings and connect directly in some situations.

The –noproxy option can be used to disable the current proxy configuration.

$ curl --noproxy http://ipburger.com

IP Burger cURL Proxies

You can simply skip the command line composition. It can be tedious, no doubt. So there’s no judgment if you’d rather have them put together for you. In fact, I encourage it.

Not just because IPBurger delivers residential curl proxies hot off the server, in seconds, but also because you’re probably doing something cool. And I’d rather you spend time doing that than playing lego with boolean bits.

You can just download a list of proxies ready with command-line argument, command line parameter, protocol host port, username, and pass in place.

All of them connect to residential proxies that are perfect for web scraping, or the other cool things you do.

FAQ

Why doesn’t cURL always use the same proxy?

It doesn’t have a way to utilize a single or specific pool of residential IPs from the list. The best option is to run multiple cURLs simultaneously from separate terminal windows, each on different residential proxies. This guarantees that every request gets its own pool of residential IP addresses.

Can I mix HTTP and HTTPS protocols? 

You can totally mix your requests as long as they have both transparent and anonymous credentials. If you plan to download files over HTTPS, cURL will always choose HTTPS on its own.

How can I configure cURL to use different proxies at once?

cURL does not have a proxy pool option. It only supports one proxy at a time. To use it with multiple proxies, you should set them as cURL options and pass them as an argument. In other words,

curl -x http://addr1:port1 -u username -p password URL_SCHEME://addr2:port2

If you want cURL to work with two or more proxies via environment variables, just add the proxy settings in your .bashrc file. Then you can access any proxy by using cURL through that specific proxy server by typing the command in the terminal without any arguments after exporting your shell profile.

Do cURL proxies need to be static? 

No. cURL proxies are residential IP addresses and change often. In other words, they can be dynamic. Dynamic residential IPs will work adequately with cURL if they’re fast enough for requests to run smoothly through them.

What is curl used for? 

You can utilize it in whatever way you see fit. Typically, it’s a command-line utility that you can use for web scraping or downloading files.

What kind of proxies do I need?

You need residential proxies that are good quality proxies because they can get past CAPTCHAs automatically without triggering alarms. You should consider using these one-size-fits-all proxies where users can select different options like speed, anonymity level, and country.

For more help about these options, type “man curl” or “curl -h” in the terminal. It will display the man page. (It’s not what you think.)