The Serpent

// Cursing the Internet since 1998

cURL Basics

Posted Feb 24, 2022

cURL is one of the most useful and powerful CLI tools for sending and receiving HTTP(S) requests. It has over 200+ CLI options, but there’s a few that come in handy on a daily basis for network engineers. Here’s a breakdown of the most common configurations.

On it’s own, cURL fetches the URL and displays the response to CLI:

root@linux:/home/user# curl theserpent.co.uk
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

Most of the time, the content being returned isn’t necessary, it’s the response headers that usually matter:

root@linux:/home/user# curl -s -D - -o /dev/null https://www.theserpent.co.uk
HTTP/2 200
server: nginx
date: Thu, 24 Feb 2022 18:59:22 GMT
content-type: text/html
content-length: 20182
last-modified: Tue, 15 Feb 2022 18:53:15 GMT
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000
accept-ranges: bytes
  • -s suppress the progress bar
  • -D dump the headers, using - dumps them to standard output
  • -o dump the response somewhere, in this case to /dev/null

Sometimes it’s useful to see the request headers being sent too:

root@linux:/home/user# curl -v -s -D - -o /dev/null https://www.theserpent.co.uk
  • -v enables debugging, which outputs request headers.

TLS

cURL can also tell you a lot about the TLS connection, though if you’re dealing ith self-signed certificates, you’ll need the following:

--insecure

You might always want to control the exact TLS version in use:

  • --sslv2 use SSLv2
  • --sslv3 use SSLv3
  • --tlsv1.0 use TLSv1.0
  • --tlsv1.1 use TLSv1.1
  • --tlsv1.2 use TLSv1.2
  • --tlsv1.3 use TLSv1.3

Now you can access sites via TLS, regardless of the certificate status.

Custom Headers

Sometimes it’s useful to override the headers being sent, or add custom ones:

root@linux:/home/user# curl -H "X-Custom:some value" https://www.example.com

Proxy Support

cURL also supports proxy usage, there’s a number of arguments for proxy support, but you can also combine them into a simple command:

root@linux:/home/user# curl -x http://192.168.1.1:80 https://www.example.com
curl: (56) Received HTTP code 407 from proxy after CONNECT

If your proxy requires authentication, pass it over using --proxy-user:

root@linux:/home/user# curl --proxy-user jeff:passw0rd -x http://192.168.1.1:80 https://www.example.com

There are other commands you can use to break down the particular proxy request you want to make:

  • --proxy-insecure don’t check the proxy certificate when doing HTTPS (basically the same as --insecure)
  • --proxy-basic use BASIC authentication (‘Proxy-Authorization’ header)
  • --proxy-header pass custom headers to the proxy

Putting it all together

All these commands get get quite cumbersome, so using configuration files really makes things easier. Create a file and add the commands, e.g.

# curl config file

# -I                  Header information only, makes HEAD request
# -s                  No progress bar
# -o /dev/null        Dump response body (use null for Windows)
# -D -                Display headers to stdout
# -H "X-Blah: Cheese" Add\overwrite header

# On-prem primary proxy

-D -
--proxy http://192.168.1.10:80
--proxy-basic
--proxy-user static:passw0rd
--proxy-insecure
--insecure
-s
-o null

Then you simply reference the configuration file using -K.

cURL Basics
Posted February 24, 2022
Written by John Payne