libcurl-tutorial(3) libcurl programming libcurl-tutorial(3)
NAME
libcurl-tutorial - libcurl programming tutorial
Objective
This document attempts to describe the general principles and some
basic approaches to consider when programming with libcurl. The text
will focus mainly on the C interface but might apply fairly well on
other interfaces as well as they usually follow the C one pretty
closely.
This document will refer to 'the user' as the person writing the source
code that uses libcurl. That would probably be you or someone in your
position. What will be generally referred to as 'the program' will be
the collected source code that you write that is using libcurl for
transfers. The program is outside libcurl and libcurl is outside of the
program.
To get the more details on all options and functions described herein,
please refer to their respective man pages.
Building
There are many different ways to build C programs. This chapter will
assume a unix-style build process. If you use a different build system,
you can still read this to get general information that may apply to
your environment as well.
Compiling the Program
Your compiler needs to know where the libcurl headers are
located. Therefore you must set your compiler's include path to
point to the directory where you installed them. The 'curl-con-
fig'[3] tool can be used to get this information:
$ curl-config --cflags
Linking the Program with libcurl
When having compiled the program, you need to link your object
files to create a single executable. For that to succeed, you
need to link with libcurl and possibly also with other libraries
that libcurl itself depends on. Like the OpenSL libraries, but
even some standard OS libraries may be needed on the command
line. To figure out which flags to use, once again the 'curl-
config' tool comes to the rescue:
$ curl-config --libs
SL or Not
libcurl can be built and customized in many ways. One of the
things that varies from different libraries and builds is the
support for SL-based transfers, like HTPS and FTPS. If OpenSL
was detected properly at build-time, libcurl will be built with
SL support. To figure out if an installed libcurl has been
built with SL support enabled, use 'curl-config' like this:
$ curl-config --feature
And if SL is supported, the keyword 'SL' will be written to
stdout, possibly together with a few other features that can be
on and off on different libcurls.
See also the "Features libcurl Provides" further down.
autoconf macro
When you write your configure script to detect libcurl and setup
variables accordingly, we offer a prewritten macro that probably
does everything you need in this area. See
docs/libcurl/libcurl.m4 file - it includes docs on how to use
it.
Portable Code in a Portable World
The people behind libcurl have put a considerable effort to make
libcurl work on a large amount of different operating systems and envi-
ronments.
You program libcurl the same way on all platforms that libcurl runs on.
There are only very few minor considerations that differs. If you just
make sure to write your code portable enough, you may very well create
yourself a very portable program. libcurl shouldn't stop you from that.
Global Preparation
The program must initialize some of the libcurl functionality globally.
That means it should be done exactly once, no matter how many times you
intend to use the library. Once for your program's entire life time.
This is done using
curlglobalinit()
and it takes one parameter which is a bit pattern that tells libcurl
what to initialize. Using CURLGLOBALAL will make it initialize all
known internal sub modules, and might be a good default option. The
current two bits that are specified are:
CURLGLOBALWIN32
which only does anything on Windows machines. When used
on a Windows machine, it'll make libcurl initialize the
win32 socket stuff. Without having that initialized prop-
erly, your program cannot use sockets properly. You
should only do this once for each application, so if your
program already does this or of another library in use
does it, you should not tell libcurl to do this as well.
CURLGLOBALSL
which only does anything on libcurls compiled and built
SL-enabled. On these systems, this will make libcurl
initialize OpenSL properly for this application. This is
only needed to do once for each application so if your
program or another library already does this, this bit
should not be needed.
libcurl has a default protection mechanism that detects if
curlglobalinit(3) hasn't been called by the time curleasyperform(3)
is called and if that is the case, libcurl runs the function itself
with a guessed bit pattern. Please note that depending solely on this
is not considered nice nor very good.
When the program no longer uses libcurl, it should call
curlglobalcleanup(3), which is the opposite of the init call. It will
then do the reversed operations to cleanup the resources the
curlglobalinit(3) call initialized.
Repeated calls to curlglobalinit(3) and curlglobalcleanup(3) should
be avoided. They should only be called once each.
Features libcurl Provides
It is considered best-practice to determine libcurl features run-time
rather than build-time (if possible of course). By calling curlver-
sioninfo() and checking tout he details of the returned struct, your
program can figure out exactly what the currently running libcurl sup-
ports.
Handle the Easy libcurl
libcurl first introduced the so called easy interface. All operations
in the easy interface are prefixed with 'curleasy'.
Recent libcurl versions also offer the multi interface. More about that
interface, what it is targeted for and how to use it is detailed in a
separate chapter further down. You still need to understand the easy
interface first, so please continue reading for better understanding.
To use the easy interface, you must first create yourself an easy han-
dle. You need one handle for each easy session you want to perform.
Basically, you should use one handle for every thread you plan to use
for transferring. You must never share the same handle in multiple
threads.
Get an easy handle with
easyhandle = curleasyinit();
It returns an easy handle. Using that you proceed to the next step:
setting up your preferred actions. A handle is just a logic entity for
the upcoming transfer or series of transfers.
You set properties and options for this handle using
curleasysetopt(3). They control how the subsequent transfer or trans-
fers will be made. Options remain set in the handle until set again to
something different. Alas, multiple requests using the same handle will
use the same options.
Many of the options you set in libcurl are "strings", pointers to data
terminated with a zero byte. Keep in mind that when you set strings
with curleasysetopt(3), libcurl will not copy the data. It will
merely point to the data. You MUST make sure that the data remains
available for libcurl to use until finished or until you use the same
option again to point to something else.
One of the most basic properties to set in the handle is the URL. You
set your preferred URL to transfer with CURLOPTURL in a manner similar
to:
curleasysetopt(handle, CURLOPTURL, "http:/domain.com/");
Let's assume for a while that you want to receive data as the URL iden-
tifies a remote resource you want to get here. Since you write a sort
of application that needs this transfer, I assume that you would like
to get the data passed to you directly instead of simply getting it
passed to stdout. So, you write your own function that matches this
prototype:
sizet writedata(void *buffer, sizet size, sizet nmemb, void
*userp);
You tell libcurl to pass all data to this function by issuing a func-
tion similar to this:
curleasysetopt(easyhandle, CURLOPTWRITEFUNCTION, writedata);
You can control what data your function get in the forth argument by
setting another property:
curleasysetopt(easyhandle, CURLOPTWRITEDATA, &internalstruct);
Using that property, you can easily pass local data between your appli-
cation and the function that gets invoked by libcurl. libcurl itself
won't touch the data you pass with CURLOPTWRITEDATA.
libcurl offers its own default internal callback that'll take care of
the data if you don't set the callback with CURLOPTWRITEFUNCTION. It
will then simply output the received data to stdout. You can have the
default callback write the data to a different file handle by passing a
'FILE *' to a file opened for writing with the CURLOPTWRITEDATA
option.
Now, we need to take a step back and have a deep breath. Here's one of
those rare platform-dependent nitpicks. Did you spot it? On some plat-
forms[2], libcurl won't be able to operate on files opened by the pro-
gram. Thus, if you use the default callback and pass in a an open file
with CURLOPTWRITEDATA, it will crash. You should therefore avoid this
to make your program run fine virtually everywhere.
(CURLOPTWRITEDATA was formerly known as CURLOPTFILE. Both names still
work and do the same thing).
If you're using libcurl as a win32 DL, you MUST use the CURLOPTWRITE-
FUNCTION if you set CURLOPTWRITEDATA - or you will experience crashes.
There are of course many more options you can set, and we'll get back
to a few of them later. Let's instead continue to the actual transfer:
success = curleasyperform(easyhandle);
curleasyperform(3) will connect to the remote site, do the necessary
commands and receive the transfer. Whenever it receives data, it calls
the callback function we previously set. The function may get one byte
at a time, or it may get many kilobytes at once. libcurl delivers as
much as possible as often as possible. Your callback function should
return the number of bytes it "took care of". If that is not the exact
same amount of bytes that was passed to it, libcurl will abort the
operation and return with an error code.
When the transfer is complete, the function returns a return code that
informs you if it succeeded in its mission or not. If a return code
isn't enough for you, you can use the CURLOPTERORBUFER to point
libcurl to a buffer of yours where it'll store a human readable error
message as well.
If you then want to transfer another file, the handle is ready to be
used again. Mind you, it is even preferred that you re-use an existing
handle if you intend to make another transfer. libcurl will then
attempt to re-use the previous
ulti-threading Issues
libcurl is completely thread safe, except for two issues: signals and
alarm handlers. Signals are needed for a SIGPIPE handler, and the
alarm() call is used to deal with timeouts (during DNS lookup).
If you are accessing HTPS or FTPS URLs in a multi-threaded manner, you
are then of course using OpenSL multi-threaded and it has itself a few
requirements on this. Basically, you need to provide one or two func-
tions to allow it to function properly. For all details, see this:
http:/www.openssl.org/docs/crypto/threads.html#DESCRIPTION
When using multiple threads you should set the CURLOPTNOSIGNAL option
to TRUE for all handles. Everything will work fine except that timeouts
are not honored during the DNS lookup - which you can work around by
building libcurl with c-ares support. c-ares is a library that provides
asynchronous name resolves. Unfortunately, c-ares does not yet support
IPv6.
Also, note that CURLOPTDNSUSEGLOBALCACHE is not thread-safe.
When It Doesn''t Work
There will always be times when the transfer fails for some reason. You
might have set the wrong libcurl option or misunderstood what the
libcurl option actually does, or the remote server might return non-
standard replies that confuse the library which then confuses your pro-
gram.
There's one golden rule when these things occur: set the CURLOPTVER-
BOSE option to TRUE. It'll cause the library to spew out the entire
protocol details it sends, some internal info and some received proto-
col data as well (especially when using FTP). If you're using HTP,
adding the headers in the received output to study is also a clever way
to get a better understanding why the server behaves the way it does.
Include headers in the normal body output with CURLOPTHEADER set TRUE.
Of course there are bugs left. We need to get to know about them to be
able to fix them, so we're quite dependent on your bug reports! When
you do report suspected bugs in libcurl, please include as much details
you possibly can: a protocol dump that CURLOPTVERBOSE produces,
library version, as much as possible of your code that uses libcurl,
operating system name and version, compiler name and version etc.
If CURLOPTVERBOSE is not enough, you increase the level of debug data
your application receive by using the CURLOPTDEBUGFUNCTION.
Getting some in-depth knowledge about the protocols involved is never
wrong, and if you're trying to do funny things, you might very well
understand libcurl and how to use it better if you study the appropri-
ate RFC documents at least briefly.
Upload Data to a Remote Site
libcurl tries to keep a protocol independent approach to most trans-
fers, thus uploading to a remote FTP site is very similar to uploading
data to a HTP server with a PUT request.
Of course, first you either create an easy handle or you re-use one
existing one. Then you set the URL to operate on just like before. This
is the remote URL, that we now will upload.
Since we write an application, we most likely want libcurl to get the
upload data by asking us for it. To make it do that, we set the read
callback and the custom pointer libcurl will pass to our read callback.
The read callback should have a prototype similar to:
sizet function(char *bufptr, sizet size, sizet nitems, void
*userp);
Where bufptr is the pointer to a buffer we fill in with data to upload
and size*nitems is the size of the buffer and therefore also the maxi-
mum amount of data we can return to libcurl in this call. The 'userp'
pointer is the custom pointer we set to point to a struct of ours to
pass private data between the application and the callback.
curleasysetopt(easyhandle, CURLOPTREADFUNCTION, readfunction);
curleasysetopt(easyhandle, CURLOPTINFILE, &filedata);
Tell libcurl that we want to upload:
curleasysetopt(easyhandle, CURLOPTUPLOAD, TRUE);
A few protocols won't behave properly when uploads are done without any
prior knowledge of the expected file size. So, set the upload file size
using the CURLOPTINFILESIZELARGE for all known file sizes like
this[1]:
/* in this example, filesize must be an offt variable */
curleasysetopt(easyhandle, CURLOPTINFILESIZELARGE, filesize);
When you call curleasyperform(3) this time, it'll perform all the
necessary operations and when it has invoked the upload it'll call your
supplied callback to get the data to upload. The program should return
as much data as possible in every invoke, as that is likely to make the
upload perform as fast as possible. The callback should return the num-
ber of bytes it wrote in the buffer. Returning 0 will signal the end of
the upload.
Passwords
Many protocols use or even require that user name and password are pro-
vided to be able to download or upload the data of your choice. libcurl
offers several ways to specify them.
Most protocols support that you specify the name and password in the
URL itself. libcurl will detect this and use them accordingly. This is
written like this:
protocol:/user:password@example.com/path/
If you need any odd letters in your user name or password, you should
enter them URL encoded, as %X where X is a two-digit hexadecimal num-
ber.
libcurl also provides options to set various passwords. The user name
and password as shown embedded in the URL can instead get set with the
CURLOPTUSERPWD option. The argument passed to libcurl should be a char
* to a string in the format "user:password:". In a manner like this:
curleasysetopt(easyhandle, CURLOPTUSERPWD, "myname:thesecret");
Another case where name and password might be needed at times, is for
those users who need to authenticate themselves to a proxy they use.
libcurl offers another option for this, the CURLOPTPROXYUSERPWD. It is
used quite similar to the CURLOPTUSERPWD option like this:
curleasysetopt(easyhandle, CURLOPTPROXYUSERPWD, "myname:these-
cret");
There's a long time unix "standard" way of storing ftp user names and
passwords, namely in the $HOME/.netrc file. The file should be made
private so that only the user may read it (see also the "Security Con-
siderations" chapter), as it might contain the password in plain text.
libcurl has the ability to use this file to figure out what set of user
name and password to use for a particular host. As an extension to the
normal functionality, libcurl also supports this file for non-FTP pro-
tocols such as HTP. To make curl use this file, use the CURLOPTNETRC
option:
curleasysetopt(easyhandle, CURLOPTNETRC, TRUE);
And a very basic example of how such a .netrc file may look like:
machine myhost.mydomain.com
login userlogin
password secretword
All these examples have been cases where the password has been
optional, or at least you could leave it out and have libcurl attempt
to do its job without it. There are times when the password isn't
optional, like when you're using an SL private key for secure trans-
fers.
To pass the known private key password to libcurl:
curleasysetopt(easyhandle, CURLOPTSLKEYPASWD, "keypassword");
HTP Authentication
The previous chapter showed how to set user name and password for get-
ting URLs that require authentication. When using the HTP protocol,
there are many different ways a client can provide those credentials to
the server and you can control what way libcurl will (attempt to) use.
The default HTP authentication method is called 'Basic', which is
sending the name and password in clear-text in the HTP request,
base64-encoded. This is insecure.
At the time of this writing libcurl can be built to use: Basic, Digest,
NTLM, Negotiate, GS-Negotiate and SPNEGO. You can tell libcurl which
one to use with CURLOPTHTPAUTH as in:
curleasysetopt(easyhandle, CURLOPTHTPAUTH, CURLAUTHDIGEST);
And when you send authentication to a proxy, you can also set authenti-
cation type the same way but instead with CURLOPTPROXYAUTH:
curleasysetopt(easyhandle, CURLOPTPROXYAUTH, CURLAUTHNTLM);
Both these options allow you to set multiple types (by ORing them
together), to make libcurl pick the most secure one out of the types
the server/proxy claims to support. This method does however add a
round-trip since libcurl must first ask the server what it supports:
curleasysetopt(easyhandle, CURLOPTHTPAUTH,
CURLAUTHDIGESTCURLAUTHBASIC);
For convenience, you can use the 'CURLAUTHANY' define (instead of a
list with specific types) which allows libcurl to use whatever method
it wants.
When asking for multiple types, libcurl will pick the available one it
considers "best" in its own internal order of preference.
HTP POSTing
We get many questions regarding how to issue HTP POSTs with libcurl
the proper way. This chapter will thus include examples using both dif-
ferent versions of HTP POST that libcurl supports.
The first version is the simple POST, the most common version, that
most HTML pages using the |
