Loading...
Searching...
No Matches
Sofia SIP User Agent Library - "msg" - Message Parser Module

# Module Meta Information

This module contains parser and functions for manipulating messages and headers for text-based protocols like SIP, HTTP or RTSP. It also provides parsing of MIME headers and MIME multipart messages common to these protocols.

Contact:\n Pekka Pessi <Pekka.Pessi@nokia-email.address.hidden>

Status:\n Sofia SIP Core library

License:\n LGPL

Contributor(s):
  - Pekka Pessi Pekka.nosp@m..Pes.nosp@m.si@no.nosp@m.kia..nosp@m.com

# Contents of msg Module

The msg module contains the public header files as follows:

- <sofia-sip/msg.h> base message interfaces
- <sofia-sip/msg_types.h> message and header struct definitions and typedefs
- <sofia-sip/msg_protos.h> prototypes of header-specific functions for generic headers
- <sofia-sip/msg_header.h> function prototypes and macros for manipulating message headers
- <sofia-sip/msg_addr.h> functions for accessing network addresses and I/O vectors associated with the message
- <sofia-sip/msg_date.h> types and functions for handling dates and times
- <sofia-sip/msg_mime.h> types, function prototypes and macros for MIME headers and multipart messages
- <sofia-sip/msg_mime_protos.h> prototypes of MIME-header-specific functions

In addition to this interface, the parser documentation contains description of the functionality required when an existing parser is extended by a new header or a parser is created for a completely new protocol. It is possible to add new headers to the parser or extend the definition of existing ones. The header files used for constructing these parsers are as follows:

- <sofia-sip/msg_parser.h> parsing functions, macros
- <sofia-sip/msg_mclass.h> message factory object definition
- <sofia-sip/msg_mclass_hash.h> hashing of header names

# Parsers, Messages and Headers

The Sofia msg module contains interface to the text-based parsers for RFC822-like message, the header and message objects. Currently, there are three parsers defined: SIP, HTTP, and MIME.

The C structure corresponding to each header is defined either in a <sofia-sip/msg_types.h> or in a protocol-specific header file. These protocol-specific header files include <sofia-sip/sip.h>, <sofia-sip/http.h>, and <sofia-sip/msg_mime.h>. For each header, there is defined a header class structure, some standard functions, and tags for including them in tag lists.

As a convention, all the identifiers for SIP headers start with prefix `sip` and all the macros with `SIP`. Same thing holds for HTTP, too: it uses prefix `http`. However, the MIME headers and the functions related to them are defined within the msg module and they use prefix `msg`. If a SIP or HTTP header uses a structure defined in <sofia-sip/msg_types.h>, there is a typedef suitable for the particular protocol, for example Accept header is defined multiple times:

typedef struct msg_accept_s sip_accept_t;
typedef struct msg_accept_s http_accept_t;
sip_accept_t
struct sip_accept_s sip_accept_t
msg_accept_s
Structure for Accept header.
Definition msg_mime.h:80

For header X of protocol NS, there are types, functions, macros and header class as follows:

- `ns_X_t` is the structure used to store parsed header,
- `ns_hclass_t` `ns_X_class`[] contains the header class for header X,
- `NS_X_INIT()` initializes a static instance of `ns_X_t`,
- `ns_X_init()` initializes a dynamic instance of `ns_X_t`,
- `ns_is_X()` tests if header object is instance of header X,
- `ns_X_make()` creates a header X object by decoding given string,
- `ns_X_format()` creates a header X object by decoding given `printf()` list,
- `ns_X_dup()` duplicates (deeply copies) the header X,
- `ns_X_copy()` copies the header X,
- `NSTAG_X()` is used include instance of `ns_X_t` in a tag list, and
- `NSTAG_X_STR()` is used to include string containing value header in a tag list.

The declarations of header tags and the prototypes for these functions can be imported separately from the type definitions, for instance, the tags related to SIP headers are declared in the include file <sofia-sip/sip_tag.h>, and the header-specific functions in <sofia-sip/sip_header.h>.

# Parsing Text Messages

Sofia text parser follows recursive-descent principle. In other words, it is a program that descends the syntax tree top-down recursively. (All syntax trees have root at top and they grow downwards.)

In the case of SIP, HTTP and other similar protocols, such a parser is very efficient. The parser can choose between different forms based on each token, as the protocol syntax is carefully designed so that it requires only minimal scan-ahead. It is also easy to extend a recursive-descent parser via a standard API, unlike, for instance, a LALR parser generated by Bison.

The abstract message module msg contains a high-level parser engine that drives the parsing process and invokes the protocol-specific parser for each header. As there is no low-layer framing between the RFC822-style messages, the parser considers any received data, be it a UDP datagram or a TCP stream, as a byte stream. The protocol-specific parsers controls how a byte stream is split into separate messages or if it consists of a single message only.

The parser engine works by separating stream into fragments, then passing the fragment to a suitable parser. A fragment is a piece of message that is parsed during a single step: the first line, each header, the empty line between headers and message body, the message body. (In case of HTTP, the message body can consists of multiple fragments known as chunks.)

The parser starts by separating the first line (e.g., request or status line) from the byte stream, then passing the line to the suitable parser. After first line comes the message headers. The parser continues parsing process by extracting headers, each on their own line, from the stream and passing contents of each header to its parser. The message structure is populated based on the parsing results. When an empty line - indicating end of headers - is encountered, the control is passed to the protocol-specific parser. Protocol-specific functions take care of extracting the possible message body from the byte stream.

After parsing process is completed, it can be given to the upper layers (typically a protocol state machine). The parser continues processing the stream and feeding the messages to protocol engine until the end of the stream is reached.

Separating byte stream to messages

When the parsing process has completed, the first line, each header, separator and the message body are all in their own fragment structure. The fragments form a dual-linked list known as fragment chain as shown in the above figure. The memory buffers for the message, the fragment chain, and a whole lot of other stuff is held by the generic message type, msg_t, defined in <sofia-sip/msg.h>. The internal structure of msg_t is known only within msg module and it is opaque to other modules.

The msg parser engine also drives the reverse process, invoking the encoding method of each fragment so that the whole outgoing message can be encoded properly.

# Message Header as a C struct

Just separating headers from each other and from the message body is not usually enough. When a header contains structured data, the header contents should be converted to a form that is convenient to use from C programs. For that purpose, the message parser needs a parsing function specific to each individual header. This parsing function divides the contents of the header into semantically meaningful segments and stores the result in the structure specific to each header.

The parser engine passes the fragment contents to the parsing function after it has separated the fragment from the rest of the message. The parser engine selects correct header class either by implication (in case of first line), or it searches for the header class from the hash table using the header name as the hash key. The header class contains a pointer to the parsing function. The parser has also special header classes for headers with errors and unknown headers, header with a name that is not regocnized by the parser.

For instance, the Accept header has following syntax:

Accept = "Accept" ":" #( media-range [ accept-params ] )
media-range = ( "*" "/" "*"
| ( type "/" "*" )
| ( type "/" subtype ) ) *( ";" parameter )
accept-params = ";" "q" "=" qvalue *( accept-extension )
accept-extension = ";" token [ "=" ( token | quoted-string ) ]

When an Accept header is parsed, the header parser function (msg_accept_d()) separates the type, subtype, and each parameter in the list to strings. The parsing result is assigned to a msg_accept_t structure, which is defined as follows:

typedef struct msg_accept_s
{
msg_common_t ac_common[1]; //< Common fragment info
msg_accept_t *ac_next; //< Pointer to next Accept header
char const *ac_type; //< Pointer to type/subtype
char const *ac_subtype; //< Points after first slash in type
msg_param_t const *ac_params; //< List of parameters
msg_param_t ac_q; //< Value of q parameter
}
msg_accept_t;
msg_accept_t
struct msg_accept_s msg_accept_t
The structure msg_accept_t contains representation of an Accept header.
Definition msg_mime.h:54
msg_common_t
struct msg_common_s msg_common_t
Common part of header.
Definition msg_types.h:89
msg_accept_s::ac_type
char const * ac_type
Pointer to type/subtype.
Definition msg_mime.h:83
msg_accept_s::ac_q
char const * ac_q
Value of q parameter.
Definition msg_mime.h:86
msg_accept_s::ac_params
msg_param_t const * ac_params
List of parameters.
Definition msg_mime.h:85
msg_accept_s::ac_subtype
char const * ac_subtype
Points after first slash in type.
Definition msg_mime.h:84
msg_accept_s::ac_next
msg_accept_t * ac_next
Pointer to next Accept header.
Definition msg_mime.h:82
msg_accept_s::ac_common
msg_common_t ac_common[1]
Common fragment info.
Definition msg_mime.h:81

The string containing the type is put into the `ac_type` field, the subtype after slash in the can be found in the `ac_subtype` field, and the list of accept-params (together with media-specific-parameters) is put in the `ac_params` array. If there is a q parameter present, a pointer to the `qvalue` is assigned to `ac_q` field.

In the beginning of the header structure there are two boilerplate members. The `ac_common`[1] contains information common to all message fragments. The `ac_next` is a pointer to next header field with the same name, in case a message contains multiple Accept headers or multiple comma-separated header fields are located in a single line.

# Representing a Message as a C struct

It is not enough to represent a message as a list of headers following each other. The programmer also needs a convenient way to access certain headers at the message level, for example, accessing directly the Accept header instead of going through all headers and examining their name. The structured view to the message is provided via a message-specific C struct. In general, its type is msg_pub_t (it provides public view to message). The protocol-specific type is sip_t, http_t or msg_multipart_t for SIP, HTTP and MIME, respectively.

So, a single message is represented by two objects, first object (msg_t) is private to the msg module and opaque by an application programmer, second (sip_t, http_t or msg_multipart_t) is a public protocol-specific structure accessible by all.

Note
  The application programmer can obtain a pointer to the protocol-specific structure from an msg_t object using msg_public() function. The msg_public() takes a protocol tag, a well-known identifier, as its argument. The SIP, HTTP and MIME already define a wrapper around msg_public(), for example, a sip_t structure can be obtained with sip_object() function (or macro).

As an example, the sip_t structure is defined as follows:

typedef struct sip_s {
msg_common_t sip_common[1]; // Used with recursive inclusion
msg_pub_t *sip_next; // Ditto
void *sip_user; // Application data
unsigned sip_size; // Size of the structure with
// extension headers
int sip_flags; // Parser flags
sip_error_t *sip_error; // Erroneous headers
sip_request_t *sip_request; // Request line
sip_status_t *sip_status; // Status line
sip_via_t *sip_via; // Via (v)
sip_route_t *sip_route; // Route
sip_record_route_t *sip_record_route; // Record-Route
sip_max_forwards_t *sip_max_forwards; // Max-Forwards
...
} sip_t;
sip_error
#define sip_error(sip)
sip_error_t
msg_error_t sip_error_t
sip_max_forwards_t
struct sip_max_forwards_s sip_max_forwards_t
sip_max_forwards
#define sip_max_forwards(sip)
sip_record_route
#define sip_record_route(sip)
sip_record_route_t
struct sip_route_s sip_record_route_t
sip_request
#define sip_request(sip)
sip_request_t
struct sip_request_s sip_request_t
sip_route
#define sip_route(sip)
sip_route_t
struct sip_route_s sip_route_t
sip_status
#define sip_status(sip)
sip_status_t
struct sip_status_s sip_status_t
sip_via
#define sip_via(sip)
sip_via_t
struct sip_via_s sip_via_t
msg_pub_t
MSG_PUB_T msg_pub_t
Public protocol-specific message structure for accessing the message.
Definition msg_types.h:81
sip_t
struct sip_s sip_t
sip_s
sip_s::sip_size
unsigned sip_size
sip_s::sip_next
msg_pub_t * sip_next
sip_s::sip_user
void * sip_user
sip_s::sip_flags
int sip_flags
sip_s::sip_common
msg_common_t sip_common[1]

As you can see above, the public sip_t structure contains the common header members that are also found in the beginning of a header structure. The sip_size indicates the size of the structure - the application can extend the parser and sip_t structure beyond the original size. The sip_flags contains various flags used during the parsing and printing process. They are documented in the <sofia-sip/msg.h>. These boilerplate members are followed by the pointers to various message elements and headers.

# Result of Parsing Process

Let us now show how a simple message is parsed and presented to the applications. As an exampe, we choose a SIP request message with method BYE, including only the mandatory fields:

BYE sip:joe@example.com SIP/2.0
Via: SIP/2.0/UDP sip.example.edu;branch=d7f2e89c.74a72681
Via: SIP/2.0/UDP pc104.example.edu:1030;maddr=110.213.33.19
From: Bobby Brown <sip:bb@example-email.address.hidden>;tag=77241a86
To: Joe User <sip:joe@example-email.address.hidden>;tag=7c6276c1
Call-ID: 4c4e911b@pc104.example.edu
CSeq: 2

The figure below shows the layout of the BYE message above after parsing:

BYE message and its representation in C

The leftmost box represents the message of type msg_t. Next box from the left reprents the sip_t structure, which contains pointers to a header objects. The next column contains the header objects. There is one header object for each message fragment. The rightmost box represents the I/O buffer used when the message was received. Note that the I/O buffer may be non-continous and composed of many separate memory areas.

The message object has link to the public message structure (m_object), to the dual-linked fragment chain (m_frags) and to the I/O buffer (m_buffer). The public message structure contains pointers to the headers according to their type. If there are multiple headers of the same type (like there are two Via headers in the above message), the headers are put into a single-linked list.

Each fragment has pointers to successing and preceding fragment. It also contains pointer to the corresponding data within the I/O buffer and its length.

The main purpose of the fragment chain is to preserve the original order of the headers. If there were an third Via header after CSeq in the message, the fragment representing it would be after the CSeq header in the fragment chain but after second Via in the header list.

# Example: Parsing a Complete Message

The following code fragment is an example of parsing a complete message. The parsing process is more hairy when there is stream to be parsed.

msg_t *parse_memory(msg_mclass_t const *mclass, char const data[], int len)
{
msg_t *msg;
int m;
msg_iovec_t iovec[2] = {{ 0 }};
msg = msg_create(mclass, 0);
if (!msg)
return NULL;
m = msg_recv_iovec(msg, iovec, 2, n, 1);
if (m < 0) {
msg_destroy(msg);
return NULL;
}
assert(m <= 2);
assert(iovec[0].mv_len + iovec[1].mv_len == n);
memcpy(iovec[0].mv_base, data, n = iovec[0].mv_len);
if (m == 2)
memcpy(iovec[1].mv_base + n, data + n, iovec[1].mv_len);
msg_recv_commit(msg, iovec[0].mv_len + iovec[1].mv_len, 1);
m = msg_extract(msg);
assert(m != 0);
if (m < 0) {
msg_destroy(msg);
return NULL;
}
return msg;
}
msg_create
msg_t * msg_create(msg_mclass_t const *mc, int flags)
Create a message.
Definition msg.c:100
msg_destroy
void msg_destroy(msg_t *)
Destroy a reference to a message.
Definition msg.c:193
msg_extract
int msg_extract(msg_t *msg)
Extract and parse a message from internal buffer.
Definition msg_parser.c:850
msg_iovec_t
struct su_iovec_s msg_iovec_t
I/O vector type.
Definition msg_addr.h:72
msg_recv_commit
isize_t msg_recv_commit(msg_t *msg, usize_t n, int eos)
Commit n bytes of buffers.
Definition msg_parser.c:527
msg_recv_iovec
issize_t msg_recv_iovec(msg_t *msg, msg_iovec_t vec[], isize_t veclen, usize_t n, int exact)
Obtain I/O vector for receiving the data.
Definition msg_parser.c:339
msg_t
struct msg_s msg_t
Message object.
Definition msg_types.h:54
msg_mclass_t
struct msg_mclass_s msg_mclass_t
Message class.
Definition msg_types.h:45

Let's go through this simple function, step by step. First, we get the data pointer and its size in bytes, len. We first initialize an I/O vector used to represent message with the parser.

msg_t *parse_memory(msg_mclass_t const *mclass, char const data[], int len)
{
msg_t *msg;
int m;
msg_iovec_t iovec[2] = {{ 0 }};

The message class mclass (a parser driver object, msg_mclass_t) is used to represent a particular protocol-specific parser instance. When a message object is created, it is given as an argument to msg_create() function:

msg = msg_create(mclass, 0);
if (!msg)
return NULL;

Next we obtain a memory buffer for data with msg_recv_iovec(). The memory buffer is usually a single continous memory area, but in some cases it may consist of two distinct areas. Therefore the iovec is used here to pass the buffers around. The iovec is also very handly as it can be directly passed to various system I/O calls.

m = msg_recv_iovec(msg, iovec, 2, n, 1);
if (m < 0) {
msg_destroy(msg);
return NULL;
}

These assumptions hold always true when you call msg_recv_iovec() first time with a complete message:

assert(m >= 1 && m <= 2);
assert(iovec[0].mv_len + iovec[1].mv_len == n);

Next, we copy the data to the I/O vector and commit the copied data to the message. Earlier with msg_recv_iovec() we allocated buffer space for data, now calling msg_recv_commit() indicates that valid data has been copied to the buffer. The last parameter to msg_recv_commit() indicates that the end of stream is encountered and no more data is to be expected.

memcpy(iovec[0].mv_base, data, n = iovec[0].mv_len);
if (m == 2)
memcpy(iovec[1].mv_base + n, data + n, iovec[1].mv_len);
msg_recv_commit(msg, iovec[0].mv_len + iovec[1].mv_len, 1);

We call msg_extract() next; it takes care of parsing the message. A fatal parsing error is indicated by returning -1. If the message is incomplete, msg_extract() returns 0. When a complete message has been parsed, a positive value is returned. We know that a message cannot be incomplete, as a call to msg_recv_commit() indicated to the parser that the end-of-stream has been encountered.

m = msg_extract(msg);
assert(m != 0);
if (m < 0) {
msg_destroy(msg);
return NULL;
}
return msg;
}

Sofia-SIP 1.12.11devel - Copyright (C) 2006 Nokia Corporation. All rights reserved. Licensed under the terms of the GNU Lesser General Public License.