- Method tokenize
A structured header field, as specified by RFC 0822, is constructed from a sequence of lexical elements.
The header value to parse.
An optional set of flags. Currently only one flag is defined:
Keep backslash-escapes in quoted-strings.
The lexical elements parsed are:
individual special characters
This function will analyze a string containing the header value, and produce an array containing the lexical elements.
Individual special characters will be returned as characters (i.e.
Quoted-strings, domain-literals and atoms will be decoded and returned as strings.
Comments are not returned in the array at all.
As domain-literals are returned as strings, there is no way to tell the domain-literal [127.0.0.1] from the quoted-string "[127.0.0.1]". Hopefully this won't cause any problems. Domain-literals are used seldom, if at all, anyway...
- See also