     [file version.texi not included]

Libidn2
*******

This manual is for Libidn2 (version VERSION, UPDATED), an implementation of IDNA2008/TR46 internationalized domain names.

Copyright © 2011--2021 Simon Josefsson

Introduction
************

Libidn2 is a free software implementation of IDNA2008, Punycode and Unicode TR46.  Its purpose is to encode and decode internationalized domain names.

The library is a rewrite of the popular but legacy libidn library, and is backwards (API) compatible with it. See Converting from libidn for more information.

For technical reference, see:

  * IDNA2008 Framework (https://tools.ietf.org/html/rfc5890)

  * IDNA2008 Protocol (https://tools.ietf.org/html/rfc5891)

  * IDNA2008 Unicode tables (https://tools.ietf.org/html/rfc5892)

  * IDNA2008 Bidi rule (https://tools.ietf.org/html/rfc5893)

  * Punycode (https://tools.ietf.org/html/rfc3492)

  * Unicode IDNA Compatibility Processing (http://www.unicode.org/reports/tr46/)

Libidn2 uses GNU libunistring (https://www.gnu.org/software/libunistring/) for Unicode processing and optionally GNU libiconv (https://www.gnu.org/software/libiconv/) for character set conversion.

The library is dual-licensed under LGPLv3 or GPLv2, see the file COPYING for detailed information.

Library Functions
*****************

Below are the interfaces of the Libidn2 library documented.

Header file idn2.h
==================

To use the functions documented in this chapter, you need to include the file idn2.h like this:

     #include <idn2.h>

Core Functions
==============

When you have the data encoded in UTF-8 form the direct interfaces to the library are as follows.

     [file idn2_to_ascii_8z.texi not included]
     [file idn2_to_unicode_8z8z.texi not included]

     [file idn2_lookup_u8.texi not included]
     [file idn2_register_u8.texi not included]

Locale Functions
================

As a convenience, the following functions are provided that will convert the input from the locale encoding format to UTF-8 and normalize the string using NFC, and then apply the core functions described earlier.

     [file idn2_to_ascii_lz.texi not included]
     [file idn2_to_unicode_8zlz.texi not included]
     [file idn2_to_unicode_lzlz.texi not included]

     [file idn2_lookup_ul.texi not included]
     [file idn2_register_ul.texi not included]

Control Flags
=============

The flags parameter can take on the following values, or a bit-wise inclusive or of any subset of the parameters:

 -- Global flag: idn2_flags IDN2_NFC_INPUT
     Apply NFC normalization on input.

 -- Global flag: idn2_flags IDN2_ALABEL_ROUNDTRIP
     Apply additional round-trip conversion of A-label inputs.

 -- Global flag: idn2_flags IDN2_TRANSITIONAL
     Perform Unicode TR46 transitional processing.

 -- Global flag: idn2_flags IDN2_NONTRANSITIONAL
     Perform Unicode TR46 non-transitional processing (default).

 -- Global flag: idn2_flags IDN2_NO_TR46
     Disable any TR#46 transitional or non-transitional processing.

 -- Global flag: idn2_flags IDN2_USE_STD3_ASCII_RULES
     Use STD3 ASCII rules. This is a TR#46 flag and is a no-op when IDN2_NO_TR46 is specified.

Error Handling
==============

     [file idn2_strerror.texi not included]
     [file idn2_strerror_name.texi not included]

Return Codes
============

The functions normally return 0 on success or a negative error code.

 -- Return code: idn2_rc IDN2_OK
     Successful return.

 -- Return code: idn2_rc IDN2_MALLOC
     Memory allocation error.

 -- Return code: idn2_rc IDN2_NO_CODESET
     Could not determine locale string encoding format.

 -- Return code: idn2_rc IDN2_ICONV_FAIL
     Could not transcode locale string to UTF-8.

 -- Return code: idn2_rc IDN2_ENCODING_ERROR
     Unicode data encoding error.

 -- Return code: idn2_rc IDN2_NFC
     Error normalizing string.

 -- Return code: idn2_rc IDN2_PUNYCODE_BAD_INPUT
     Punycode invalid input.

 -- Return code: idn2_rc IDN2_PUNYCODE_BIG_OUTPUT
     Punycode output buffer too small.

 -- Return code: idn2_rc IDN2_PUNYCODE_OVERFLOW
     Punycode conversion would overflow.

 -- Return code: idn2_rc IDN2_TOO_BIG_DOMAIN
     Domain name longer than 255 characters.

 -- Return code: idn2_rc IDN2_TOO_BIG_LABEL
     Domain label longer than 63 characters.

 -- Return code: idn2_rc IDN2_INVALID_ALABEL
     Input A-label is not valid.

 -- Return code: idn2_rc IDN2_UALABEL_MISMATCH
     Input A-label and U-label does not match.

 -- Return code: idn2_rc IDN2_INVALID_FLAGS
     Invalid combination of flags.

 -- Return code: idn2_rc IDN2_NOT_NFC
     String is not NFC.

 -- Return code: idn2_rc IDN2_2HYPHEN
     String has forbidden two hyphens.

 -- Return code: idn2_rc IDN2_HYPHEN_STARTEND
     String has forbidden starting/ending hyphen.

 -- Return code: idn2_rc IDN2_LEADING_COMBINING
     String has forbidden leading combining character.

 -- Return code: idn2_rc IDN2_DISALLOWED
     String has disallowed character.

 -- Return code: idn2_rc IDN2_CONTEXTJ
     String has forbidden context-j character.

 -- Return code: idn2_rc IDN2_CONTEXTJ_NO_RULE
     String has context-j character with no rull.

 -- Return code: idn2_rc IDN2_CONTEXTO
     String has forbidden context-o character.

 -- Return code: idn2_rc IDN2_CONTEXTO_NO_RULE
     String has context-o character with no rull.

 -- Return code: idn2_rc IDN2_UNASSIGNED
     String has forbidden unassigned character.

 -- Return code: idn2_rc IDN2_BIDI
     String has forbidden bi-directional properties.

 -- Return code: idn2_rc IDN2_DOT_IN_LABEL
     Label has forbidden dot (TR46).

 -- Return code: idn2_rc IDN2_INVALID_TRANSITIONAL
     Label has character forbidden in transitional mode (TR46).

 -- Return code: idn2_rc IDN2_INVALID_NONTRANSITIONAL
     Label has character forbidden in non-transitional mode (TR46).

Memory Handling
===============

     [file idn2_free.texi not included]

Version Check
=============

It is often desirable to check that the version of Libidn2 used is indeed one which fits all requirements.  Even with binary compatibility new features may have been introduced but due to problem with the dynamic linker an old version is actually used.  So you may want to check that the version is okay right after program startup.

     [file idn2_check_version.texi not included]

The normal way to use the function is to put something similar to the following first in your main:

       if (!idn2_check_version (IDN2_VERSION))
         {
           printf ("idn2_check_version() failed:\n"
                   "Header file incompatible with shared library.\n");
           exit(EXIT_FAILURE);
         }

Converting from libidn
**********************

This library is backwards (API) compatible with the libidn library (https://www.gnu.org/software/libidn/).

Although it is recommended for new software to use the native libidn2 functions (i.e., the ones prefixed with idn2), old software isn't always feasible to modify.

Converting with minimal modifications
=====================================

As such, libidn2, provides compatibility macros which switch all libidn functions, to libidn2 functions in a backwards compatible way. To take advantage of these compatibility functions, it is sufficient to replace the idna.h header in legacy code, with idn2.h. That would transform the software from using libidn, i.e., IDNA2003, to using libidn2 with IDNA2008 non-transitional encoding.

Converting to native APIs
=========================

However, it is recommended to switch applications to the IDN2 native APIs. The following table provides a mapping of libidn code snippets to libidn2, for switching to IDNA2008.

libidn   libidn2

rc = idna_to_ascii_8z (buf, &p, IDNA_USE_STD3_ASCII_RULES);
if (rc != IDNA_SUCCESS)

rc = idn2_to_ascii_8z (buf, &p, IDN2_USE_STD3_ASCII_RULES);
if (rc != IDN2_OK)

rc = idna_to_ascii_8z (buf, &p, 0 /* any other flags */);
if (rc != IDNA_SUCCESS)

/* we recommend to use the default flags (0), so that
 * the default behavior of libidn2 applies. */
rc = idn2_to_ascii_8z (buf, &p, 0);
if (rc != IDN2_OK)

rc = idna_to_unicode_8z8z (buf, &p, 0 /* any flags */);
if (rc != IDNA_SUCCESS)

rc = idn2_to_unicode_8z8z (buf, &p, 0);
if (rc != IDN2_OK)

Note that, although the table only lists the UTF-8 functions, the mapping is identical for every other one on the family of toUnicode and toAscii. As the IDNA2003 details differ signicantly to IDNA2008, not all flags used in the libidn functions map to any specific flags; it is typically safe to use the suggested libidn2 flags. Exceptionally the libidn flag IDNA_USE_STD3_ASCII_RULES is mapped to IDN2_USE_STD3_ASCII_RULES.

Converting with backwards compatibility
=======================================

In several cases where IDNA2008 mappings do not exist whereas IDNA2003 mappings do, software like browsers take a backwards compatible approach. That is convert the domain to IDNA2008 form, and if that fails try the IDNA2003 conversion. The following example demonstrates that approach.

rc = idn2_to_ascii_8z (buf, &p, IDN2_NONTRANSITIONAL); /* IDNA2008 */
if (rc == IDN2_DISALLOWED)
  rc = idn2_to_ascii_8z (buf, &p, IDN2_TRANSITIONAL); /* IDNA2003 - compatible */

Using libidn and libidn2 code
=============================

In the special case of software that needs to support both libraries (e.g., both IDNA2003 and IDNA2008), you must define IDN2_SKIP_LIBIDN_COMPAT prior to including idn2.h in order to disable compatibility code which overlaps with libidn functionality. That would allow software to use both libraries' functions.

Stringprep and libidn2
======================

The original libidn library includes functionality for the stringprep processing in stringprep.h. That functionality was an integral part of an IDNA2003 implementation, but it does not apply to IDNA2008. Furthermore, stringprep processing has been replaced by the PRECIS framework (RFC8264).

For the reasons above, libidn2 does not implement stringprep or any other string processing protocols unrelated to IDNA2008. Applications requiring the stringprep processing should continue using the original libidn, and new applications should consider using the PRECIS framework.

Examples
********

This chapter contains example code which illustrate how Libidn2 is used when you write your own application.

ToASCII example
===============

This example demonstrates how the library is used to convert internationalized domain names into ASCII compatible names (ACE). It expects input to be in UTF-8 form.

     [file example-toascii.c not included]

ToUnicode example
=================

This example demonstrates how the library is used to convert ASCII compatible names (ACE) to internationalized domain names. Both input and output are in UTF-8 form.

     [file example-tounicode.c not included]

Lookup
======

This example demonstrates how a domain name is processed before it is lookup in the DNS. The input expected is in the locale encoding.

     [file lookup.c not included]

Register
========

This example demonstrates how a domain label is processed before it is registered in the DNS. The input expected is in the locale encoding.

     [file register.c not included]

Invoking idn2
*************

idn2 translates internationalized domain names to the IDNA2008 encoded format, either for lookup or registration.

If strings are specified on the command line, they are used as input and the computed output is printed to standard output stdout. If no strings are specified on the command line, the program read data, line by line, from the standard input stdin, and print the computed output to standard output.  What processing is performed (e.g., lookup or register) is indicated by options.  If any errors are encountered, the execution of the applications is aborted.

All strings are expected to be encoded in the preferred charset used by your locale.  Use --debug to find out what this charset is. On POSIX systems you may use the LANG environment variable to specify a different locale.

To process a string that starts with -, for example -foo, use -- to signal the end of parameters, as in idn2 -r -- -foo.

Options
=======

idn2 recognizes these commands:

     [file idn2-help.texi not included]

Environment Variables
=====================

On POSIX systems the LANG environment variable can be used to override the system locale for the command being invoked.  The system locale may influence what character set is used to decode data (i.e., strings on the command line or data read from the standard input stream), and to encode data to the standard output.  If your system is set up correctly, however, the application will use the correct locale and character set automatically.  Example usage:

     $ LANG=en_US.UTF-8 idn2
     ...

Examples
========

Standard usage, reading input from standard input and disabling license and usage instructions:

     jas@latte:~$ idn2 --quiet
     räksmörgås.se
     xn--rksmrgs-5wao1o.se
     ...

Reading input from the command line:

     jas@latte:~$ idn2 räksmörgås.se blåbærgrød.no
     xn--rksmrgs-5wao1o.se
     xn--blbrgrd-fxak7p.no
     jas@latte:~$

Testing the IDNA2008 Register function:

     jas@latte:~$ idn2 --register fußball
     xn--fuball-cta
     jas@latte:~$

Troubleshooting
===============

Getting character data encoded right, and making sure Libidn2 use the same encoding, can be difficult.  The reason for this is that most systems may encode character data in more than one character encoding, i.e., using UTF-8 together with ISO-8859-1 or ISO-2022-JP.  This problem is likely to continue to exist until only one character encoding come out as the evolutionary winner, or (more likely, at least to some extents) forever.

The first step to troubleshooting character encoding problems with Libidn2 is to use the --debug parameter to find out which character set encoding idn2 believe your locale uses.

     jas@latte:~$ idn2 --debug --quiet ""
     Charset: UTF-8

     jas@latte:~$

If it prints ANSI_X3.4-1968 (i.e., US-ASCII), this indicate you have not configured your locale properly.  To configure the locale, you can, for example, use LANG=sv_SE.UTF-8; export LANG at a /bin/sh prompt, to set up your locale for a Swedish environment using UTF-8 as the encoding.

Sometimes idn2 appear to be unable to translate from your system locale into UTF-8 (which is used internally), and you will get an error message like this:

     idn2: lookup: could not convert string to UTF-8

One explanation is that you didn't install the iconv conversion tools.  You can find it as a standalone library in GNU Libiconv (https://www.gnu.org/software/libiconv/).  On many GNU/Linux systems, this library is part of the system, but you may have to install additional packages to be able to use it.

Another explanation is that the error is correct and you are feeding idn2 invalid data.  This can happen inadvertently if you are not careful with the character set encoding you use.  For example, if your shell run in a ISO-8859-1 environment, and you invoke idn2 with the LANG environment variable as follows, you will feed it ISO-8859-1 characters but force it to believe they are UTF-8.  Naturally this will lead to an error, unless the byte sequences happen to be valid UTF-8.  Note that even if you don't get an error, the output may be incorrect in this situation, because ISO-8859-1 and UTF-8 does not in general encode the same characters as the same byte sequences.

     jas@latte:~$ idn2 --quiet --debug ""
     Charset: ISO-8859-1

     jas@latte:~$ LANG=sv_SE.UTF-8 idn2 --debug räksmörgås
     Charset: UTF-8
     input[0] = 0x72
     input[1] = 0xc3
     input[2] = 0xa4
     input[3] = 0xc3
     input[4] = 0xa4
     input[5] = 0x6b
     input[6] = 0x73
     input[7] = 0x6d
     input[8] = 0xc3
     input[9] = 0xb6
     input[10] = 0x72
     input[11] = 0x67
     input[12] = 0xc3
     input[13] = 0xa5
     input[14] = 0x73
     UCS-4 input[0] = U+0072
     UCS-4 input[1] = U+00e4
     UCS-4 input[2] = U+00e4
     UCS-4 input[3] = U+006b
     UCS-4 input[4] = U+0073
     UCS-4 input[5] = U+006d
     UCS-4 input[6] = U+00f6
     UCS-4 input[7] = U+0072
     UCS-4 input[8] = U+0067
     UCS-4 input[9] = U+00e5
     UCS-4 input[10] = U+0073
     output[0] = 0x72
     output[1] = 0xc3
     output[2] = 0xa4
     output[3] = 0xc3
     output[4] = 0xa4
     output[5] = 0x6b
     output[6] = 0x73
     output[7] = 0x6d
     output[8] = 0xc3
     output[9] = 0xb6
     output[10] = 0x72
     output[11] = 0x67
     output[12] = 0xc3
     output[13] = 0xa5
     output[14] = 0x73
     UCS-4 output[0] = U+0072
     UCS-4 output[1] = U+00e4
     UCS-4 output[2] = U+00e4
     UCS-4 output[3] = U+006b
     UCS-4 output[4] = U+0073
     UCS-4 output[5] = U+006d
     UCS-4 output[6] = U+00f6
     UCS-4 output[7] = U+0072
     UCS-4 output[8] = U+0067
     UCS-4 output[9] = U+00e5
     UCS-4 output[10] = U+0073
     xn--rksmrgs-5waap8p
     jas@latte:~$

The sense moral here is to forget about LANG (instead, configure your system locale properly) unless you know what you are doing, and if you want to use LANG, do it carefully and after verifying with --debug that you get the desired results.