class implementation

Defined at line 5081 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

An implementation of simdutf for a particular CPU architecture.

Also used to maintain the currently active implementation. The active

implementation is automatically initialized on first use to the most advanced

implementation supported by the host.

Public Methods

encoding_type autodetect_encoding (const char * input, size_t length)

This function will try to detect the encoding

Parameters

input the string to identify

length the length of the string in bytes.

Returns

the encoding type detected

int detect_encodings (const char * input, size_t length)

This function will try to detect the possible encodings in one pass

Parameters

input the string to identify

length the length of the string in bytes.

Returns

the encoding type detected

result convert_utf8_to_latin1_with_errors (const char * input, size_t length, char * latin1_output)

Convert possibly broken UTF-8 string into latin1 string with errors.

If the string cannot be represented as Latin1, an error

code is returned.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

Parameters

input the UTF-8 string to convert

length the length of the string in bytes

latin1_output the pointer to buffer that can hold conversion result

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of code units validated

if successful.

size_t convert_valid_utf8_to_latin1 (const char * input, size_t length, char * latin1_output)

Convert valid UTF-8 string into latin1 string.

This function assumes that the input string is valid UTF-8 and that it can

be represented as Latin1. If you violate this assumption, the result is

implementation defined and may include system-dependent behavior such as

crashes.

This function is for expert users only and not part of our public API. Use

convert_utf8_to_latin1 instead.

This function is not BOM-aware.

Parameters

input the UTF-8 string to convert

length the length of the string in bytes

latin1_output the pointer to buffer that can hold conversion result

Returns

the number of written char; 0 if the input was not valid UTF-8

string

size_t convert_utf8_to_utf16le (const char * input, size_t length, char16_t * utf16_output)

Convert possibly broken UTF-8 string into UTF-16LE string.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

Parameters

input the UTF-8 string to convert

length the length of the string in bytes

utf16_output the pointer to buffer that can hold conversion result

Returns

the number of written char16_t; 0 if the input was not valid UTF-8

string

size_t convert_utf8_to_utf16be (const char * input, size_t length, char16_t * utf16_output)

Convert possibly broken UTF-8 string into UTF-16BE string.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

Parameters

input the UTF-8 string to convert

length the length of the string in bytes

utf16_output the pointer to buffer that can hold conversion result

Returns

the number of written char16_t; 0 if the input was not valid UTF-8

string

result convert_utf8_to_utf16le_with_errors (const char * input, size_t length, char16_t * utf16_output)

Convert possibly broken UTF-8 string into UTF-16LE string and stop on

error.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

Parameters

input the UTF-8 string to convert

length the length of the string in bytes

utf16_output the pointer to buffer that can hold conversion result

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of code units validated

if successful.

result convert_utf8_to_utf16be_with_errors (const char * input, size_t length, char16_t * utf16_output)

Convert possibly broken UTF-8 string into UTF-16BE string and stop on

error.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

Parameters

input the UTF-8 string to convert

length the length of the string in bytes

utf16_output the pointer to buffer that can hold conversion result

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of code units validated

if successful.

size_t convert_utf16be_to_utf8_with_replacement (const char16_t * input, size_t length, char * utf8_buffer)

Convert possibly broken UTF-16BE string into UTF-8 string, replacing

unpaired surrogates with the Unicode replacement character U+FFFD.

This function always succeeds: unpaired surrogates are replaced with

U+FFFD (3 bytes in UTF-8: 0xEF 0xBF 0xBD).

This function is not BOM-aware.

Parameters

input the UTF-16BE string to convert

length the length of the string in 2-byte code units(char16_t)

utf8_buffer the pointer to buffer that can hold conversion result

Returns

number of written code units

size_t convert_valid_utf16le_to_utf8 (const char16_t * input, size_t length, char * utf8_buffer)

Convert valid UTF-16LE string into UTF-8 string.

This function assumes that the input string is valid UTF-16LE.

This function is not BOM-aware.

Parameters

input the UTF-16LE string to convert

length the length of the string in 2-byte code units(char16_t)

utf8_buffer the pointer to a buffer that can hold the conversionresult

Returns

number of written code units; 0 if conversion is not possible

size_t convert_valid_utf16be_to_utf8 (const char16_t * input, size_t length, char * utf8_buffer)

Convert valid UTF-16BE string into UTF-8 string.

This function assumes that the input string is valid UTF-16BE.

This function is not BOM-aware.

Parameters

input the UTF-16BE string to convert

length the length of the string in 2-byte code units(char16_t)

utf8_buffer the pointer to a buffer that can hold the conversionresult

Returns

number of written code units; 0 if conversion is not possible

size_t convert_utf16le_to_utf32 (const char16_t * input, size_t length, char32_t * utf32_buffer)

Convert possibly broken UTF-16LE string into UTF-32 string.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-16LE string to convert

length the length of the string in 2-byte code units(char16_t)

utf32_buffer the pointer to buffer that can hold conversion result

Returns

number of written code units; 0 if input is not a valid UTF-16LE

string

size_t convert_utf16be_to_utf32 (const char16_t * input, size_t length, char32_t * utf32_buffer)

Convert possibly broken UTF-16BE string into UTF-32 string.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-16BE string to convert

length the length of the string in 2-byte code units(char16_t)

utf32_buffer the pointer to buffer that can hold conversion result

Returns

number of written code units; 0 if input is not a valid UTF-16BE

string

result convert_utf16le_to_utf32_with_errors (const char16_t * input, size_t length, char32_t * utf32_buffer)

Convert possibly broken UTF-16LE string into UTF-32 string and stop on

error.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-16LE string to convert

length the length of the string in 2-byte code units(char16_t)

utf32_buffer the pointer to buffer that can hold conversion result

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of char32_t written if

successful.

result convert_utf16be_to_utf32_with_errors (const char16_t * input, size_t length, char32_t * utf32_buffer)

Convert possibly broken UTF-16BE string into UTF-32 string and stop on

error.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-16BE string to convert

length the length of the string in 2-byte code units(char16_t)

utf32_buffer the pointer to buffer that can hold conversion result

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of char32_t written if

successful.

size_t convert_valid_utf16le_to_utf32 (const char16_t * input, size_t length, char32_t * utf32_buffer)

Convert valid UTF-16LE string into UTF-32 string.

This function assumes that the input string is valid UTF-16LE.

This function is not BOM-aware.

Parameters

input the UTF-16LE string to convert

length the length of the string in 2-byte code units(char16_t)

utf32_buffer the pointer to a buffer that can hold the conversionresult

Returns

number of written code units; 0 if conversion is not possible

size_t convert_valid_utf16be_to_utf32 (const char16_t * input, size_t length, char32_t * utf32_buffer)

Convert valid UTF-16LE string into UTF-32BE string.

This function assumes that the input string is valid UTF-16BE.

This function is not BOM-aware.

Parameters

input the UTF-16BE string to convert

length the length of the string in 2-byte code units(char16_t)

utf32_buffer the pointer to a buffer that can hold the conversionresult

Returns

number of written code units; 0 if conversion is not possible

std::string_view name ()

The name of this implementation.

const implementation *impl = simdutf::active_implementation;

cout

"simdutf is optimized for "

impl->name()

"("

impl->description()

")"

endl;

Returns

the name of the implementation, e.g. "haswell", "westmere", "arm64"

Defined at line 5092 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

std::string_view description ()

The description of this implementation.

const implementation *impl = simdutf::active_implementation;

cout

"simdutf is optimized for "

impl->name()

"("

impl->description()

")"

endl;

Returns

the name of the implementation, e.g. "haswell", "westmere", "arm64"

Defined at line 5103 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

uint32_t required_instruction_sets ()

For internal implementation use

The instruction sets this implementation is compiled against.

Returns

a mask of all required `internal::instruction_set::` values

Defined at line 5143 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

bool supported_by_runtime_system ()

The instruction sets this implementation is compiled against

and the current CPU match. This function may poll the current CPU/system

and should therefore not be called too often if performance is a concern.

Returns

true if the implementation can be safely used on the current system

(determined at runtime)

bool validate_utf8 (const char * buf, size_t len)

Validate the UTF-8 string.

Overridden by each implementation.

Parameters

buf the UTF-8 string to validate.

len the length of the string in bytes.

Returns

true if and only if the string is valid UTF-8.

result validate_utf8_with_errors (const char * buf, size_t len)

Validate the UTF-8 string and stop on errors.

Overridden by each implementation.

Parameters

buf the UTF-8 string to validate.

len the length of the string in bytes.

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of code units validated

if successful.

bool validate_ascii (const char * buf, size_t len)

Validate the ASCII string.

Overridden by each implementation.

Parameters

buf the ASCII string to validate.

len the length of the string in bytes.

Returns

true if and only if the string is valid ASCII.

result validate_ascii_with_errors (const char * buf, size_t len)

Validate the ASCII string and stop on error.

Overridden by each implementation.

Parameters

buf the ASCII string to validate.

len the length of the string in bytes.

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of code units validated

if successful.

bool validate_utf16be_as_ascii (const char16_t * buf, size_t len)

Validate the ASCII string as a UTF-16BE sequence.

An UTF-16 sequence is considered an ASCII sequence

if it could be converted to an ASCII string losslessly.

Overridden by each implementation.

Parameters

buf the UTF-16BE string to validate.

len the length of the string in bytes.

Returns

true if and only if the string is valid ASCII.

bool validate_utf16le_as_ascii (const char16_t * buf, size_t len)

Validate the ASCII string as a UTF-16LE sequence.

An UTF-16 sequence is considered an ASCII sequence

if it could be converted to an ASCII string losslessly.

Overridden by each implementation.

Parameters

buf the UTF-16LE string to validate.

len the length of the string in bytes.

Returns

true if and only if the string is valid ASCII.

bool validate_utf16le (const char16_t * buf, size_t len)

Validate the UTF-16LE string.This function may be best when you expect

the input to be almost always valid. Otherwise, consider using

validate_utf16le_with_errors.

Overridden by each implementation.

This function is not BOM-aware.

Parameters

buf the UTF-16LE string to validate.

len the length of the string in number of 2-byte code units(char16_t).

Returns

true if and only if the string is valid UTF-16LE.

bool validate_utf16be (const char16_t * buf, size_t len)

Validate the UTF-16BE string. This function may be best when you expect

the input to be almost always valid. Otherwise, consider using

validate_utf16be_with_errors.

Overridden by each implementation.

This function is not BOM-aware.

Parameters

buf the UTF-16BE string to validate.

len the length of the string in number of 2-byte code units(char16_t).

Returns

true if and only if the string is valid UTF-16BE.

result validate_utf16le_with_errors (const char16_t * buf, size_t len)

Validate the UTF-16LE string and stop on error. It might be faster than

validate_utf16le when an error is expected to occur early.

Overridden by each implementation.

This function is not BOM-aware.

Parameters

buf the UTF-16LE string to validate.

len the length of the string in number of 2-byte code units(char16_t).

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of code units validated

if successful.

result validate_utf16be_with_errors (const char16_t * buf, size_t len)

Validate the UTF-16BE string and stop on error. It might be faster than

validate_utf16be when an error is expected to occur early.

Overridden by each implementation.

This function is not BOM-aware.

Parameters

buf the UTF-16BE string to validate.

len the length of the string in number of 2-byte code units(char16_t).

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of code units validated

if successful.

void to_well_formed_utf16le (const char16_t * input, size_t len, char16_t * output)

Copies the UTF-16LE string while replacing mismatched surrogates with the

Unicode replacement character U+FFFD. We allow the input and output to be

the same buffer so that the correction is done in-place.

Overridden by each implementation.

Parameters

input the UTF-16LE string to correct.

len the length of the string in number of 2-byte code units(char16_t).

output the output buffer.

void to_well_formed_utf16be (const char16_t * input, size_t len, char16_t * output)

Copies the UTF-16BE string while replacing mismatched surrogates with the

Unicode replacement character U+FFFD. We allow the input and output to be

the same buffer so that the correction is done in-place.

Overridden by each implementation.

Parameters

input the UTF-16BE string to correct.

len the length of the string in number of 2-byte code units(char16_t).

output the output buffer.

bool validate_utf32 (const char32_t * buf, size_t len)

Validate the UTF-32 string.

Overridden by each implementation.

This function is not BOM-aware.

Parameters

buf the UTF-32 string to validate.

len the length of the string in number of 4-byte code units(char32_t).

Returns

true if and only if the string is valid UTF-32.

result validate_utf32_with_errors (const char32_t * buf, size_t len)

Validate the UTF-32 string and stop on error.

Overridden by each implementation.

This function is not BOM-aware.

Parameters

buf the UTF-32 string to validate.

len the length of the string in number of 4-byte code units(char32_t).

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of code units validated

if successful.

size_t convert_latin1_to_utf8 (const char * input, size_t length, char * utf8_output)

Convert Latin1 string into UTF-8 string.

This function is suitable to work with inputs from untrusted sources.

Parameters

input the Latin1 string to convert

length the length of the string in bytes

utf8_output the pointer to buffer that can hold conversion result

Returns

the number of written char; 0 if conversion is not possible

size_t convert_latin1_to_utf16le (const char * input, size_t length, char16_t * utf16_output)

Convert possibly Latin1 string into UTF-16LE string.

This function is suitable to work with inputs from untrusted sources.

Parameters

input the Latin1 string to convert

length the length of the string in bytes

utf16_output the pointer to buffer that can hold conversion result

Returns

the number of written char16_t; 0 if conversion is not possible

size_t convert_latin1_to_utf16be (const char * input, size_t length, char16_t * utf16_output)

Convert Latin1 string into UTF-16BE string.

This function is suitable to work with inputs from untrusted sources.

Parameters

input the Latin1 string to convert

length the length of the string in bytes

utf16_output the pointer to buffer that can hold conversion result

Returns

the number of written char16_t; 0 if conversion is not possible

size_t convert_latin1_to_utf32 (const char * input, size_t length, char32_t * utf32_buffer)

Convert Latin1 string into UTF-32 string.

This function is suitable to work with inputs from untrusted sources.

Parameters

input the Latin1 string to convert

length the length of the string in bytes

utf32_buffer the pointer to buffer that can hold conversion result

Returns

the number of written char32_t; 0 if conversion is not possible

size_t convert_utf8_to_latin1 (const char * input, size_t length, char * latin1_output)

Convert possibly broken UTF-8 string into latin1 string.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

Parameters

input the UTF-8 string to convert

length the length of the string in bytes

latin1_output the pointer to buffer that can hold conversion result

Returns

the number of written char; 0 if the input was not valid UTF-8

string or if it cannot be represented as Latin1

result utf8_length_from_utf16le_with_replacement (const char16_t * input, size_t length)

Compute the number of bytes that this UTF-16LE string would require in

UTF-8 format even when the UTF-16LE content contains mismatched

surrogates that have to be replaced by the replacement character (0xFFFD).

Parameters

input the UTF-16LE string to convert

length the length of the string in 2-byte code units(char16_t)

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) where the count is the number of bytes required to

encode the UTF-16LE string as UTF-8, and the error code is either SUCCESS

or SURROGATE. The count is correct regardless of the error field.

When SURROGATE is returned, it does not indicate an error in the case of

this function: it indicates that at least one surrogate has been

encountered: the surrogates may be matched or not (thus this function does

not validate). If the returned error code is SUCCESS, then the input

contains no surrogate, is in the Basic Multilingual Plane, and is

necessarily valid.

result utf8_length_from_utf16be_with_replacement (const char16_t * input, size_t length)

Compute the number of bytes that this UTF-16BE string would require in

UTF-8 format even when the UTF-16BE content contains mismatched

surrogates that have to be replaced by the replacement character (0xFFFD).

Parameters

input the UTF-16BE string to convert

length the length of the string in 2-byte code units(char16_t)

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) where the count is the number of bytes required to

encode the UTF-16BE string as UTF-8, and the error code is either SUCCESS

or SURROGATE. The count is correct regardless of the error field.

When SURROGATE is returned, it does not indicate an error in the case of

this function: it indicates that at least one surrogate has been

encountered: the surrogates may be matched or not (thus this function does

not validate). If the returned error code is SUCCESS, then the input

contains no surrogate, is in the Basic Multilingual Plane, and is

necessarily valid.

size_t convert_utf8_to_utf32 (const char * input, size_t length, char32_t * utf32_output)

Convert possibly broken UTF-8 string into UTF-32 string.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

Parameters

input the UTF-8 string to convert

length the length of the string in bytes

utf32_output the pointer to buffer that can hold conversion result

Returns

the number of written char16_t; 0 if the input was not valid UTF-8

string

result convert_utf8_to_utf32_with_errors (const char * input, size_t length, char32_t * utf32_output)

Convert possibly broken UTF-8 string into UTF-32 string and stop on error.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

Parameters

input the UTF-8 string to convert

length the length of the string in bytes

utf32_output the pointer to buffer that can hold conversion result

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of char32_t written if

successful.

size_t convert_valid_utf8_to_utf16le (const char * input, size_t length, char16_t * utf16_buffer)

Convert valid UTF-8 string into UTF-16LE string.

This function assumes that the input string is valid UTF-8.

Parameters

input the UTF-8 string to convert

length the length of the string in bytes

utf16_buffer the pointer to buffer that can hold conversion result

Returns

the number of written char16_t

size_t convert_valid_utf8_to_utf16be (const char * input, size_t length, char16_t * utf16_buffer)

Convert valid UTF-8 string into UTF-16BE string.

This function assumes that the input string is valid UTF-8.

Parameters

input the UTF-8 string to convert

length the length of the string in bytes

utf16_buffer the pointer to buffer that can hold conversion result

Returns

the number of written char16_t

size_t convert_valid_utf8_to_utf32 (const char * input, size_t length, char32_t * utf32_buffer)

Convert valid UTF-8 string into UTF-32 string.

This function assumes that the input string is valid UTF-8.

Parameters

input the UTF-8 string to convert

length the length of the string in bytes

utf32_buffer the pointer to buffer that can hold conversion result

Returns

the number of written char32_t

size_t utf16_length_from_utf8 (const char * input, size_t length)

Compute the number of 2-byte code units that this UTF-8 string would

require in UTF-16LE format.

This function does not validate the input. It is acceptable to pass invalid

UTF-8 strings but in such cases the result is implementation defined.

Parameters

input the UTF-8 string to process

length the length of the string in bytes

Returns

the number of char16_t code units required to encode the UTF-8

string as UTF-16LE

size_t utf32_length_from_utf8 (const char * input, size_t length)

Compute the number of 4-byte code units that this UTF-8 string would

require in UTF-32 format.

This function is equivalent to count_utf8. It is acceptable to pass invalid

UTF-8 strings but in such cases the result is implementation defined.

This function does not validate the input.

Parameters

input the UTF-8 string to process

length the length of the string in bytes

Returns

the number of char32_t code units required to encode the UTF-8

string as UTF-32

size_t convert_utf16le_to_latin1 (const char16_t * input, size_t length, char * latin1_buffer)

Convert possibly broken UTF-16LE string into Latin1 string.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-16LE string to convert

length the length of the string in 2-byte code units(char16_t)

latin1_buffer the pointer to buffer that can hold conversionresult

Returns

number of written code units; 0 if input is not a valid UTF-16LE

string or if it cannot be represented as Latin1

size_t convert_utf16be_to_latin1 (const char16_t * input, size_t length, char * latin1_buffer)

Convert possibly broken UTF-16BE string into Latin1 string.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-16BE string to convert

length the length of the string in 2-byte code units(char16_t)

latin1_buffer the pointer to buffer that can hold conversionresult

Returns

number of written code units; 0 if input is not a valid UTF-16BE

string or if it cannot be represented as Latin1

result convert_utf16le_to_latin1_with_errors (const char16_t * input, size_t length, char * latin1_buffer)

Convert possibly broken UTF-16LE string into Latin1 string.

If the string cannot be represented as Latin1, an error

is returned.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-16LE string to convert

length the length of the string in 2-byte code units(char16_t)

latin1_buffer the pointer to buffer that can hold conversionresult

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of char written if

successful.

result convert_utf16be_to_latin1_with_errors (const char16_t * input, size_t length, char * latin1_buffer)

Convert possibly broken UTF-16BE string into Latin1 string.

If the string cannot be represented as Latin1, an error

is returned.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-16BE string to convert

length the length of the string in 2-byte code units(char16_t)

latin1_buffer the pointer to buffer that can hold conversionresult

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of char written if

successful.

size_t convert_valid_utf16le_to_latin1 (const char16_t * input, size_t length, char * latin1_buffer)

Convert valid UTF-16LE string into Latin1 string.

This function assumes that the input string is valid UTF-L16LE and that it

can be represented as Latin1. If you violate this assumption, the result is

implementation defined and may include system-dependent behavior such as

crashes.

This function is for expert users only and not part of our public API. Use

convert_utf16le_to_latin1 instead.

This function is not BOM-aware.

Parameters

input the UTF-16LE string to convert

length the length of the string in 2-byte code units(char16_t)

latin1_buffer the pointer to buffer that can hold conversionresult

Returns

number of written code units; 0 if conversion is not possible

size_t convert_valid_utf16be_to_latin1 (const char16_t * input, size_t length, char * latin1_buffer)

Convert valid UTF-16BE string into Latin1 string.

This function assumes that the input string is valid UTF16-BE and that it

can be represented as Latin1. If you violate this assumption, the result is

implementation defined and may include system-dependent behavior such as

crashes.

This function is for expert users only and not part of our public API. Use

convert_utf16be_to_latin1 instead.

This function is not BOM-aware.

Parameters

input the UTF-16BE string to convert

length the length of the string in 2-byte code units(char16_t)

latin1_buffer the pointer to buffer that can hold conversionresult

Returns

number of written code units; 0 if conversion is not possible

size_t convert_utf16le_to_utf8 (const char16_t * input, size_t length, char * utf8_buffer)

Convert possibly broken UTF-16LE string into UTF-8 string.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-16LE string to convert

length the length of the string in 2-byte code units(char16_t)

utf8_buffer the pointer to buffer that can hold conversion result

Returns

number of written code units; 0 if input is not a valid UTF-16LE

string

size_t convert_utf16be_to_utf8 (const char16_t * input, size_t length, char * utf8_buffer)

Convert possibly broken UTF-16BE string into UTF-8 string.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-16BE string to convert

length the length of the string in 2-byte code units(char16_t)

utf8_buffer the pointer to buffer that can hold conversion result

Returns

number of written code units; 0 if input is not a valid UTF-16BE

string

result convert_utf16le_to_utf8_with_errors (const char16_t * input, size_t length, char * utf8_buffer)

Convert possibly broken UTF-16LE string into UTF-8 string and stop on

error.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-16LE string to convert

length the length of the string in 2-byte code units(char16_t)

utf8_buffer the pointer to buffer that can hold conversion result

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of char written if

successful.

result convert_utf16be_to_utf8_with_errors (const char16_t * input, size_t length, char * utf8_buffer)

Convert possibly broken UTF-16BE string into UTF-8 string and stop on

error.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-16BE string to convert

length the length of the string in 2-byte code units(char16_t)

utf8_buffer the pointer to buffer that can hold conversion result

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of char written if

successful.

size_t convert_utf16le_to_utf8_with_replacement (const char16_t * input, size_t length, char * utf8_buffer)

Convert possibly broken UTF-16LE string into UTF-8 string, replacing

unpaired surrogates with the Unicode replacement character U+FFFD.

This function always succeeds: unpaired surrogates are replaced with

U+FFFD (3 bytes in UTF-8: 0xEF 0xBF 0xBD).

This function is not BOM-aware.

Parameters

input the UTF-16LE string to convert

length the length of the string in 2-byte code units(char16_t)

utf8_buffer the pointer to buffer that can hold conversion result

Returns

number of written code units

size_t utf8_length_from_utf16le (const char16_t * input, size_t length)

Compute the number of bytes that this UTF-16LE string would require in

UTF-8 format.

This function does not validate the input. It is acceptable to pass invalid

UTF-16 strings but in such cases the result is implementation defined.

This function is not BOM-aware.

Parameters

input the UTF-16LE string to convert

length the length of the string in 2-byte code units(char16_t)

Returns

the number of bytes required to encode the UTF-16LE string as UTF-8

size_t utf8_length_from_utf16be (const char16_t * input, size_t length)

Compute the number of bytes that this UTF-16BE string would require in

UTF-8 format.

This function does not validate the input. It is acceptable to pass invalid

UTF-16 strings but in such cases the result is implementation defined.

This function is not BOM-aware.

Parameters

input the UTF-16BE string to convert

length the length of the string in 2-byte code units(char16_t)

Returns

the number of bytes required to encode the UTF-16BE string as UTF-8

size_t convert_utf32_to_latin1 (const char32_t * input, size_t length, char * latin1_buffer)

Convert possibly broken UTF-32 string into Latin1 string.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-32 string to convert

length the length of the string in 4-byte code units(char32_t)

latin1_buffer the pointer to buffer that can hold conversionresult

Returns

number of written code units; 0 if input is not a valid UTF-32

string

result convert_utf32_to_latin1_with_errors (const char32_t * input, size_t length, char * latin1_buffer)

Convert possibly broken UTF-32 string into Latin1 string and stop on error.

If the string cannot be represented as Latin1, an error is returned.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-32 string to convert

length the length of the string in 4-byte code units(char32_t)

latin1_buffer the pointer to buffer that can hold conversionresult

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of char written if

successful.

size_t convert_valid_utf32_to_latin1 (const char32_t * input, size_t length, char * latin1_buffer)

Convert valid UTF-32 string into Latin1 string.

This function assumes that the input string is valid UTF-32 and can be

represented as Latin1. If you violate this assumption, the result is

implementation defined and may include system-dependent behavior such as

crashes.

This function is for expert users only and not part of our public API. Use

convert_utf32_to_latin1 instead.

This function is not BOM-aware.

Parameters

input the UTF-32 string to convert

length the length of the string in 4-byte code units(char32_t)

latin1_buffer the pointer to a buffer that can hold the conversionresult

Returns

number of written code units; 0 if conversion is not possible

size_t convert_utf32_to_utf8 (const char32_t * input, size_t length, char * utf8_buffer)

Convert possibly broken UTF-32 string into UTF-8 string.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-32 string to convert

length the length of the string in 4-byte code units(char32_t)

utf8_buffer the pointer to buffer that can hold conversion result

Returns

number of written code units; 0 if input is not a valid UTF-32

string

result convert_utf32_to_utf8_with_errors (const char32_t * input, size_t length, char * utf8_buffer)

Convert possibly broken UTF-32 string into UTF-8 string and stop on error.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-32 string to convert

length the length of the string in 4-byte code units(char32_t)

utf8_buffer the pointer to buffer that can hold conversion result

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of char written if

successful.

size_t convert_valid_utf32_to_utf8 (const char32_t * input, size_t length, char * utf8_buffer)

Convert valid UTF-32 string into UTF-8 string.

This function assumes that the input string is valid UTF-32.

This function is not BOM-aware.

Parameters

input the UTF-32 string to convert

length the length of the string in 4-byte code units(char32_t)

utf8_buffer the pointer to a buffer that can hold the conversionresult

Returns

number of written code units; 0 if conversion is not possible

size_t convert_utf32_to_utf16le (const char32_t * input, size_t length, char16_t * utf16_buffer)

Convert possibly broken UTF-32 string into UTF-16LE string.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-32 string to convert

length the length of the string in 4-byte code units(char32_t)

utf16_buffer the pointer to buffer that can hold conversion result

Returns

number of written code units; 0 if input is not a valid UTF-32

string

size_t convert_utf32_to_utf16be (const char32_t * input, size_t length, char16_t * utf16_buffer)

Convert possibly broken UTF-32 string into UTF-16BE string.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-32 string to convert

length the length of the string in 4-byte code units(char32_t)

utf16_buffer the pointer to buffer that can hold conversion result

Returns

number of written code units; 0 if input is not a valid UTF-32

string

result convert_utf32_to_utf16le_with_errors (const char32_t * input, size_t length, char16_t * utf16_buffer)

Convert possibly broken UTF-32 string into UTF-16LE string and stop on

error.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-32 string to convert

length the length of the string in 4-byte code units(char32_t)

utf16_buffer the pointer to buffer that can hold conversion result

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of char16_t written if

successful.

result convert_utf32_to_utf16be_with_errors (const char32_t * input, size_t length, char16_t * utf16_buffer)

Convert possibly broken UTF-32 string into UTF-16BE string and stop on

error.

During the conversion also validation of the input string is done.

This function is suitable to work with inputs from untrusted sources.

This function is not BOM-aware.

Parameters

input the UTF-32 string to convert

length the length of the string in 4-byte code units(char32_t)

utf16_buffer the pointer to buffer that can hold conversion result

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in code units) if any, or the number of char16_t written if

successful.

size_t convert_valid_utf32_to_utf16le (const char32_t * input, size_t length, char16_t * utf16_buffer)

Convert valid UTF-32 string into UTF-16LE string.

This function assumes that the input string is valid UTF-32.

This function is not BOM-aware.

Parameters

input the UTF-32 string to convert

length the length of the string in 4-byte code units(char32_t)

utf16_buffer the pointer to a buffer that can hold the conversionresult

Returns

number of written code units; 0 if conversion is not possible

size_t convert_valid_utf32_to_utf16be (const char32_t * input, size_t length, char16_t * utf16_buffer)

Convert valid UTF-32 string into UTF-16BE string.

This function assumes that the input string is valid UTF-32.

This function is not BOM-aware.

Parameters

input the UTF-32 string to convert

length the length of the string in 4-byte code units(char32_t)

utf16_buffer the pointer to a buffer that can hold the conversionresult

Returns

number of written code units; 0 if conversion is not possible

void change_endianness_utf16 (const char16_t * input, size_t length, char16_t * output)

Change the endianness of the input. Can be used to go from UTF-16LE to

UTF-16BE or from UTF-16BE to UTF-16LE.

This function does not validate the input.

This function is not BOM-aware.

Parameters

input the UTF-16 string to process

length the length of the string in 2-byte code units(char16_t)

output the pointer to a buffer that can hold the conversionresult

size_t binary_length_from_base64 (const char * input, size_t length)

Compute the binary length from a base64 input with ASCII spaces.

This function is useful for well-formed base64 inputs that may contain

ASCII spaces (such as line breaks). For such inputs, the result is exact.

The function counts non-whitespace characters (ASCII value > 0x20) and

subtracts padding characters ('=') found at the end.

Parameters

input the base64 input to process

length the length of the base64 input in bytes

Returns

number of binary bytes

size_t binary_length_from_base64 (const char16_t * input, size_t length)

Compute the binary length from a base64 input with ASCII spaces.

This function is useful for well-formed base64 inputs that may contain

ASCII spaces (such as line breaks). For such inputs, the result is exact.

The function counts non-whitespace characters (ASCII value > 0x20) and

subtracts padding characters ('=') found at the end.

Parameters

input the base64 input to process, in ASCII stored as 16-bitunits

length the length of the base64 input in 16-bit units

Returns

number of binary bytes

result base64_to_binary (const char * input, size_t length, char * output, base64_options options, last_chunk_handling_options last_chunk_options)

Convert a base64 input to a binary output.

This function follows the WHATWG forgiving-base64 format, which means that

it will ignore any ASCII spaces in the input. You may provide a padded

input (with one or two equal signs at the end) or an unpadded input

(without any equal signs at the end).

See https://infra.spec.whatwg.org/#forgiving-base64-decode

This function will fail in case of invalid input. When last_chunk_options =

loose, there are two possible reasons for failure: the input contains a

number of base64 characters that when divided by 4, leaves a single

remainder character (BASE64_INPUT_REMAINDER), or the input contains a

character that is not a valid base64 character (INVALID_BASE64_CHARACTER).

You should call this function with a buffer that is at least

maximal_binary_length_from_base64(input, length) bytes long. If you fail to

provide that much space, the function may cause a buffer overflow.

Parameters

input the base64 string to process

length the length of the string in bytes

output the pointer to a buffer that can hold the conversionresult (should be at least maximal_binary_length_from_base64(input, length)bytes long).

options the base64 options to use, can be base64_default orbase64_url, is base64_default by default.

last_chunk_options the handling of the last chunk (default: loose)

Returns

a result pair struct (of type simdutf::result containing the two

fields error and count) with an error code and either position of the error

(in the input in bytes) if any, or the number of bytes written if

successful.

full_result base64_to_binary_details (const char * input, size_t length, char * output, base64_options options, last_chunk_handling_options last_chunk_options)

Convert a base64 input to a binary output while returning more details

than base64_to_binary.