Namespaces

• namespace BOM

• namespace detail

• namespace internal

• namespace literals

• namespace scalar

Enumerations

enum error_code

enum encoding_type

enum endianness

enum base64_options : uint64_t

enum last_chunk_handling_options : uint64_t

Records

• class full_result

• class implementation

• class result

Functions

• simdutf::encoding_type autodetect_encoding (const char * input, size_t length)

  Autodetect the encoding of the input, a single encoding is recommended.

  E.g., the function might return simdutf::encoding_type::UTF8,

  simdutf::encoding_type::UTF16_LE, simdutf::encoding_type::UTF16_BE, or

  simdutf::encoding_type::UTF32_LE.

  Parameters

  input the string to analyze.

  length the length of the string in bytes.

  Returns

  the detected encoding type

• size_t convert_utf8_to_utf16be (const char * input, size_t length, char16_t * utf16_output)

  Convert possibly broken UTF-8 string into UTF-16BE string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  utf16_output the pointer to buffer that can hold conversion result

  Returns

  the number of written char16_t; 0 if the input was not valid UTF-8

  string

• result convert_utf8_to_latin1_with_errors (const char * input, size_t length, char * latin1_output)

  Convert possibly broken UTF-8 string into latin1 string with errors.

  If the string cannot be represented as Latin1, an error

  code is returned.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  latin1_output the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of code units validated if

  successful.

• result convert_utf8_to_utf16_with_errors (const char * input, size_t length, char16_t * utf16_output)

  Using native endianness, convert possibly broken UTF-8 string into UTF-16

  string and stop on error.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  utf16_output the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char16_t written if

  successful.

• size_t convert_utf16_to_latin1 (const char16_t * input, size_t length, char * latin1_buffer)

  Using native endianness, convert possibly broken UTF-16 string into Latin1

  string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16 string to convert

  length the length of the string in 2-byte code units (char16_t)

  latin1_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if input is not a valid UTF-16 string

  or if it cannot be represented as Latin1

• size_t atomic_binary_to_base64 (const char * input, size_t length, char * output, base64_options options)

  atomic_binary_to_base64

  Convert a binary input to a base64 output, using atomic accesses.

  This function comes with a potentially significant performance

  penalty, but it may be useful in some cases where the input

  buffers are shared between threads, to avoid undefined

  behavior in case of data races.

  The function is for advanced users. Its main use case is when

  to silence sanitizer warnings. We have no documented use case

  where this function is actually necessary in terms of practical correctness.

  This function is only available when simdutf is compiled with

  C++20 support and __cpp_lib_atomic_ref >= 201806L. You may check

  the availability of this function by checking the macro

  SIMDUTF_ATOMIC_REF.

  The default option (simdutf::base64_default) uses the characters `+` and `/`

  as part of its alphabet. Further, it adds padding (`=`) at the end of the

  output to ensure that the output length is a multiple of four.

  The URL option (simdutf::base64_url) uses the characters `-` and `_` as part

  of its alphabet. No padding is added at the end of the output.

  This function always succeeds.

  This function is considered experimental. It is not tested by default

  (see the CMake option SIMDUTF_ATOMIC_BASE64_TESTS) nor is it fuzz tested.

  It is not documented in the public API documentation (README). It is

  offered on a best effort basis. We rely on the community for further

  testing and feedback.

  Parameters

  input the binary to process

  length the length of the input in bytes

  output the pointer to a buffer that can hold the conversionresult (should be at least base64_length_from_binary(length) bytes long)

  options the base64 options to use, can be base64_default orbase64_url, is base64_default by default.

  Returns

  number of written bytes, will be equal to

  base64_length_from_binary(length, options)

• result atomic_base64_to_binary_safe (const char16_t * input, size_t length, char * output, size_t & outlen, base64_options options, last_chunk_handling_options last_chunk_options, bool decode_up_to_bad_char)

• bool match_system (endianness e)

  Defined at line 41 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/encoding_types.h

• std::string_view error_to_string (error_code code)

  Defined at line 47 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/error.h

• template <typename chartype>

  result base64_to_binary_safe_impl (const chartype * input, size_t length, char * output, size_t & outlen, base64_options options, last_chunk_handling_options last_chunk_handling_options, bool decode_up_to_bad_char)

  Defined at line 57 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/base64_implementation.h

• simdutf::encoding_type autodetect_encoding (const uint8_t * input, size_t length)

  Defined at line 189 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &input>

  simdutf::encoding_type autodetect_encoding (const detail::input_span_of_byte_like auto & input)

  Autodetect the encoding of the input, a single encoding is recommended.

  E.g., the function might return simdutf::encoding_type::UTF8,

  simdutf::encoding_type::UTF16_LE, simdutf::encoding_type::UTF16_BE, or

  simdutf::encoding_type::UTF32_LE.

  Parameters

  input the string to analyze. can be a anything span-like that has adata() and size() that points to character data: std::string,std::string_view, std::vector<char>, std::span<conststd::byte> etc.

  Returns

  the detected encoding type

  Defined at line 205 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &input>

  int detect_encodings (const detail::input_span_of_byte_like auto & input)

  Defined at line 231 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &utf8_input>

  size_t convert_utf8_to_utf16be (const detail::input_span_of_byte_like auto & utf8_input, std::span<char16_t> utf16_output)

  Defined at line 1223 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &utf8_input, auto &&latin1_output>

  result convert_utf8_to_latin1_with_errors (const detail::input_span_of_byte_like auto & utf8_input, detail::output_span_of_byte_like auto && latin1_output)

  Defined at line 1262 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &utf8_input>

  result convert_utf8_to_utf16_with_errors (const detail::input_span_of_byte_like auto & utf8_input, std::span<char16_t> utf16_output)

  Defined at line 1300 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  size_t convert_utf16_to_utf8_safe (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 1827 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&latin1_output>

  size_t convert_utf16_to_latin1 (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && latin1_output)

  Defined at line 1878 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&latin1_output>

  result convert_utf16_to_latin1_with_errors (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && latin1_output)

  Defined at line 2063 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result convert_utf16be_to_utf32_with_errors (std::span<const char16_t> utf16_input, std::span<char32_t> utf32_output)

  Defined at line 2802 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• std::string_view to_string (encoding_type bom)

• int detect_encodings (const char * input, size_t length)

  Autodetect the possible encodings of the input in one pass.

  E.g., if the input might be UTF-16LE or UTF-8, this function returns

  the value (simdutf::encoding_type::UTF8 | simdutf::encoding_type::UTF16_LE).

  Overridden by each implementation.

  Parameters

  input the string to analyze.

  length the length of the string in bytes.

  Returns

  the detected encoding type

• result convert_utf8_to_utf16le_with_errors (const char * input, size_t length, char16_t * utf16_output)

  Convert possibly broken UTF-8 string into UTF-16LE string and stop on error.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  utf16_output the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char16_t written if

  successful.

• result convert_utf8_to_utf16be_with_errors (const char * input, size_t length, char16_t * utf16_output)

  Convert possibly broken UTF-8 string into UTF-16BE string and stop on error.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  utf16_output the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char16_t written if

  successful.

• size_t convert_utf16le_to_latin1 (const char16_t * input, size_t length, char * latin1_buffer)

  Convert possibly broken UTF-16LE string into Latin1 string.

  If the string cannot be represented as Latin1, an error

  is returned.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16LE string to convert

  length the length of the string in 2-byte code units (char16_t)

  latin1_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if input is not a valid UTF-16LE

  string or if it cannot be represented as Latin1

• size_t convert_utf16be_to_latin1 (const char16_t * input, size_t length, char * latin1_buffer)

  Convert possibly broken UTF-16BE string into Latin1 string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16BE string to convert

  length the length of the string in 2-byte code units (char16_t)

  latin1_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if input is not a valid UTF-16BE

  string or if it cannot be represented as Latin1

• size_t convert_utf16be_to_utf8 (const char16_t * input, size_t length, char * utf8_buffer)

  Convert possibly broken UTF-16BE string into UTF-8 string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16BE string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf8_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if input is not a valid UTF-16LE

  string

• result convert_utf16_to_latin1_with_errors (const char16_t * input, size_t length, char * latin1_buffer)

  Using native endianness, convert possibly broken UTF-16 string into Latin1

  string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16 string to convert

  length the length of the string in 2-byte code units (char16_t)

  latin1_buffer the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char written if

  successful.

• result convert_utf16_to_utf8_with_errors (const char16_t * input, size_t length, char * utf8_buffer)

  Using native endianness, convert possibly broken UTF-16 string into UTF-8

  string and stop on error.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16 string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf8_buffer the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char written if

  successful.

• size_t convert_valid_utf16be_to_latin1 (const char16_t * input, size_t length, char * latin1_buffer)

  Convert valid UTF-16BE string into Latin1 string.

  This function assumes that the input string is valid UTF-16BE and that it can

  be represented as Latin1. If you violate this assumption, the result is

  implementation defined and may include system-dependent behavior such as

  crashes.

  This function is for expert users only and not part of our public API. Use

  convert_utf16be_to_latin1 instead. The function may be removed from the

  library in the future.

  This function is not BOM-aware.

  Parameters

  input the UTF-16BE string to convert

  length the length of the string in 2-byte code units (char16_t)

  latin1_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if conversion is not possible

• size_t convert_utf16le_to_utf32 (const char16_t * input, size_t length, char32_t * utf32_buffer)

  Convert possibly broken UTF-16LE string into UTF-32 string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16LE string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf32_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if input is not a valid UTF-16LE

  string

• size_t convert_utf16be_to_utf32 (const char16_t * input, size_t length, char32_t * utf32_buffer)

  Convert possibly broken UTF-16BE string into UTF-32 string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16BE string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf32_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if input is not a valid UTF-16LE

  string

• result convert_utf16_to_utf32_with_errors (const char16_t * input, size_t length, char32_t * utf32_buffer)

  Using native endianness, convert possibly broken UTF-16 string into

  UTF-32 string and stop on error.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16 string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf32_buffer the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char32_t written if

  successful.

• size_t convert_valid_utf16_to_utf32 (const char16_t * input, size_t length, char32_t * utf32_buffer)

  Using native endianness, convert valid UTF-16 string into UTF-32 string.

  This function assumes that the input string is valid UTF-16 (native

  endianness).

  This function is not BOM-aware.

  Parameters

  input the UTF-16 string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf32_buffer the pointer to a buffer that can hold the conversionresult

  Returns

  number of written code units; 0 if conversion is not possible

• result convert_utf32_to_utf8_with_errors (const char32_t * input, size_t length, char * utf8_buffer)

  Convert possibly broken UTF-32 string into UTF-8 string and stop on error.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  utf8_buffer the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char written if

  successful.

• size_t convert_utf32_to_utf16 (const char32_t * input, size_t length, char16_t * utf16_buffer)

  Using native endianness, convert possibly broken UTF-32 string into a UTF-16

  string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  utf16_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if input is not a valid UTF-32 string

• size_t convert_utf32_to_latin1 (const char32_t * input, size_t length, char * latin1_buffer)

  Convert possibly broken UTF-32 string into Latin1 string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  latin1_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if input is not a valid UTF-32 string

  or if it cannot be represented as Latin1

• result convert_utf32_to_utf16_with_errors (const char32_t * input, size_t length, char16_t * utf16_buffer)

  Using native endianness, convert possibly broken UTF-32 string into UTF-16

  string and stop on error.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  utf16_buffer the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char16_t written if

  successful.

• int detect_encodings (const uint8_t * input, size_t length)

  Defined at line 226 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &utf8_input>

  result convert_utf8_to_utf16le_with_errors (const detail::input_span_of_byte_like auto & utf8_input, std::span<char16_t> utf16_output)

  Defined at line 1335 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &utf8_input>

  result convert_utf8_to_utf16be_with_errors (const detail::input_span_of_byte_like auto & utf8_input, std::span<char16_t> utf16_output)

  Defined at line 1370 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&latin1_output>

  size_t convert_utf16le_to_latin1 (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && latin1_output)

  Defined at line 1915 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&latin1_output>

  size_t convert_utf16be_to_latin1 (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && latin1_output)

  Defined at line 1950 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  result convert_utf16_to_utf8_with_errors (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 2177 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&latin1_output>

  size_t convert_valid_utf16be_to_latin1 (std::span<const char16_t> valid_utf16_input, detail::output_span_of_byte_like auto && latin1_output)

  Defined at line 2518 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t convert_utf16le_to_utf32 (std::span<const char16_t> utf16_input, std::span<char32_t> utf32_output)

  Defined at line 2662 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t convert_utf16be_to_utf32 (std::span<const char16_t> utf16_input, std::span<char32_t> utf32_output)

  Defined at line 2695 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result convert_utf16_to_utf32_with_errors (std::span<const char16_t> utf16_input, std::span<char32_t> utf32_output)

  Defined at line 2731 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t convert_valid_utf16_to_utf32 (std::span<const char16_t> valid_utf16_input, std::span<char32_t> utf32_output)

  Defined at line 2836 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  result convert_utf32_to_utf8_with_errors (std::span<const char32_t> utf32_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 3103 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t convert_utf32_to_utf16 (std::span<const char32_t> utf32_input, std::span<char16_t> utf16_output)

  Defined at line 3174 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&latin1_output>

  size_t convert_utf32_to_latin1 (std::span<const char32_t> utf32_input, detail::output_span_of_byte_like auto && latin1_output)

  Defined at line 3241 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t convert_utf32_to_utf16be (std::span<const char32_t> utf32_input, std::span<char16_t> utf16_output)

  Defined at line 3388 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result convert_utf32_to_utf16_with_errors (std::span<const char32_t> utf32_input, std::span<char16_t> utf16_output)

  Defined at line 3424 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool validate_utf8 (const char * buf, size_t len)

  Validate the UTF-8 string. This function may be best when you expect

  the input to be almost always valid. Otherwise, consider using

  validate_utf8_with_errors.

  Overridden by each implementation.

  Parameters

  buf the UTF-8 string to validate.

  len the length of the string in bytes.

  Returns

  true if and only if the string is valid UTF-8.

• bool validate_ascii (const char * buf, size_t len)

  Validate the ASCII string.

  Overridden by each implementation.

  Parameters

  buf the ASCII string to validate.

  len the length of the string in bytes.

  Returns

  true if and only if the string is valid ASCII.

• bool validate_utf16_as_ascii (const char16_t * buf, size_t len)

  Validate the ASCII string as a UTF-16 sequence.

  An UTF-16 sequence is considered an ASCII sequence

  if it could be converted to an ASCII string losslessly.

  Overridden by each implementation.

  Parameters

  buf the UTF-16 string to validate.

  len the length of the string in bytes.

  Returns

  true if and only if the string is valid ASCII.

• void to_well_formed_utf16le (const char16_t * input, size_t len, char16_t * output)

  Fixes an ill-formed UTF-16LE string by replacing mismatched surrogates with

  the Unicode replacement character U+FFFD. If input and output points to

  different memory areas, the procedure copies string, and it's expected that

  output memory is at least as big as the input. It's also possible to set

  input equal output, that makes replacements an in-place operation.

  Parameters

  input the UTF-16LE string to correct.

  len the length of the string in number of 2-byte code units(char16_t).

  output the output buffer.

• void to_well_formed_utf16be (const char16_t * input, size_t len, char16_t * output)

  Fixes an ill-formed UTF-16BE string by replacing mismatched surrogates with

  the Unicode replacement character U+FFFD. If input and output points to

  different memory areas, the procedure copies string, and it's expected that

  output memory is at least as big as the input. It's also possible to set

  input equal output, that makes replacements an in-place operation.

  Parameters

  input the UTF-16BE string to correct.

  len the length of the string in number of 2-byte code units(char16_t).

  output the output buffer.

• void to_well_formed_utf16 (const char16_t * input, size_t len, char16_t * output)

  Fixes an ill-formed UTF-16 string by replacing mismatched surrogates with the

  Unicode replacement character U+FFFD. If input and output points to different

  memory areas, the procedure copies string, and it's expected that output

  memory is at least as big as the input. It's also possible to set input equal

  output, that makes replacements an in-place operation.

  Parameters

  input the UTF-16 string to correct.

  len the length of the string in number of 2-byte code units(char16_t).

  output the output buffer.

• size_t convert_latin1_to_utf8 (const char * input, size_t length, char * utf8_output)

  Convert Latin1 string into UTF-8 string.

  This function is suitable to work with inputs from untrusted sources.

  Parameters

  input the Latin1 string to convert

  length the length of the string in bytes

  utf8_output the pointer to buffer that can hold conversion result

  Returns

  the number of written char; 0 if conversion is not possible

• size_t convert_latin1_to_utf16le (const char * input, size_t length, char16_t * utf16_output)

  Convert possibly Latin1 string into UTF-16LE string.

  This function is suitable to work with inputs from untrusted sources.

  Parameters

  input the Latin1 string to convert

  length the length of the string in bytes

  utf16_output the pointer to buffer that can hold conversion result

  Returns

  the number of written char16_t; 0 if conversion is not possible

• size_t convert_latin1_to_utf16be (const char * input, size_t length, char16_t * utf16_output)

  Convert Latin1 string into UTF-16BE string.

  This function is suitable to work with inputs from untrusted sources.

  Parameters

  input the Latin1 string to convert

  length the length of the string in bytes

  utf16_output the pointer to buffer that can hold conversion result

  Returns

  the number of written char16_t; 0 if conversion is not possible

• size_t convert_latin1_to_utf16 (const char * input, size_t length, char16_t * utf16_output)

  Using native endianness, convert a Latin1 string into a UTF-16 string.

  Parameters

  input the Latin1 string to convert

  length the length of the string in bytes

  utf16_output the pointer to buffer that can hold conversion result

  Returns

  the number of written char16_t.

• size_t convert_utf8_to_utf32 (const char * input, size_t length, char32_t * utf32_output)

  Convert possibly broken UTF-8 string into UTF-32 string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  utf32_output the pointer to buffer that can hold conversion result

  Returns

  the number of written char32_t; 0 if the input was not valid UTF-8

  string

• size_t convert_utf16le_to_utf8 (const char16_t * input, size_t length, char * utf8_buffer)

  Convert possibly broken UTF-16LE string into UTF-8 string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16LE string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf8_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if input is not a valid UTF-16LE

  string

• result convert_utf16le_to_latin1_with_errors (const char16_t * input, size_t length, char * latin1_buffer)

  Convert possibly broken UTF-16LE string into Latin1 string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16LE string to convert

  length the length of the string in 2-byte code units (char16_t)

  latin1_buffer the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char written if

  successful.

• result convert_utf16be_to_latin1_with_errors (const char16_t * input, size_t length, char * latin1_buffer)

  Convert possibly broken UTF-16BE string into Latin1 string.

  If the string cannot be represented as Latin1, an error

  is returned.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16BE string to convert

  length the length of the string in 2-byte code units (char16_t)

  latin1_buffer the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char written if

  successful.

• size_t convert_valid_utf16le_to_utf8 (const char16_t * input, size_t length, char * utf8_buffer)

  Convert valid UTF-16LE string into UTF-8 string.

  This function assumes that the input string is valid UTF-16LE

  This function is not BOM-aware.

  Parameters

  input the UTF-16LE string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf8_buffer the pointer to a buffer that can hold the conversionresult

  Returns

  number of written code units; 0 if conversion is not possible

• size_t convert_valid_utf16be_to_utf8 (const char16_t * input, size_t length, char * utf8_buffer)

  Convert valid UTF-16BE string into UTF-8 string.

  This function assumes that the input string is valid UTF-16BE.

  This function is not BOM-aware.

  Parameters

  input the UTF-16BE string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf8_buffer the pointer to a buffer that can hold the conversionresult

  Returns

  number of written code units; 0 if conversion is not possible

• size_t convert_utf16_to_utf32 (const char16_t * input, size_t length, char32_t * utf32_buffer)

  Using native endianness, convert possibly broken UTF-16 string into UTF-32

  string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16 string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf32_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if input is not a valid UTF-16LE

  string

• result convert_utf16le_to_utf32_with_errors (const char16_t * input, size_t length, char32_t * utf32_buffer)

  Convert possibly broken UTF-16LE string into UTF-32 string and stop on error.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16LE string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf32_buffer the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char32_t written if

  successful.

• result convert_utf16be_to_utf32_with_errors (const char16_t * input, size_t length, char32_t * utf32_buffer)

  Convert possibly broken UTF-16BE string into UTF-32 string and stop on error.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16BE string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf32_buffer the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char32_t written if

  successful.

• size_t convert_valid_utf16le_to_utf32 (const char16_t * input, size_t length, char32_t * utf32_buffer)

  Convert valid UTF-16LE string into UTF-32 string.

  This function assumes that the input string is valid UTF-16LE.

  This function is not BOM-aware.

  Parameters

  input the UTF-16LE string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf32_buffer the pointer to a buffer that can hold the conversionresult

  Returns

  number of written code units; 0 if conversion is not possible

• size_t convert_valid_utf16be_to_utf32 (const char16_t * input, size_t length, char32_t * utf32_buffer)

  Convert valid UTF-16BE string into UTF-32 string.

  This function assumes that the input string is valid UTF-16LE.

  This function is not BOM-aware.

  Parameters

  input the UTF-16BE string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf32_buffer the pointer to a buffer that can hold the conversionresult

  Returns

  number of written code units; 0 if conversion is not possible

• template <auto &input>

  bool validate_utf8 (const detail::input_span_of_byte_like auto & input)

  Defined at line 253 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool validate_utf16_as_ascii (std::span<const char16_t> input)

  Defined at line 377 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result validate_utf16be_with_errors (std::span<const char16_t> input)

  Defined at line 631 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• void to_well_formed_utf16le (std::span<const char16_t> input, std::span<char16_t> output)

  Defined at line 660 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• void to_well_formed_utf16be (std::span<const char16_t> input, std::span<char16_t> output)

  Defined at line 690 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• void to_well_formed_utf16 (std::span<const char16_t> input, std::span<char16_t> output)

  Defined at line 720 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &input>

  size_t convert_latin1_to_utf16be (const detail::input_span_of_byte_like auto & input, std::span<char16_t> output)

  Defined at line 927 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &utf8_input>

  size_t convert_utf8_to_utf32 (const detail::input_span_of_byte_like auto & utf8_input, std::span<char32_t> utf32_output)

  Defined at line 1405 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  size_t convert_utf16le_to_utf8 (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 1988 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  size_t convert_utf16be_to_utf8 (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 2024 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&latin1_output>

  result convert_utf16le_to_latin1_with_errors (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && latin1_output)

  Defined at line 2099 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&latin1_output>

  result convert_utf16be_to_latin1_with_errors (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && latin1_output)

  Defined at line 2137 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  size_t convert_valid_utf16le_to_utf8 (std::span<const char16_t> valid_utf16_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 2556 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  size_t convert_valid_utf16be_to_utf8 (std::span<const char16_t> valid_utf16_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 2590 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t convert_utf16_to_utf32 (std::span<const char16_t> utf16_input, std::span<char32_t> utf32_output)

  Defined at line 2628 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result convert_utf16le_to_utf32_with_errors (std::span<const char16_t> utf16_input, std::span<char32_t> utf32_output)

  Defined at line 2766 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t convert_valid_utf16le_to_utf32 (std::span<const char16_t> valid_utf16_input, std::span<char32_t> utf32_output)

  Defined at line 2870 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t convert_valid_utf16be_to_utf32 (std::span<const char16_t> valid_utf16_input, std::span<char32_t> utf32_output)

  Defined at line 2904 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t utf16_length_from_utf32 (std::span<const char32_t> valid_utf32_input)

  Defined at line 3696 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t count_utf16 (std::span<const char16_t> valid_utf16_input)

  Defined at line 3831 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t count_utf16be (std::span<const char16_t> valid_utf16_input)

  Defined at line 3893 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool is_partial (last_chunk_handling_options options)

  Defined at line 4121 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• const char * find (const char * start, const char * end, char character)

  Find the first occurrence of a character in a string. If the character is

  not found, return a pointer to the end of the string.

  Parameters

  start the start of the string

  end the end of the string

  character the character to find

  Returns

  a pointer to the first occurrence of the character in the string,

  or a pointer to the end of the string if the character is not found.

  Defined at line 4143 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• const char16_t * find (const char16_t * start, const char16_t * end, char16_t character)

  Defined at line 4157 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t base64_length_from_binary_with_lines (size_t length, base64_options options, size_t line_length)

  Provide the base64 length in bytes given the length of a binary input,

  taking into account line breaks.

  Parameters

  length the length of the input in bytes

  options the base64 options to use (default: base64_default)

  line_length the length of lines, must be at least 4 (otherwise it isinterpreted as 4),

  Returns

  number of base64 bytes

  Defined at line 4447 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool base64_ignorable (char input, base64_options options)

  Check if a character is an ignorable base64 character.

  Checking a large input, character by character, is not computationally

  efficient.

  Parameters

  input the character to check

  options the base64 options to use, is base64_default by default.

  Returns

  true if the character is an ignorable base64 character, false

  otherwise.

  Defined at line 4847 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool base64_ignorable (char16_t input, base64_options options)

  Defined at line 4851 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &binary_input, auto &&output>

  std::tuple<result, std::size_t> atomic_base64_to_binary_safe (const detail::input_span_of_byte_like auto & binary_input, detail::output_span_of_byte_like auto && output, base64_options options, last_chunk_handling_options last_chunk_options, bool decode_up_to_bad_char)

  span overload

  Returns

  a tuple of result and outlen

  Defined at line 5036 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&binary_output>

  std::tuple<result, std::size_t> atomic_base64_to_binary_safe (std::span<const char16_t> base64_input, detail::output_span_of_byte_like auto && binary_output, base64_options options, last_chunk_handling_options last_chunk_options, bool decode_up_to_bad_char)

  span overload

  Returns

  a tuple of result and outlen

  Defined at line 5055 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result validate_utf8_with_errors (const char * buf, size_t len)

  Validate the UTF-8 string and stop on error.

  Overridden by each implementation.

  Parameters

  buf the UTF-8 string to validate.

  len the length of the string in bytes.

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of code units validated if

  successful.

• bool validate_utf16be_as_ascii (const char16_t * buf, size_t len)

  Validate the ASCII string as a UTF-16BE sequence.

  An UTF-16 sequence is considered an ASCII sequence

  if it could be converted to an ASCII string losslessly.

  Overridden by each implementation.

  Parameters

  buf the UTF-16BE string to validate.

  len the length of the string in bytes.

  Returns

  true if and only if the string is valid ASCII.

• bool validate_utf32 (const char32_t * buf, size_t len)

  Validate the UTF-32 string. This function may be best when you expect

  the input to be almost always valid. Otherwise, consider using

  validate_utf32_with_errors.

  Overridden by each implementation.

  This function is not BOM-aware.

  Parameters

  buf the UTF-32 string to validate.

  len the length of the string in number of 4-byte code units(char32_t).

  Returns

  true if and only if the string is valid UTF-32.

• size_t convert_latin1_to_utf32 (const char * input, size_t length, char32_t * utf32_buffer)

  Convert Latin1 string into UTF-32 string.

  This function is suitable to work with inputs from untrusted sources.

  Parameters

  input the Latin1 string to convert

  length the length of the string in bytes

  utf32_buffer the pointer to buffer that can hold conversion result

  Returns

  the number of written char32_t; 0 if conversion is not possible

• result convert_utf8_to_utf32_with_errors (const char * input, size_t length, char32_t * utf32_output)

  Convert possibly broken UTF-8 string into UTF-32 string and stop on error.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  utf32_output the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char32_t written if

  successful.

• result convert_utf16le_to_utf8_with_errors (const char16_t * input, size_t length, char * utf8_buffer)

  Convert possibly broken UTF-16LE string into UTF-8 string and stop on error.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16LE string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf8_buffer the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char written if

  successful.

• result convert_utf16be_to_utf8_with_errors (const char16_t * input, size_t length, char * utf8_buffer)

  Convert possibly broken UTF-16BE string into UTF-8 string and stop on error.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16BE string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf8_buffer the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char written if

  successful.

• size_t convert_utf16_to_utf8_with_replacement (const char16_t * input, size_t length, char * utf8_buffer)

  Convert possibly broken UTF-16 string (native endianness) into UTF-8 string,

  replacing unpaired surrogates with the Unicode replacement character U+FFFD.

  This function always succeeds: unpaired surrogates are replaced with

  U+FFFD (3 bytes in UTF-8: 0xEF 0xBF 0xBD).

  This function is not BOM-aware.

  Parameters

  input the UTF-16 string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf8_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units

• size_t utf8_length_from_utf16 (const char16_t * input, size_t length)

  Using native endianness; Compute the number of bytes that this UTF-16

  string would require in UTF-8 format.

  This function does not validate the input. It is acceptable to pass invalid

  UTF-16 strings but in such cases the result is implementation defined.

  Parameters

  input the UTF-16 string to convert

  length the length of the string in 2-byte code units (char16_t)

  Returns

  the number of bytes required to encode the UTF-16LE string as UTF-8

• size_t convert_valid_utf32_to_utf8 (const char32_t * input, size_t length, char * utf8_buffer)

  Convert valid UTF-32 string into UTF-8 string.

  This function assumes that the input string is valid UTF-32.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  utf8_buffer the pointer to a buffer that can hold the conversionresult

  Returns

  number of written code units; 0 if conversion is not possible

• size_t convert_utf32_to_utf16le (const char32_t * input, size_t length, char16_t * utf16_buffer)

  Convert possibly broken UTF-32 string into UTF-16LE string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  utf16_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if input is not a valid UTF-32 string

• result convert_utf32_to_latin1_with_errors (const char32_t * input, size_t length, char * latin1_buffer)

  Convert possibly broken UTF-32 string into Latin1 string and stop on error.

  If the string cannot be represented as Latin1, an error is returned.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  latin1_buffer the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char written if

  successful.

• result convert_utf32_to_utf16le_with_errors (const char32_t * input, size_t length, char16_t * utf16_buffer)

  Convert possibly broken UTF-32 string into UTF-16LE string and stop on error.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  utf16_buffer the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char16_t written if

  successful.

• result convert_utf32_to_utf16be_with_errors (const char32_t * input, size_t length, char16_t * utf16_buffer)

  Convert possibly broken UTF-32 string into UTF-16BE string and stop on error.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  utf16_buffer the pointer to buffer that can hold conversion result

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of char16_t written if

  successful.

• size_t utf32_length_from_utf16 (const char16_t * input, size_t length)

  Using native endianness; Compute the number of bytes that this UTF-16

  string would require in UTF-32 format.

  This function is equivalent to count_utf16.

  This function does not validate the input. It is acceptable to pass invalid

  UTF-16 strings but in such cases the result is implementation defined.

  This function is not BOM-aware.

  Parameters

  input the UTF-16 string to convert

  length the length of the string in 2-byte code units (char16_t)

  Returns

  the number of bytes required to encode the UTF-16LE string as UTF-32

• size_t count_utf16le (const char16_t * input, size_t length)

  Count the number of code points (characters) in the string assuming that

  it is valid.

  This function assumes that the input string is valid UTF-16LE.

  It is acceptable to pass invalid UTF-16 strings but in such cases

  the result is implementation defined.

  This function is not BOM-aware.

  Parameters

  input the UTF-16LE string to process

  length the length of the string in 2-byte code units (char16_t)

  Returns

  number of code points

• size_t count_utf16be (const char16_t * input, size_t length)

  Count the number of code points (characters) in the string assuming that

  it is valid.

  This function assumes that the input string is valid UTF-16BE.

  It is acceptable to pass invalid UTF-16 strings but in such cases

  the result is implementation defined.

  This function is not BOM-aware.

  Parameters

  input the UTF-16BE string to process

  length the length of the string in 2-byte code units (char16_t)

  Returns

  number of code points

• size_t trim_partial_utf16be (const char16_t * input, size_t length)

  Given a valid UTF-16BE string having a possibly truncated last character,

  this function checks the end of string. If the last character is truncated

  (or partial), then it returns a shorter length (shorter by 1 unit) so that

  the short UTF-16BE strings only contain complete characters. If there is no

  truncated character, the original length is returned.

  This function assumes that the input string is valid UTF-16BE, but possibly

  truncated.

  Parameters

  input the UTF-16BE string to process

  length the length of the string in bytes

  Returns

  the length of the string in bytes, possibly shorter by 1 unit

• size_t trim_partial_utf16 (const char16_t * input, size_t length)

  Given a valid UTF-16 string having a possibly truncated last character,

  this function checks the end of string. If the last character is truncated

  (or partial), then it returns a shorter length (shorter by 1 unit) so that

  the short UTF-16 strings only contain complete characters. If there is no

  truncated character, the original length is returned.

  This function assumes that the input string is valid UTF-16, but possibly

  truncated. We use the native endianness.

  Parameters

  input the UTF-16 string to process

  length the length of the string in bytes

  Returns

  the length of the string in unit, possibly shorter by 1 unit

• size_t maximal_binary_length_from_base64 (const char * input, size_t length)

  Provide the maximal binary length in bytes given the base64 input.

  As long as the input does not contain ignorable characters (e.g., ASCII

  spaces or linefeed characters), the result is exact. In particular, the

  function checks for padding characters.

  The function is fast (constant time). It checks up to two characters at

  the end of the string. The input is not otherwise validated or read.

  Parameters

  input the base64 input to process

  length the length of the base64 input in bytes

  Returns

  maximum number of binary bytes

• size_t maximal_binary_length_from_base64 (const char16_t * input, size_t length)

  Provide the maximal binary length in bytes given the base64 input.

  As long as the input does not contain ignorable characters (e.g., ASCII

  spaces or linefeed characters), the result is exact. In particular, the

  function checks for padding characters.

  The function is fast (constant time). It checks up to two characters at

  the end of the string. The input is not otherwise validated or read.

  Parameters

  input the base64 input to process, in ASCII stored as 16-bitunits

  length the length of the base64 input in 16-bit units

  Returns

  maximal number of binary bytes

• size_t binary_length_from_base64 (const char * input, size_t length)

  Compute the binary length from a base64 input.

  This function is useful for base64 inputs that may contain ASCII whitespaces

  (such as line breaks). For such inputs, the result is exact, and for any

  inputs the result can be used to size the output buffer passed to

  `base64_to_binary`.

  The function ignores whitespace and does not require padding characters

  ('=').

  Parameters

  input the base64 input to process

  length the length of the base64 input in bytes

  Returns

  number of binary bytes

• result base64_to_binary (const char * input, size_t length, char * output, base64_options options, last_chunk_handling_options last_chunk_options)

  Convert a base64 input to a binary output.

  This function follows the WHATWG forgiving-base64 format, which means that it

  will ignore any ASCII spaces in the input. You may provide a padded input

  (with one or two equal signs at the end) or an unpadded input (without any

  equal signs at the end).

  See https://infra.spec.whatwg.org/#forgiving-base64-decode

  This function will fail in case of invalid input. When last_chunk_options =

  loose, there are two possible reasons for failure: the input contains a

  number of base64 characters that when divided by 4, leaves a single remainder

  character (BASE64_INPUT_REMAINDER), or the input contains a character that is

  not a valid base64 character (INVALID_BASE64_CHARACTER).

  When the error is INVALID_BASE64_CHARACTER, r.count contains the index in the

  input where the invalid character was found. When the error is

  BASE64_INPUT_REMAINDER, then r.count contains the number of bytes decoded.

  The default option (simdutf::base64_default) expects the characters `+` and

  `/` as part of its alphabet. The URL option (simdutf::base64_url) expects the

  characters `-` and `_` as part of its alphabet.

  The padding (`=`) is validated if present. There may be at most two padding

  characters at the end of the input. If there are any padding characters, the

  total number of characters (excluding spaces but including padding

  characters) must be divisible by four.

  You should call this function with a buffer that is at least

  maximal_binary_length_from_base64(input, length) bytes long. If you fail to

  provide that much space, the function may cause a buffer overflow.

  Advanced users may want to tailor how the last chunk is handled. By default,

  we use a loose (forgiving) approach but we also support a strict approach

  as well as a stop_before_partial approach, as per the following proposal:

  https://tc39.es/proposal-arraybuffer-base64/spec/#sec-frombase64

  Parameters

  input the base64 string to process

  length the length of the string in bytes

  output the pointer to a buffer that can hold the conversionresult (should be at least maximal_binary_length_from_base64(input, length)bytes long).

  options the base64 options to use, usually base64_default orbase64_url, and base64_default by default.

  last_chunk_options the last chunk handling options,last_chunk_handling_options::loose by defaultbut can also be last_chunk_handling_options::strict orlast_chunk_handling_options::stop_before_partial.

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in bytes) if any, or the number of bytes written if successful.

• result base64_to_binary_safe (const char * input, size_t length, char * output, size_t & outlen, base64_options options, last_chunk_handling_options last_chunk_options, bool decode_up_to_bad_char)

  Convert a base64 input to a binary output.

  This function follows the WHATWG forgiving-base64 format, which means that it

  will ignore any ASCII spaces in the input. You may provide a padded input

  (with one or two equal signs at the end) or an unpadded input (without any

  equal signs at the end).

  See https://infra.spec.whatwg.org/#forgiving-base64-decode

  This function will fail in case of invalid input. When last_chunk_options =

  loose, there are three possible reasons for failure: the input contains a

  number of base64 characters that when divided by 4, leaves a single remainder

  character (BASE64_INPUT_REMAINDER), the input contains a character that is

  not a valid base64 character (INVALID_BASE64_CHARACTER), or the output buffer

  is too small (OUTPUT_BUFFER_TOO_SMALL).

  When OUTPUT_BUFFER_TOO_SMALL, we return both the number of bytes written

  and the number of units processed, see description of the parameters and

  returned value.

  When the error is INVALID_BASE64_CHARACTER, r.count contains the index in the

  input where the invalid character was found. When the error is

  BASE64_INPUT_REMAINDER, then r.count contains the number of bytes decoded.

  The default option (simdutf::base64_default) expects the characters `+` and

  `/` as part of its alphabet. The URL option (simdutf::base64_url) expects the

  characters `-` and `_` as part of its alphabet.

  The padding (`=`) is validated if present. There may be at most two padding

  characters at the end of the input. If there are any padding characters, the

  total number of characters (excluding spaces but including padding

  characters) must be divisible by four.

  The INVALID_BASE64_CHARACTER cases are considered fatal and you are expected

  to discard the output unless the parameter decode_up_to_bad_char is set to

  true. In that case, the function will decode up to the first invalid

  character. Extra padding characters ('=') are considered invalid characters.

  Advanced users may want to tailor how the last chunk is handled. By default,

  we use a loose (forgiving) approach but we also support a strict approach

  as well as a stop_before_partial approach, as per the following proposal:

  https://tc39.es/proposal-arraybuffer-base64/spec/#sec-frombase64

  Parameters

  input the base64 string to process, in ASCII stored as 8-bitor 16-bit units

  length the length of the string in 8-bit or 16-bit units.

  output the pointer to a buffer that can hold the conversionresult.

  outlen the number of bytes that can be written in the outputbuffer. Upon return, it is modified to reflect how many bytes were written.

  options the base64 options to use, can be base64_default orbase64_url, is base64_default by default.

  last_chunk_options the last chunk handling options,last_chunk_handling_options::loose by defaultbut can also be last_chunk_handling_options::strict orlast_chunk_handling_options::stop_before_partial.

  decode_up_to_bad_char if true, the function will decode up to thefirst invalid character. By default (false), it is assumed that the outputbuffer is to be discarded. When there are multiple errors in the input,using decode_up_to_bad_char might trigger a different error.

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and position of the

  INVALID_BASE64_CHARACTER error (in the input in units) if any, or the number

  of units processed if successful.

• result base64_to_binary_safe (const char16_t * input, size_t length, char * output, size_t & outlen, base64_options options, last_chunk_handling_options last_chunk_options, bool decode_up_to_bad_char)

  the span overload has moved to the bottom of the file

• result atomic_base64_to_binary_safe (const char * input, size_t length, char * output, size_t & outlen, base64_options options, last_chunk_handling_options last_chunk_options, bool decode_up_to_bad_char)

  Convert a base64 input to a binary output with a size limit and using atomic

  operations.

  Like `base64_to_binary_safe` but using atomic operations, this function is

  thread-safe for concurrent memory access, allowing the output

  buffers to be shared between threads without undefined behavior in case of

  data races.

  This function comes with a potentially significant performance penalty, but

  is useful when thread safety is needed during base64 decoding.

  This function is only available when simdutf is compiled with

  C++20 support and __cpp_lib_atomic_ref >= 201806L. You may check

  the availability of this function by checking the macro

  SIMDUTF_ATOMIC_REF.

  This function is considered experimental. It is not tested by default

  (see the CMake option SIMDUTF_ATOMIC_BASE64_TESTS) nor is it fuzz tested.

  It is not documented in the public API documentation (README). It is

  offered on a best effort basis. We rely on the community for further

  testing and feedback.

  Parameters

  input the base64 input to decode

  length the length of the input in bytes

  output the pointer to buffer that can hold the conversionresult

  outlen the number of bytes that can be written in the outputbuffer. Upon return, it is modified to reflect how many bytes were written.

  options the base64 options to use (default, url, etc.)

  last_chunk_options the last chunk handling options (loose, strict,stop_before_partial)

  decode_up_to_bad_char if true, the function will decode up to thefirst invalid character. By default (false), it is assumed that the outputbuffer is to be discarded. When there are multiple errors in the input,using decode_up_to_bad_char might trigger a different error.

  Returns

  a result struct with an error code and count indicating error

  position or success

• template <auto &input>

  result validate_utf8_with_errors (const detail::input_span_of_byte_like auto & input)

  Defined at line 285 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &input>

  bool validate_ascii (const detail::input_span_of_byte_like auto & input)

  Defined at line 314 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &utf8_input>

  result convert_utf8_to_utf32_with_errors (const detail::input_span_of_byte_like auto & utf8_input, std::span<char32_t> utf32_output)

  Defined at line 1439 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  result convert_utf16le_to_utf8_with_errors (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 2214 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  result convert_utf16be_to_utf8_with_errors (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 2251 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  size_t convert_utf16be_to_utf8_with_replacement (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 2321 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  size_t convert_utf16_to_utf8_with_replacement (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 2356 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t utf8_length_from_utf16 (std::span<const char16_t> valid_utf16_input)

  Defined at line 2938 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  size_t convert_valid_utf32_to_utf8 (std::span<const char32_t> valid_utf32_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 3137 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t convert_utf32_to_utf16le (std::span<const char32_t> utf32_input, std::span<char16_t> utf16_output)

  Defined at line 3206 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&latin1_output>

  result convert_utf32_to_latin1_with_errors (std::span<const char32_t> utf32_input, detail::output_span_of_byte_like auto && latin1_output)

  Defined at line 3279 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result convert_utf32_to_utf16le_with_errors (std::span<const char32_t> utf32_input, std::span<char16_t> utf16_output)

  Defined at line 3459 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result convert_utf32_to_utf16be_with_errors (std::span<const char32_t> utf32_input, std::span<char16_t> utf16_output)

  Defined at line 3495 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t utf32_length_from_utf16 (std::span<const char16_t> valid_utf16_input)

  Defined at line 3729 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t count_utf16le (std::span<const char16_t> valid_utf16_input)

  Defined at line 3862 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t trim_partial_utf16be (std::span<const char16_t> valid_utf16_input)

  Defined at line 3991 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t trim_partial_utf16 (std::span<const char16_t> valid_utf16_input)

  Defined at line 4055 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• std::string_view to_string (base64_options options)

  Defined at line 4180 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• std::string_view to_string (last_chunk_handling_options options)

  Defined at line 4202 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &input>

  size_t maximal_binary_length_from_base64 (const detail::input_span_of_byte_like auto & input)

  Defined at line 4232 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t maximal_binary_length_from_base64 (std::span<const char16_t> input)

  Defined at line 4265 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &input>

  size_t binary_length_from_base64 (const detail::input_span_of_byte_like auto & input)

  Defined at line 4296 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t binary_length_from_base64 (std::span<const char16_t> input)

  Defined at line 4330 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &input, auto &&binary_output>

  result base64_to_binary (const detail::input_span_of_byte_like auto & input, detail::output_span_of_byte_like auto && binary_output, base64_options options, last_chunk_handling_options last_chunk_options)

  Defined at line 4403 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t base64_length_from_binary (size_t length, base64_options options)

  Provide the base64 length in bytes given the length of a binary input.

  Parameters

  length the length of the input in bytes

  options the base64 options to use (default: base64_default)

  Returns

  number of base64 bytes

  Defined at line 4432 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool base64_valid (char input, base64_options options)

  Check if a character is a valid base64 character.

  Checking a large input, character by character, is not computationally

  efficient.

  Note that padding characters are not considered valid base64 characters in

  this context, nor are spaces.

  Parameters

  input the character to check

  options the base64 options to use, is base64_default by default.

  Returns

  true if the character is a base64 character, false otherwise.

  Defined at line 4868 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool base64_valid (char16_t input, base64_options options)

  Defined at line 4872 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool base64_valid_or_padding (char input, base64_options options)

  Check if a character is a valid base64 character or the padding character

  ('='). Checking a large input, character by character, is not computationally

  efficient.

  Parameters

  input the character to check

  options the base64 options to use, is base64_default by default.

  Returns

  true if the character is a base64 character, false otherwise.

  Defined at line 4886 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool base64_valid_or_padding (char16_t input, base64_options options)

  Defined at line 4891 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &input, auto &&binary_output>

  std::tuple<result, std::size_t> base64_to_binary_safe (const detail::input_span_of_byte_like auto & input, detail::output_span_of_byte_like auto && binary_output, base64_options options, last_chunk_handling_options last_chunk_options, bool decode_up_to_bad_char)

  span overload

  Returns

  a tuple of result and outlen

  Defined at line 7178 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&binary_output>

  std::tuple<result, std::size_t> base64_to_binary_safe (std::span<const char16_t> input, detail::output_span_of_byte_like auto && binary_output, base64_options options, last_chunk_handling_options last_chunk_options, bool decode_up_to_bad_char)

  span overload

  Returns

  a tuple of result and outlen

  Defined at line 7217 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result validate_ascii_with_errors (const char * buf, size_t len)

  Validate the ASCII string and stop on error. It might be faster than

  validate_utf8 when an error is expected to occur early.

  Overridden by each implementation.

  Parameters

  buf the ASCII string to validate.

  len the length of the string in bytes.

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of code units validated if

  successful.

• size_t convert_valid_utf8_to_latin1 (const char * input, size_t length, char * latin1_output)

  Convert valid UTF-8 string into latin1 string.

  This function assumes that the input string is valid UTF-8 and that it can be

  represented as Latin1. If you violate this assumption, the result is

  implementation defined and may include system-dependent behavior such as

  crashes.

  This function is for expert users only and not part of our public API. Use

  convert_utf8_to_latin1 instead. The function may be removed from the library

  in the future.

  This function is not BOM-aware.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  latin1_output the pointer to buffer that can hold conversion result

  Returns

  the number of written char; 0 if the input was not valid UTF-8 string

• size_t convert_utf16le_to_utf8_with_replacement (const char16_t * input, size_t length, char * utf8_buffer)

  Convert possibly broken UTF-16LE string into UTF-8 string, replacing

  unpaired surrogates with the Unicode replacement character U+FFFD.

  This function always succeeds: unpaired surrogates are replaced with

  U+FFFD (3 bytes in UTF-8: 0xEF 0xBF 0xBD).

  This function is not BOM-aware.

  Parameters

  input the UTF-16LE string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf8_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units

• size_t convert_utf16be_to_utf8_with_replacement (const char16_t * input, size_t length, char * utf8_buffer)

  Convert possibly broken UTF-16BE string into UTF-8 string, replacing

  unpaired surrogates with the Unicode replacement character U+FFFD.

  This function always succeeds: unpaired surrogates are replaced with

  U+FFFD (3 bytes in UTF-8: 0xEF 0xBF 0xBD).

  This function is not BOM-aware.

  Parameters

  input the UTF-16BE string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf8_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units

• result utf8_length_from_utf16_with_replacement (const char16_t * input, size_t length)

  Using native endianness; compute the number of bytes that this UTF-16

  string would require in UTF-8 format even when the UTF-16LE content contains

  mismatched surrogates that have to be replaced by the replacement character

  (0xFFFD).

  Parameters

  input the UTF-16 string to convert

  length the length of the string in 2-byte code units (char16_t)

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) where the count is the number of bytes required to

  encode the UTF-16 string as UTF-8, and the error code is either SUCCESS or

  SURROGATE. The count is correct regardless of the error field.

  When SURROGATE is returned, it does not indicate an error in the case of this

  function: it indicates that at least one surrogate has been encountered: the

  surrogates may be matched or not (thus this function does not validate). If

  the returned error code is SUCCESS, then the input contains no surrogate, is

  in the Basic Multilingual Plane, and is necessarily valid.

• size_t convert_valid_utf32_to_latin1 (const char32_t * input, size_t length, char * latin1_buffer)

  Convert valid UTF-32 string into Latin1 string.

  This function assumes that the input string is valid UTF-32 and that it can

  be represented as Latin1. If you violate this assumption, the result is

  implementation defined and may include system-dependent behavior such as

  crashes.

  This function is for expert users only and not part of our public API. Use

  convert_utf32_to_latin1 instead. The function may be removed from the library

  in the future.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  latin1_buffer the pointer to a buffer that can hold the conversionresult

  Returns

  number of written code units; 0 if conversion is not possible

• size_t convert_utf32_to_utf16be (const char32_t * input, size_t length, char16_t * utf16_buffer)

  Convert possibly broken UTF-32 string into UTF-16BE string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  utf16_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if input is not a valid UTF-32 string

• size_t utf32_length_from_utf16le (const char16_t * input, size_t length)

  Compute the number of bytes that this UTF-16LE string would require in UTF-32

  format.

  This function is equivalent to count_utf16le.

  This function does not validate the input. It is acceptable to pass invalid

  UTF-16 strings but in such cases the result is implementation defined.

  This function is not BOM-aware.

  Parameters

  input the UTF-16LE string to convert

  length the length of the string in 2-byte code units (char16_t)

  Returns

  the number of bytes required to encode the UTF-16LE string as UTF-32

• size_t utf32_length_from_utf16be (const char16_t * input, size_t length)

  Compute the number of bytes that this UTF-16BE string would require in UTF-32

  format.

  This function is equivalent to count_utf16be.

  This function does not validate the input. It is acceptable to pass invalid

  UTF-16 strings but in such cases the result is implementation defined.

  This function is not BOM-aware.

  Parameters

  input the UTF-16BE string to convert

  length the length of the string in 2-byte code units (char16_t)

  Returns

  the number of bytes required to encode the UTF-16BE string as UTF-32

• size_t count_utf16 (const char16_t * input, size_t length)

  Count the number of code points (characters) in the string assuming that

  it is valid.

  This function assumes that the input string is valid UTF-16 (native

  endianness). It is acceptable to pass invalid UTF-16 strings but in such

  cases the result is implementation defined.

  This function is not BOM-aware.

  Parameters

  input the UTF-16 string to process

  length the length of the string in 2-byte code units (char16_t)

  Returns

  number of code points

• size_t count_utf8 (const char * input, size_t length)

  Count the number of code points (characters) in the string assuming that

  it is valid.

  This function assumes that the input string is valid UTF-8.

  It is acceptable to pass invalid UTF-8 strings but in such cases

  the result is implementation defined.

  Parameters

  input the UTF-8 string to process

  length the length of the string in bytes

  Returns

  number of code points

• size_t trim_partial_utf16le (const char16_t * input, size_t length)

  Given a valid UTF-16LE string having a possibly truncated last character,

  this function checks the end of string. If the last character is truncated

  (or partial), then it returns a shorter length (shorter by 1 unit) so that

  the short UTF-16LE strings only contain complete characters. If there is no

  truncated character, the original length is returned.

  This function assumes that the input string is valid UTF-16LE, but possibly

  truncated.

  Parameters

  input the UTF-16LE string to process

  length the length of the string in bytes

  Returns

  the length of the string in unit, possibly shorter by 1 unit

• size_t binary_length_from_base64 (const char16_t * input, size_t length)

  Compute the binary length from a base64 input.

  This function is useful for base64 inputs that may contain ASCII whitespaces

  (such as line breaks). For such inputs, the result is exact, and for any

  inputs the result can be used to size the output buffer passed to

  `base64_to_binary`.

  The function ignores whitespace and does not require padding characters

  ('=').

  Parameters

  input the base64 input to process, in ASCII stored as 16-bitunits

  length the length of the base64 input in 16-bit units

  Returns

  number of binary bytes

• size_t binary_to_base64 (const char * input, size_t length, char * output, base64_options options)

  Convert a binary input to a base64 output.

  The default option (simdutf::base64_default) uses the characters `+` and `/`

  as part of its alphabet. Further, it adds padding (`=`) at the end of the

  output to ensure that the output length is a multiple of four.

  The URL option (simdutf::base64_url) uses the characters `-` and `_` as part

  of its alphabet. No padding is added at the end of the output.

  This function always succeeds.

  Parameters

  input the binary to process

  length the length of the input in bytes

  output the pointer to a buffer that can hold the conversionresult (should be at least base64_length_from_binary(length) bytes long)

  options the base64 options to use, can be base64_default orbase64_url, is base64_default by default.

  Returns

  number of written bytes, will be equal to

  base64_length_from_binary(length, options)

• template <auto &input>

  result validate_ascii_with_errors (const detail::input_span_of_byte_like auto & input)

  Defined at line 345 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool validate_utf16be_as_ascii (std::span<const char16_t> input)

  Defined at line 405 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &valid_utf8_input, auto &&latin1_output>

  size_t convert_valid_utf8_to_latin1 (const detail::input_span_of_byte_like auto & valid_utf8_input, detail::output_span_of_byte_like auto && latin1_output)

  Defined at line 1481 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  size_t convert_utf16le_to_utf8_with_replacement (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 2286 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result utf8_length_from_utf16_with_replacement (std::span<const char16_t> valid_utf16_input)

  Defined at line 2974 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&latin1_output>

  size_t convert_valid_utf32_to_latin1 (std::span<const char32_t> valid_utf32_input, detail::output_span_of_byte_like auto && latin1_output)

  Defined at line 3320 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t utf32_length_from_utf16le (std::span<const char16_t> valid_utf16_input)

  Defined at line 3762 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t utf32_length_from_utf16be (std::span<const char16_t> valid_utf16_input)

  Defined at line 3796 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &valid_utf8_input>

  size_t count_utf8 (const detail::input_span_of_byte_like auto & valid_utf8_input)

  Defined at line 3924 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t trim_partial_utf16le (std::span<const char16_t> valid_utf16_input)

  Defined at line 4023 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &input, auto &&binary_output>

  size_t binary_to_base64 (const detail::input_span_of_byte_like auto & input, detail::output_span_of_byte_like auto && binary_output, base64_options options)

  Defined at line 4479 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool validate_utf16le_as_ascii (const char16_t * buf, size_t len)

  Validate the ASCII string as a UTF-16LE sequence.

  An UTF-16 sequence is considered an ASCII sequence

  if it could be converted to an ASCII string losslessly.

  Overridden by each implementation.

  Parameters

  buf the UTF-16LE string to validate.

  len the length of the string in bytes.

  Returns

  true if and only if the string is valid ASCII.

• bool validate_utf16 (const char16_t * buf, size_t len)

  Using native endianness; Validate the UTF-16 string.

  This function may be best when you expect the input to be almost always

  valid. Otherwise, consider using validate_utf16_with_errors.

  Overridden by each implementation.

  This function is not BOM-aware.

  Parameters

  buf the UTF-16 string to validate.

  len the length of the string in number of 2-byte code units(char16_t).

  Returns

  true if and only if the string is valid UTF-16.

• bool validate_utf16le (const char16_t * buf, size_t len)

  Validate the UTF-16LE string. This function may be best when you expect

  the input to be almost always valid. Otherwise, consider using

  validate_utf16le_with_errors.

  Overridden by each implementation.

  This function is not BOM-aware.

  Parameters

  buf the UTF-16LE string to validate.

  len the length of the string in number of 2-byte code units(char16_t).

  Returns

  true if and only if the string is valid UTF-16LE.

• bool validate_utf16be (const char16_t * buf, size_t len)

  Validate the UTF-16BE string. This function may be best when you expect

  the input to be almost always valid. Otherwise, consider using

  validate_utf16be_with_errors.

  Overridden by each implementation.

  This function is not BOM-aware.

  Parameters

  buf the UTF-16BE string to validate.

  len the length of the string in number of 2-byte code units(char16_t).

  Returns

  true if and only if the string is valid UTF-16BE.

• result validate_utf16_with_errors (const char16_t * buf, size_t len)

  Using native endianness; Validate the UTF-16 string and stop on error.

  It might be faster than validate_utf16 when an error is expected to occur

  early.

  Overridden by each implementation.

  This function is not BOM-aware.

  Parameters

  buf the UTF-16 string to validate.

  len the length of the string in number of 2-byte code units(char16_t).

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of code units validated if

  successful.

• size_t convert_valid_utf8_to_utf16 (const char * input, size_t length, char16_t * utf16_buffer)

  Using native endianness, convert valid UTF-8 string into a UTF-16 string.

  This function assumes that the input string is valid UTF-8.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  utf16_buffer the pointer to buffer that can hold conversion result

  Returns

  the number of written char16_t

• size_t convert_valid_utf16_to_utf8 (const char16_t * input, size_t length, char * utf8_buffer)

  Using native endianness, convert valid UTF-16 string into UTF-8 string.

  This function assumes that the input string is valid UTF-16.

  This function is not BOM-aware.

  Parameters

  input the UTF-16 string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf8_buffer the pointer to a buffer that can hold the conversionresult

  Returns

  number of written code units; 0 if conversion is not possible

• size_t convert_valid_utf16_to_latin1 (const char16_t * input, size_t length, char * latin1_buffer)

  Using native endianness, convert UTF-16 string into Latin1 string.

  This function assumes that the input string is valid UTF-16 and that it can

  be represented as Latin1. If you violate this assumption, the result is

  implementation defined and may include system-dependent behavior such as

  crashes.

  This function is for expert users only and not part of our public API. Use

  convert_utf16_to_latin1 instead. The function may be removed from the library

  in the future.

  This function is not BOM-aware.

  Parameters

  input the UTF-16 string to convert

  length the length of the string in 2-byte code units (char16_t)

  latin1_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if conversion is not possible

• size_t utf8_length_from_utf16le (const char16_t * input, size_t length)

  Compute the number of bytes that this UTF-16LE string would require in UTF-8

  format.

  This function does not validate the input. It is acceptable to pass invalid

  UTF-16 strings but in such cases the result is implementation defined.

  Parameters

  input the UTF-16LE string to convert

  length the length of the string in 2-byte code units (char16_t)

  Returns

  the number of bytes required to encode the UTF-16LE string as UTF-8

• size_t utf8_length_from_utf16be (const char16_t * input, size_t length)

  Compute the number of bytes that this UTF-16BE string would require in UTF-8

  format.

  This function does not validate the input. It is acceptable to pass invalid

  UTF-16 strings but in such cases the result is implementation defined.

  Parameters

  input the UTF-16BE string to convert

  length the length of the string in 2-byte code units (char16_t)

  Returns

  the number of bytes required to encode the UTF-16BE string as UTF-8

• size_t convert_utf32_to_utf8 (const char32_t * input, size_t length, char * utf8_buffer)

  Convert possibly broken UTF-32 string into UTF-8 string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  utf8_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if input is not a valid UTF-32 string

• size_t convert_valid_utf32_to_utf16 (const char32_t * input, size_t length, char16_t * utf16_buffer)

  Using native endianness, convert valid UTF-32 string into a UTF-16 string.

  This function assumes that the input string is valid UTF-32.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  utf16_buffer the pointer to a buffer that can hold the conversionresult

  Returns

  number of written code units; 0 if conversion is not possible

• size_t trim_partial_utf8 (const char * input, size_t length)

  Given a valid UTF-8 string having a possibly truncated last character,

  this function checks the end of string. If the last character is truncated

  (or partial), then it returns a shorter length (shorter by 1 to 3 bytes) so

  that the short UTF-8 strings only contain complete characters. If there is no

  truncated character, the original length is returned.

  This function assumes that the input string is valid UTF-8, but possibly

  truncated.

  Parameters

  input the UTF-8 string to process

  length the length of the string in bytes

  Returns

  the length of the string in bytes, possibly shorter by 1 to 3 bytes

• size_t binary_to_base64_with_lines (const char * input, size_t length, char * output, size_t line_length, base64_options options)

  Convert a binary input to a base64 output with line breaks.

  The default option (simdutf::base64_default) uses the characters `+` and `/`

  as part of its alphabet. Further, it adds padding (`=`) at the end of the

  output to ensure that the output length is a multiple of four.

  The URL option (simdutf::base64_url) uses the characters `-` and `_` as part

  of its alphabet. No padding is added at the end of the output.

  This function always succeeds.

  Parameters

  input the binary to process

  length the length of the input in bytes

  output the pointer to a buffer that can hold the conversionresult (should be at least base64_length_from_binary_with_lines(length,options, line_length) bytes long)

  line_length the length of lines, must be at least 4 (otherwise it isinterpreted as 4),

  options the base64 options to use, can be base64_default orbase64_url, is base64_default by default.

  Returns

  number of written bytes, will be equal to

  base64_length_from_binary_with_lines(length, options)

• bool validate_utf16le_as_ascii (std::span<const char16_t> input)

  Defined at line 433 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool validate_utf16 (std::span<const char16_t> input)

  Defined at line 466 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool validate_utf16le (std::span<const char16_t> input)

  Defined at line 499 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool validate_utf16be (std::span<const char16_t> input)

  Defined at line 532 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result validate_utf16_with_errors (std::span<const char16_t> input)

  Defined at line 565 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &valid_utf8_input>

  size_t convert_valid_utf8_to_utf16 (const detail::input_span_of_byte_like auto & valid_utf8_input, std::span<char16_t> utf16_output)

  Defined at line 1514 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &valid_utf8_input>

  size_t convert_valid_utf8_to_utf16be (const detail::input_span_of_byte_like auto & valid_utf8_input, std::span<char16_t> utf16_output)

  Defined at line 1577 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  size_t convert_valid_utf16_to_utf8 (std::span<const char16_t> valid_utf16_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 2392 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&latin1_output>

  size_t convert_valid_utf16_to_latin1 (std::span<const char16_t> valid_utf16_input, detail::output_span_of_byte_like auto && latin1_output)

  Defined at line 2434 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t utf8_length_from_utf16le (std::span<const char16_t> valid_utf16_input)

  Defined at line 3004 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t utf8_length_from_utf16be (std::span<const char16_t> valid_utf16_input)

  Defined at line 3033 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  size_t convert_utf32_to_utf8 (std::span<const char32_t> utf32_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 3067 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t latin1_length_from_utf32 (size_t length)

  Compute the number of bytes that this UTF-32 string would require in Latin1

  format.

  This function does not validate the input. It is acceptable to pass invalid

  UTF-32 strings but in such cases the result is implementation defined.

  This function is not BOM-aware.

  Parameters

  length the length of the string in 4-byte code units (char32_t)

  Returns

  the number of bytes required to encode the UTF-32 string as Latin1

  Defined at line 3352 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t utf32_length_from_latin1 (size_t length)

  Compute the number of bytes that this Latin1 string would require in UTF-32

  format.

  Parameters

  length the length of the string in Latin1 code units (char)

  Returns

  the length of the string in 4-byte code units (char32_t) required to

  encode the Latin1 string as UTF-32

  Defined at line 3365 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t convert_valid_utf32_to_utf16 (std::span<const char32_t> valid_utf32_input, std::span<char16_t> utf16_output)

  Defined at line 3528 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result validate_utf16le_with_errors (const char16_t * buf, size_t len)

  Validate the UTF-16LE string and stop on error. It might be faster than

  validate_utf16le when an error is expected to occur early.

  Overridden by each implementation.

  This function is not BOM-aware.

  Parameters

  buf the UTF-16LE string to validate.

  len the length of the string in number of 2-byte code units(char16_t).

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of code units validated if

  successful.

• result validate_utf16be_with_errors (const char16_t * buf, size_t len)

  Validate the UTF-16BE string and stop on error. It might be faster than

  validate_utf16be when an error is expected to occur early.

  Overridden by each implementation.

  This function is not BOM-aware.

  Parameters

  buf the UTF-16BE string to validate.

  len the length of the string in number of 2-byte code units(char16_t).

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of code units validated if

  successful.

• size_t convert_valid_utf8_to_utf16le (const char * input, size_t length, char16_t * utf16_buffer)

  Convert valid UTF-8 string into UTF-16LE string.

  This function assumes that the input string is valid UTF-8.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  utf16_buffer the pointer to buffer that can hold conversion result

  Returns

  the number of written char16_t

• size_t convert_valid_utf8_to_utf16be (const char * input, size_t length, char16_t * utf16_buffer)

  Convert valid UTF-8 string into UTF-16BE string.

  This function assumes that the input string is valid UTF-8.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  utf16_buffer the pointer to buffer that can hold conversion result

  Returns

  the number of written char16_t

• size_t convert_valid_utf16le_to_latin1 (const char16_t * input, size_t length, char * latin1_buffer)

  Convert valid UTF-16LE string into Latin1 string.

  This function assumes that the input string is valid UTF-16LE and that it can

  be represented as Latin1. If you violate this assumption, the result is

  implementation defined and may include system-dependent behavior such as

  crashes.

  This function is for expert users only and not part of our public API. Use

  convert_utf16le_to_latin1 instead. The function may be removed from the

  library in the future.

  This function is not BOM-aware.

  Parameters

  input the UTF-16LE string to convert

  length the length of the string in 2-byte code units (char16_t)

  latin1_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if conversion is not possible

• size_t convert_valid_utf32_to_utf16le (const char32_t * input, size_t length, char16_t * utf16_buffer)

  Convert valid UTF-32 string into UTF-16LE string.

  This function assumes that the input string is valid UTF-32.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  utf16_buffer the pointer to a buffer that can hold the conversionresult

  Returns

  number of written code units; 0 if conversion is not possible

• size_t convert_valid_utf32_to_utf16be (const char32_t * input, size_t length, char16_t * utf16_buffer)

  Convert valid UTF-32 string into UTF-16BE string.

  This function assumes that the input string is valid UTF-32.

  This function is not BOM-aware.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  utf16_buffer the pointer to a buffer that can hold the conversionresult

  Returns

  number of written code units; 0 if conversion is not possible

• void change_endianness_utf16 (const char16_t * input, size_t length, char16_t * output)

  Change the endianness of the input. Can be used to go from UTF-16LE to

  UTF-16BE or from UTF-16BE to UTF-16LE.

  This function does not validate the input.

  This function is not BOM-aware.

  Parameters

  input the UTF-16 string to process

  length the length of the string in 2-byte code units (char16_t)

  output the pointer to a buffer that can hold the conversionresult

• result validate_utf16le_with_errors (std::span<const char16_t> input)

  Defined at line 598 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• bool validate_utf32 (std::span<const char32_t> input)

  Defined at line 755 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &latin1_input, auto &&utf8_output>

  size_t convert_latin1_to_utf8 (const detail::input_span_of_byte_like auto & latin1_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 820 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t latin1_length_from_utf16 (size_t length)

  Compute the number of bytes that this UTF-16 string would require in Latin1

  format.

  Parameters

  length the length of the string in Latin1 code units (char)

  Returns

  the length of the string in Latin1 code units (char) required to

  encode the UTF-16 string as Latin1

  Defined at line 951 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t utf16_length_from_latin1 (size_t length)

  Compute the number of code units that this Latin1 string would require in

  UTF-16 format.

  Parameters

  length the length of the string in Latin1 code units (char)

  Returns

  the length of the string in 2-byte code units (char16_t) required to

  encode the Latin1 string as UTF-16

  Defined at line 964 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &latin1_input>

  size_t convert_latin1_to_utf32 (const detail::input_span_of_byte_like auto & latin1_input, std::span<char32_t> utf32_output)

  Defined at line 984 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &valid_utf8_input>

  size_t convert_valid_utf8_to_utf16le (const detail::input_span_of_byte_like auto & valid_utf8_input, std::span<char16_t> utf16_output)

  Defined at line 1545 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&latin1_output>

  size_t convert_valid_utf16le_to_latin1 (std::span<const char16_t> valid_utf16_input, detail::output_span_of_byte_like auto && latin1_output)

  Defined at line 2476 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t convert_valid_utf32_to_utf16le (std::span<const char32_t> valid_utf32_input, std::span<char16_t> utf16_output)

  Defined at line 3563 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t convert_valid_utf32_to_utf16be (std::span<const char32_t> valid_utf32_input, std::span<char16_t> utf16_output)

  Defined at line 3597 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• void change_endianness_utf16 (std::span<const char16_t> utf16_input, std::span<char16_t> utf16_output)

  Defined at line 3633 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &valid_utf8_input>

  size_t trim_partial_utf8 (const detail::input_span_of_byte_like auto & valid_utf8_input)

  Defined at line 3955 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &input, auto &&binary_output>

  size_t binary_to_base64_with_lines (const detail::input_span_of_byte_like auto & input, detail::output_span_of_byte_like auto && binary_output, size_t line_length, base64_options options)

  Defined at line 4526 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &input, auto &&binary_output>

  size_t atomic_binary_to_base64 (const detail::input_span_of_byte_like auto & input, detail::output_span_of_byte_like auto && binary_output, base64_options options)

  Defined at line 4592 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result validate_utf32_with_errors (const char32_t * buf, size_t len)

  Validate the UTF-32 string and stop on error. It might be faster than

  validate_utf32 when an error is expected to occur early.

  Overridden by each implementation.

  This function is not BOM-aware.

  Parameters

  buf the UTF-32 string to validate.

  len the length of the string in number of 4-byte code units(char32_t).

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and either position of the error

  (in the input in code units) if any, or the number of code units validated if

  successful.

• size_t convert_latin1_to_utf8_safe (const char * input, size_t length, char * utf8_output, size_t utf8_len)

  Convert Latin1 string into UTF-8 string with output limit.

  This function is suitable to work with inputs from untrusted sources.

  We write as many characters as possible.

  Parameters

  input the Latin1 string to convert

  length the length of the string in bytes

  utf8_output the pointer to buffer that can hold conversion result

  utf8_len the maximum output length

  Returns

  the number of written char; 0 if conversion is not possible

• size_t convert_utf8_to_latin1 (const char * input, size_t length, char * latin1_output)

  Convert possibly broken UTF-8 string into latin1 string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  latin1_output the pointer to buffer that can hold conversion result

  Returns

  the number of written char; 0 if the input was not valid UTF-8 string

  or if it cannot be represented as Latin1

• size_t convert_valid_utf8_to_utf32 (const char * input, size_t length, char32_t * utf32_buffer)

  Convert valid UTF-8 string into UTF-32 string.

  This function assumes that the input string is valid UTF-8.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  utf32_buffer the pointer to buffer that can hold conversion result

  Returns

  the number of written char32_t

• size_t utf8_length_from_utf32 (const char32_t * input, size_t length)

  Compute the number of bytes that this UTF-32 string would require in UTF-8

  format.

  This function does not validate the input. It is acceptable to pass invalid

  UTF-32 strings but in such cases the result is implementation defined.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  Returns

  the number of bytes required to encode the UTF-32 string as UTF-8

• size_t utf16_length_from_utf32 (const char32_t * input, size_t length)

  Compute the number of two-byte code units that this UTF-32 string would

  require in UTF-16 format.

  This function does not validate the input. It is acceptable to pass invalid

  UTF-32 strings but in such cases the result is implementation defined.

  Parameters

  input the UTF-32 string to convert

  length the length of the string in 4-byte code units (char32_t)

  Returns

  the number of bytes required to encode the UTF-32 string as UTF-16

• result base64_to_binary (const char16_t * input, size_t length, char * output, base64_options options, last_chunk_handling_options last_chunk_options)

  Convert a base64 input to a binary output.

  This function follows the WHATWG forgiving-base64 format, which means that it

  will ignore any ASCII spaces in the input. You may provide a padded input

  (with one or two equal signs at the end) or an unpadded input (without any

  equal signs at the end).

  See https://infra.spec.whatwg.org/#forgiving-base64-decode

  This function will fail in case of invalid input. When last_chunk_options =

  loose, there are two possible reasons for failure: the input contains a

  number of base64 characters that when divided by 4, leaves a single remainder

  character (BASE64_INPUT_REMAINDER), or the input contains a character that is

  not a valid base64 character (INVALID_BASE64_CHARACTER).

  When the error is INVALID_BASE64_CHARACTER, r.count contains the index in the

  input where the invalid character was found. When the error is

  BASE64_INPUT_REMAINDER, then r.count contains the number of bytes decoded.

  The default option (simdutf::base64_default) expects the characters `+` and

  `/` as part of its alphabet. The URL option (simdutf::base64_url) expects the

  characters `-` and `_` as part of its alphabet.

  The padding (`=`) is validated if present. There may be at most two padding

  characters at the end of the input. If there are any padding characters, the

  total number of characters (excluding spaces but including padding

  characters) must be divisible by four.

  You should call this function with a buffer that is at least

  maximal_binary_length_from_base64(input, length) bytes long. If you fail

  to provide that much space, the function may cause a buffer overflow.

  Advanced users may want to tailor how the last chunk is handled. By default,

  we use a loose (forgiving) approach but we also support a strict approach

  as well as a stop_before_partial approach, as per the following proposal:

  https://tc39.es/proposal-arraybuffer-base64/spec/#sec-frombase64

  Parameters

  input the base64 string to process, in ASCII stored as 16-bitunits

  length the length of the string in 16-bit units

  output the pointer to a buffer that can hold the conversionresult (should be at least maximal_binary_length_from_base64(input, length)bytes long).

  options the base64 options to use, can be base64_default orbase64_url, is base64_default by default.

  last_chunk_options the last chunk handling options,last_chunk_handling_options::loose by defaultbut can also be last_chunk_handling_options::strict orlast_chunk_handling_options::stop_before_partial.

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) with an error code and position of the

  INVALID_BASE64_CHARACTER error (in the input in units) if any, or the number

  of bytes written if successful.

• result validate_utf32_with_errors (std::span<const char32_t> input)

  Defined at line 790 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &input, auto &&utf8_output>

  size_t convert_latin1_to_utf8_safe (const detail::input_span_of_byte_like auto & input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 857 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &latin1_input>

  size_t convert_latin1_to_utf16le (const detail::input_span_of_byte_like auto & latin1_input, std::span<char16_t> utf16_output)

  Defined at line 896 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &input, auto &&output>

  size_t convert_utf8_to_latin1 (const detail::input_span_of_byte_like auto & input, detail::output_span_of_byte_like auto && output)

  Defined at line 1020 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t convert_utf8_to_utf16 (const char * input, size_t length, char16_t * utf16_output)

  Using native endianness, convert possibly broken UTF-8 string into a UTF-16

  string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  utf16_output the pointer to buffer that can hold conversion result

  Returns

  the number of written char16_t; 0 if the input was not valid UTF-8

  string

• template <auto &input>

  size_t convert_utf8_to_utf16 (const detail::input_span_of_byte_like auto & input, std::span<char16_t> output)

  Defined at line 1056 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result utf8_length_from_utf16be_with_replacement (std::span<const char16_t> valid_utf16_input)

  Defined at line 1128 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &input>

  size_t convert_latin1_to_utf16 (const detail::input_span_of_byte_like auto & input, std::span<char16_t> output)

  Defined at line 1158 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &valid_utf8_input>

  size_t convert_valid_utf8_to_utf32 (const detail::input_span_of_byte_like auto & valid_utf8_input, std::span<char32_t> utf32_output)

  Defined at line 1610 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• result utf8_length_from_utf16le_with_replacement (const char16_t * input, size_t length)

  Compute the number of bytes that this UTF-16LE string would require in UTF-8

  format even when the UTF-16LE content contains mismatched surrogates

  that have to be replaced by the replacement character (0xFFFD).

  Parameters

  input the UTF-16LE string to convert

  length the length of the string in 2-byte code units (char16_t)

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) where the count is the number of bytes required to

  encode the UTF-16LE string as UTF-8, and the error code is either SUCCESS or

  SURROGATE. The count is correct regardless of the error field.

  When SURROGATE is returned, it does not indicate an error in the case of this

  function: it indicates that at least one surrogate has been encountered: the

  surrogates may be matched or not (thus this function does not validate). If

  the returned error code is SUCCESS, then the input contains no surrogate, is

  in the Basic Multilingual Plane, and is necessarily valid.

• result utf8_length_from_utf16be_with_replacement (const char16_t * input, size_t length)

  Compute the number of bytes that this UTF-16BE string would require in UTF-8

  format even when the UTF-16BE content contains mismatched surrogates

  that have to be replaced by the replacement character (0xFFFD).

  Parameters

  input the UTF-16BE string to convert

  length the length of the string in 2-byte code units (char16_t)

  Returns

  a result pair struct (of type simdutf::result containing the two

  fields error and count) where the count is the number of bytes required to

  encode the UTF-16BE string as UTF-8, and the error code is either SUCCESS or

  SURROGATE. The count is correct regardless of the error field.

  When SURROGATE is returned, it does not indicate an error in the case of this

  function: it indicates that at least one surrogate has been encountered: the

  surrogates may be matched or not (thus this function does not validate). If

  the returned error code is SUCCESS, then the input contains no surrogate, is

  in the Basic Multilingual Plane, and is necessarily valid.

• size_t convert_utf8_to_utf16le (const char * input, size_t length, char16_t * utf16_output)

  Convert possibly broken UTF-8 string into UTF-16LE string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in bytes

  utf16_output the pointer to buffer that can hold conversion result

  Returns

  the number of written char16_t; 0 if the input was not valid UTF-8

  string

• size_t utf8_length_from_latin1 (const char * input, size_t length)

  Return the number of bytes that this Latin1 string would require in UTF-8

  format.

  Parameters

  input the Latin1 string to convert

  length the length of the string bytes

  Returns

  the number of bytes required to encode the Latin1 string as UTF-8

• size_t latin1_length_from_utf8 (const char * input, size_t length)

  Compute the number of bytes that this UTF-8 string would require in Latin1

  format.

  This function does not validate the input. It is acceptable to pass invalid

  UTF-8 strings but in such cases the result is implementation defined.

  This function is not BOM-aware.

  Parameters

  input the UTF-8 string to convert

  length the length of the string in byte

  Returns

  the number of bytes required to encode the UTF-8 string as Latin1

• result utf8_length_from_utf16le_with_replacement (std::span<const char16_t> valid_utf16_input)

  Defined at line 1092 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &utf8_input>

  size_t convert_utf8_to_utf16le (const detail::input_span_of_byte_like auto & utf8_input, std::span<char16_t> utf16_output)

  Defined at line 1191 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &latin1_input>

  size_t utf8_length_from_latin1 (const detail::input_span_of_byte_like auto & latin1_input)

  Defined at line 1641 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &valid_utf8_input>

  size_t latin1_length_from_utf8 (const detail::input_span_of_byte_like auto & valid_utf8_input)

  Defined at line 1674 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t utf16_length_from_utf8 (const char * input, size_t length)

  Compute the number of 2-byte code units that this UTF-8 string would require

  in UTF-16LE format.

  This function does not validate the input. It is acceptable to pass invalid

  UTF-8 strings but in such cases the result is implementation defined.

  This function is not BOM-aware.

  Parameters

  input the UTF-8 string to process

  length the length of the string in bytes

  Returns

  the number of char16_t code units required to encode the UTF-8 string

  as UTF-16LE

• template <auto &valid_utf8_input>

  size_t utf16_length_from_utf8 (const detail::input_span_of_byte_like auto & valid_utf8_input)

  Defined at line 1710 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t utf32_length_from_utf8 (const char * input, size_t length)

  Compute the number of 4-byte code units that this UTF-8 string would require

  in UTF-32 format.

  This function is equivalent to count_utf8

  This function does not validate the input. It is acceptable to pass invalid

  UTF-8 strings but in such cases the result is implementation defined.

  This function is not BOM-aware.

  Parameters

  input the UTF-8 string to process

  length the length of the string in bytes

  Returns

  the number of char32_t code units required to encode the UTF-8 string

  as UTF-32

• size_t convert_utf16_to_utf8 (const char16_t * input, size_t length, char * utf8_buffer)

  Using native endianness, convert possibly broken UTF-16 string into UTF-8

  string.

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16 string to convert

  length the length of the string in 2-byte code units (char16_t)

  utf8_buffer the pointer to buffer that can hold conversion result

  Returns

  number of written code units; 0 if input is not a valid UTF-16LE

  string

• template <auto &valid_utf8_input>

  size_t utf32_length_from_utf8 (const detail::input_span_of_byte_like auto & valid_utf8_input)

  Defined at line 1748 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&utf8_output>

  size_t convert_utf16_to_utf8 (std::span<const char16_t> utf16_input, detail::output_span_of_byte_like auto && utf8_output)

  Defined at line 1787 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• size_t convert_utf16_to_utf8_safe (const char16_t * input, size_t length, char * utf8_output, size_t utf8_len)

  Using native endianness, convert possibly broken UTF-16 string into UTF-8

  string with output limit.

  We write as many characters as possible into the output buffer,

  During the conversion also validation of the input string is done.

  This function is suitable to work with inputs from untrusted sources.

  This function is not BOM-aware.

  Parameters

  input the UTF-16 string to convert

  length the length of the string in 16-bit code units (char16_t)

  utf8_output the pointer to buffer that can hold conversion result

  utf8_len the maximum output length

  Returns

  the number of written char; 0 if conversion is not possible

• size_t utf8_length_from_utf32 (std::span<const char32_t> valid_utf32_input)

  Defined at line 3665 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• template <auto &&binary_output>

  result base64_to_binary (std::span<const char16_t> input, detail::output_span_of_byte_like auto && binary_output, base64_options options, last_chunk_handling_options last_chunk_options)

  Defined at line 4665 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• full_result base64_to_binary_details (const char * input, size_t length, char * output, base64_options options, last_chunk_handling_options last_chunk_options)

  Convert a base64 input to a binary output while returning more details

  than base64_to_binary.

  This function follows the WHATWG forgiving-base64 format, which means that it

  will ignore any ASCII spaces in the input. You may provide a padded input

  (with one or two equal signs at the end) or an unpadded input (without any

  equal signs at the end).

  See https://infra.spec.whatwg.org/#forgiving-base64-decode

  Unlike base64_to_binary, this function returns a full_result with both

  input_count and output_count, so you always know how much input was consumed

  and how much output was written. There are three cases where the input may

  not be fully consumed:

  1. stop_before_partial: When last_chunk_options is set to

  stop_before_partial, any incomplete 4-character group at the end of the

  input is left unconsumed. This is useful for streaming/chunked decoding

  where you can carry over the unconsumed input to the next chunk.

  2. INVALID_BASE64_CHARACTER: The input contains a character that is not a

  valid base64 character. In this case, input_count indicates where the

  invalid character was found.

  3. BASE64_INPUT_REMAINDER: When last_chunk_options is loose, the input

  contains a number of base64 characters that, when divided by 4, leaves

  a single remainder character (which cannot encode any bytes).

  You should call this function with a buffer that is at least

  maximal_binary_length_from_base64(input, length) bytes long. If you fail to

  provide that much space, the function may cause a buffer overflow.

  Parameters

  input the base64 string to process

  length the length of the string in bytes

  output the pointer to a buffer that can hold the conversionresult (should be at least maximal_binary_length_from_base64(input, length)bytes long).

  options the base64 options to use, can be base64_default orbase64_url, is base64_default by default.

  last_chunk_options the last chunk handling options,last_chunk_handling_options::loose by defaultbut can also be last_chunk_handling_options::strict orlast_chunk_handling_options::stop_before_partial.

  Returns

  a full_result struct (of type simdutf::full_result containing the

  three fields error, input_count and output_count).

• template <auto &input, auto &&binary_output>

  full_result base64_to_binary_details (const detail::input_span_of_byte_like auto & input, detail::output_span_of_byte_like auto && binary_output, base64_options options, last_chunk_handling_options last_chunk_options)

  Defined at line 4739 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• full_result base64_to_binary_details (const char16_t * input, size_t length, char * output, base64_options options, last_chunk_handling_options last_chunk_options)

  Convert a base64 input to a binary output while returning more details

  than base64_to_binary.

  This function follows the WHATWG forgiving-base64 format, which means that it

  will ignore any ASCII spaces in the input. You may provide a padded input

  (with one or two equal signs at the end) or an unpadded input (without any

  equal signs at the end).

  See https://infra.spec.whatwg.org/#forgiving-base64-decode

  Unlike base64_to_binary, this function returns a full_result with both

  input_count and output_count, so you always know how much input was consumed

  and how much output was written. There are three cases where the input may

  not be fully consumed:

  1. stop_before_partial: When last_chunk_options is set to

  stop_before_partial, any incomplete 4-character group at the end of the

  input is left unconsumed. This is useful for streaming/chunked decoding

  where you can carry over the unconsumed input to the next chunk.

  2. INVALID_BASE64_CHARACTER: The input contains a character that is not a

  valid base64 character. In this case, input_count indicates where the

  invalid character was found.

  3. BASE64_INPUT_REMAINDER: When last_chunk_options is loose, the input

  contains a number of base64 characters that, when divided by 4, leaves

  a single remainder character (which cannot encode any bytes).

  You should call this function with a buffer that is at least

  maximal_binary_length_from_base64(input, length) bytes long. If you fail to

  provide that much space, the function may cause a buffer overflow.

  Parameters

  input the base64 string to process, in ASCII stored as 16-bitunits

  length the length of the string in 16-bit units

  output the pointer to a buffer that can hold the conversionresult (should be at least maximal_binary_length_from_base64(input, length)bytes long).

  options the base64 options to use, can be base64_default orbase64_url, is base64_default by default.

  last_chunk_options the last chunk handling options,last_chunk_handling_options::loose by defaultbut can also be last_chunk_handling_options::strict orlast_chunk_handling_options::stop_before_partial.

  Returns

  a full_result struct (of type simdutf::full_result containing the

  three fields error, input_count and output_count).

• template <auto &&binary_output>

  full_result base64_to_binary_details (std::span<const char16_t> input, detail::output_span_of_byte_like auto && binary_output, base64_options options, last_chunk_handling_options last_chunk_options)

  Defined at line 4815 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/implementation.h

• const internal::available_implementation_list & get_available_implementations ()

  The list of available implementations compiled into simdutf.

• internal::atomic_ptr<const implementation> & get_active_implementation ()

  The active implementation.

  Automatically initialized on first use to the most advanced implementation

  supported by this hardware.

• template <typename chartype>

  result slow_base64_to_binary_safe_impl (const chartype * input, size_t length, char * output, size_t & outlen, base64_options options, last_chunk_handling_options last_chunk_options)

  Defined at line 9 of file ../../third_party/github.com/simdutf/simdutf/src/include/simdutf/base64_implementation.h